PowerInfer

PowerInfer is a high-speed inference engine for running large language models locally, built by the IPADS lab at Shanghai Jiao Tong University. Its core idea is exploiting activation sparsity: only a subset of neurons (the hot ones) fire for any given token, so the engine keeps cold neurons on CPU and hot neurons on GPU. The project reports up to 11.69x faster inference than llama.cpp on the same hardware while keeping accuracy, though that figure comes from the project's own benchmarks and your mileage will vary by model and machine.

It is designed for local deployment on a single consumer GPU, supports ReLU-sparse model families such as Llama 2 and Falcon variants, and stays compatible with llama.cpp's model weights and most of its examples (server, batched generation) for easy migration. Supported platforms include x86-64 Linux and Windows with NVIDIA GPUs, plus Apple Silicon in CPU-only mode.

Honest limitations: the speedups only materialize on sparsity-friendly model architectures, so dense models see no gain. The project is research-lab maintained rather than backed by a company, so support and release cadence are community-driven. Model coverage is narrower than llama.cpp or vLLM, and Mac GPU acceleration has historically lagged the CUDA path.

Build from source on GitHub (Linux/Windows with CUDA, or macOS CPU-only) and point it at a supported ReLU-sparse model.