vLLM for Fast Inference

What it is

vLLM is an open-source inference and serving engine for large language models, designed for high throughput and memory efficiency in production. It is a server-side library, not a desktop app: you use it when you need to serve a model to many users at once and want to squeeze the most out of your GPUs. If Ollama is for running a model on your laptop, vLLM is for running it for ten thousand users.

It began at UC Berkeley's Sky Computing Lab, developed by Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, and colleagues. The launch blog post, "vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention," appeared on June 20, 2023, and the underlying PagedAttention paper was presented at SOSP 2023, the top systems conference. Berkeley contributed the project to the Linux Foundation in July 2024, and the PyTorch Foundation took it on as a hosted project in 2025. The creators later launched a commercial startup, Inferact, which raised a $150 million seed round reported in January 2026.

How it works

The centerpiece is PagedAttention, a new way of managing the transformer's key-value cache during generation. When a model generates text, it caches the keys and values of previous tokens to avoid recomputing them. Traditional serving pre-allocates large contiguous blocks of GPU memory for each request's cache, which wastes space through fragmentation and over-reservation, like a restaurant that seats every party of two at a table for eight.

PagedAttention borrows the operating system's virtual memory idea: it stores the cache in small non-contiguous blocks, or pages, allocated on demand. Memory that would have sat empty gets used, and the waste nearly disappears. The "v" in the name originally stood for "virtual," a nod to that inspiration, according to a maintainer.

Around that core, vLLM layers continuous batching of incoming requests, so new requests join the processing stream without waiting for a batch to finish; chunked prefill, which splits long prompts into manageable pieces; prefix caching, which reuses computation for repeated prompt prefixes; speculative decoding, which drafts tokens with a small model and verifies them with the large one; and support for quantized formats including FP8, INT8, INT4, GPTQ, AWQ, and GGUF. It serves over a hundred Hugging Face model architectures, offers an OpenAI-compatible API server plus Anthropic Messages and gRPC interfaces, supports distributed inference across GPUs with multiple parallelism strategies, and runs on NVIDIA, AMD, Intel, and Google TPU hardware.

The headline performance claim comes from the authors' own launch blog: up to 24 times the throughput of Hugging Face Transformers on their evaluated workloads, with no changes to the model architecture. That is the authors' self-reported figure against one specific baseline, not an independent benchmark, and it describes throughput under load rather than single-request latency.

Key details

vLLM is Apache 2.0 licensed and free to use, including commercially. There is no published pricing because there is nothing to buy; the commercial vehicle is the Inferact startup, whose pricing was not public as of this writing. The project's governance under the Linux and PyTorch Foundations gives it a neutral home, which matters to the cloud providers and hardware vendors that contribute to it.

Limitations

vLLM is infrastructure, and it behaves like infrastructure. Deploying it well requires GPU hardware, networking, monitoring, and operational effort, which makes it far more complex than running a model locally with a tool like Ollama. The 24x throughput figure describes the authors' tests against one baseline, and real-world speedups depend on the workload, the hardware, and the model; memory-bound workloads benefit most, and the gains shrink when the GPUs were not the bottleneck.

It is primarily a Python and CUDA stack, so teams outside that ecosystem face a learning curve, and keeping up with its rapid development requires attention. For anyone serving open models at scale, though, it has become the default engine to beat, and most serious inference providers either use it or compete directly against it.

Who runs it

vLLM's adoption curve is the clearest evidence that the performance claims held up in practice. Major inference providers built their serving stacks on it or around it, and it became the default answer when teams asked how to serve open models at scale. The project's governance helped: hosted under the Linux Foundation and then the PyTorch Foundation, it reads as neutral infrastructure rather than any single vendor's project, which made adoption safe for competitors to share.

The ecosystem around it grew quickly. Deployment tooling, monitoring integrations, and managed platforms abstracted away the operational complexity for teams that wanted vLLM's performance without running it themselves. Hardware vendors contributed backend support because fast inference sells chips, creating a virtuous cycle: better kernels attracted more users, whose workloads motivated more optimization.

The Inferact startup, founded by the original creators with $150 million in seed funding, is the commercial bet that inference infrastructure is a business, not just a project. The open-source core remains Apache licensed and foundation-governed, which constrains how much value the startup can capture exclusively. That tension, between the commons and the company, is familiar from every successful infrastructure project. So far the commons is winning: vLLM's position as the engine to beat looks secure, and the competition it faces mostly runs on ideas it pioneered.