vLLM and LLM Inference Serving

vLLM is an open-source engine that runs large language model inference on GPUs and serves the results through an OpenAI-compatible API. It exists to make LLM serving fast and cheap by cutting the memory wasted on the key-value cache, so more requests fit on one GPU.

itArtificial intelligence and machine learning

Don't Panic — vLLM and LLM Inference Serving

vLLM is the bit of an LLM system that takes a model which is very good at predicting the next token and asks it to do that for many people without turning a GPU into an expensive waiting room. It accepts requests through an OpenAI-compatible API, then makes the hardware spend more of its time producing tokens and less of it admiring unused memory. That is a surprisingly specialised job, because generation happens one token at a time.

The first important passenger is the KV cache, the per-request attention state that grows as a prompt and response grow. Before vLLM, a server could reserve one large continuous area for every request, sized for the longest possible conversation. Most conversations finish early. The unused reservation cannot help anybody else, which is efficient only if the GPU is collecting empty seats as a hobby.

vLLM's answer is PagedAttention. It breaks that cache into fixed-size blocks and gives a request another block only when it needs one. The blocks need not sit next to each other, so the service can use the memory it has rather than the neatly rectangular memory it wishes it had. Shared prompt prefixes can also reuse their already-computed blocks. That matters when many requests begin with the same system prompt or document.

The other moving part is continuous batching, where the scheduler rebuilds the batch at every generation step. A request that finishes leaves; a waiting request can take its place. This avoids the static-batching ritual in which a short request sits finished while the longest response in the group continues its novel. The scheduler balances prompt processing, called prefill, against ongoing token generation, called decode, while the cache places a hard limit on the work it can admit.

The architecture is not a single mysterious box. An API server handles HTTP and tokenization. An engine core schedules requests and manages cache blocks. GPU workers load weight shards and run forward passes. When one GPU is not enough, tensor parallelism splits layer work within a node, pipeline parallelism splits layers across stages, and data parallelism adds replicas for throughput. More machinery helps only when the model, interconnect, and workload call for it; a quiet single-GPU service may not need this whole travelling circus.

Start with the Intro for the full route from request to worker. Use the Slides to see how cache, scheduler, and parallelism fit together. Keep the Cheatsheet nearby when reading startup capacity signals or choosing a scaling strategy. The Field Notes then covers the practical traps: cache pressure, preemption, prompt shape, and the evidence that says which constraint has actually arrived.

Don't Panic — vLLM and LLM Inference Serving

Relevant careers