vLLM and LLM Inference Serving

vLLM is an open-source inference and serving engine for large language models. It runs a model on GPU (and other accelerators), accepts generation requests, and returns text, embeddings, or structured outputs through an OpenAI-compatible HTTP API. The project began in the Sky Computing Lab at UC Berkeley and introduced PagedAttention, the technique that made high-throughput LLM serving practical. This course covers what vLLM is, how it works inside, how it scales, and when it is the right tool for serving an LLM.

Why LLM serving is a distinct problem

A trained LLM is a large set of weights. Inference takes a sequence of input tokens and produces output tokens one at a time. Each generated token depends on the previous tokens, so generation is iterative: one forward pass of the model per token. This iteration is what makes LLM serving unlike serving a classifier or a regressor.

Two properties dominate the design space. First, token generation is memory-IO bound, not compute bound. Loading the model weights and the per-request attention state from GPU memory takes longer than the math that produces the next token. Throughput is therefore governed by how large a batch fits in GPU memory, not by peak FLOPs. Second, each request holds per-token state — the key-value (KV) cache — that grows with sequence length. A 13B parameter model consumes roughly 1 MB of GPU memory per token of sequence state. On an A100 with 40 GB, after loading the 26 GB of weights, only about 14 thousand tokens of KV state fit at once. That ceiling caps the batch size and therefore the throughput.

The job of an LLM serving engine is to use that scarce GPU memory efficiently — to keep the GPU busy with useful work, to fit as many concurrent requests as possible, and to do it without exploding latency.

How naive batching fails LLMs

The simplest approach is static batching: collect N requests, run them as a batch, and finish the batch when every request emits its end-of-sequence token. The problem is that requests finish at different times. In a chat workload, one prompt may produce two tokens and another two hundred. Once the short request finishes, its slot in the batch sits idle until the longest request in the batch completes. The longer the variance in output length, the more GPU cycles are wasted on finished requests. With variable prompts and variable outputs, static batching leaves the GPU underutilized most of the time.

A second problem is memory reservation. A serving system that pre-allocates a contiguous buffer for each request's KV cache, sized for the maximum possible sequence length, reserves memory the request will likely never use. Most sequences end well before the maximum length. The reserved-but-unused space cannot serve another request, so effective batch size shrinks and throughput drops.

Continuous batching

Continuous batching (also called iteration-level or dynamic batching) addresses the first problem. Instead of holding the batch fixed for the lifetime of its members, the scheduler re-evaluates the batch at every decode iteration. When a request finishes, the scheduler drops it and admits a waiting request into the freed slot on the next iteration. The GPU never waits for the slowest request in a batch to finish before starting new work.

vLLM and LLM Inference Serving

Why LLM serving is a distinct problem

How naive batching fails LLMs

Continuous batching

Relevant careers