Don't Panic — Machine Learning Model Serving

Model serving is the machinery that lets an application ask a trained model for a prediction without wheeling a notebook into production and hoping nobody notices. The model artifact is loaded, an endpoint accepts input, inference runs, and a response returns. That is the corridor version. The longer version involves queues, versioning, and hardware that has opinions about being kept busy.

The useful picture is a request moving through a small obstacle course: parsing and preprocessing, model inference, then postprocessing and a response. Somewhere around it sit health checks, logs, metrics, replicas, and a load balancer. None are decorative. A prediction that arrives after its latency budget, or vanishes when one machine does, has achieved a kind of interpretive freedom that applications rarely appreciate.

The central bargain is batching. A GPU prefers several inputs in one forward pass, so a serving runtime may collect requests for a short time and process them together. This improves throughput, but every request spends some time waiting. Set a maximum batch size and a maximum wait time; otherwise the system can be wonderfully efficient at making a lonely request wait for company.

A model artifact is also not the whole deployment. The input format, preprocessing, output format, and model version all shape what clients receive. That is why a safe rollout keeps versions identifiable and sends only a small traffic slice to a candidate first. Canary, blue-green, and shadow deployments differ in routing, but they share the same modest ambition: discover a bad change before it owns all the traffic.

Large language models add a special nuisance. They generate tokens one at a time and retain a key-value cache, which is memory used to continue the sequence. The cache grows with the conversation, so LLM runtimes such as vLLM treat memory management and continuous batching as part of serving, not as an afterthought with a nicer logo.

Read the Intro for the full architecture and the four inference patterns. Use Slides when the latency, throughput, rollout, and scaling choices need a compact map. Keep the Cheatsheet nearby when comparing formats, runtimes, and metrics. Then try the Practice Reference and exercise: a tiny versioned endpoint makes the difference between a callable function and a service pleasantly, and usefully, difficult to ignore.