Don't Panic - KServe and Kubernetes Model Serving

KServe is a model-serving control plane on Kubernetes. You declare an inference workload, and KServe reconciles that declaration into compute, storage, networking, and scaling resources. The model server then exposes a data-plane API that applications call for predictions or generated output. KServe does not train models, replace a model registry, or make a model accurate. It standardizes how trained artifacts become operated services.

The primary resource is InferenceService. Its predictor names the serving runtime, model format, model location, and compute needs. Optional transformer and explainer components add pre-processing, post-processing, or explanations. ServingRuntime and ClusterServingRuntime separate runtime images and defaults from a particular model so platform teams and model teams can own different layers.

Two deployment modes change the operating story. Standard mode uses ordinary Deployments, Services, and HPA or KEDA. Knative mode can scale to zero and reactivate on traffic, which saves idle capacity and can put cold start on the first request. Choose the mode from latency and utilization behavior, not from feature count. Capacity planning must include model memory, accelerator memory, concurrency, batching, and load time. CPU alone rarely describes inference saturation.

The Open Inference Protocol (V2) gives runtimes a common HTTP or gRPC shape for readiness, metadata, and inference. Protocol compatibility does not imply equal performance or framework features. InferenceGraph can sequence, switch, ensemble, or split calls when composition is required, at the cost of more network hops and failure modes.

Read the Intro for control-plane versus data-plane separation. Use the Cheatsheet when you need the resource and mode map. Landscape places KServe among related serving tools; Updates and Upstream track the kserve/kserve release line that changes these APIs.