KServe and Kubernetes Model Serving

KServe is a model-serving control plane built on Kubernetes. You declare an inference workload as a custom resource, and KServe reconciles that declaration into compute, storage, networking, and scaling resources. The model server then exposes a data-plane API that applications call to obtain predictions or generated output.

This separation is the central mental model. The control plane decides what should run and keeps Kubernetes resources aligned with that desired state. The data plane loads model artifacts and handles inference requests. KServe does not train models, replace a model registry, or make a model accurate. It standardizes how trained models become operated services inside a Kubernetes cluster.

From declaration to prediction

The primary predictive resource is InferenceService. Its required predictor describes the serving runtime, model format, model location, and compute needs. Optional transformer and explainer components add pre-processing, post-processing, or explanations. The KServe controller watches the resource and creates underlying Deployments, Services, routing objects, and autoscaling configuration.

A request enters through a Kubernetes networking layer. In Standard mode, Gateway API is the recommended path, with Ingress available for limited compatibility. In Knative mode, Knative supplies revisions, request-driven autoscaling, and scale-to-zero. Traffic reaches a serving runtime, which loads the model and implements an inference protocol. The response returns through the same network path.

KServe and Kubernetes Model Serving

From declaration to prediction

Relevant careers