NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is software that loads trained machine learning models and serves their predictions through standard network APIs. It places scheduling, batching, model execution, and operational metrics between client applications and model runtimes on CPUs or GPUs.

itArtificial intelligence and machine learning

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is an open-source server for running trained machine learning models in production. A client sends tensors through HTTP or gRPC. Triton routes each request through a scheduler to a model backend, runs inference on a CPU or GPU, and returns output tensors. The server standardizes this path across model formats without replacing training frameworks or cluster orchestration.

Triton is useful when a team needs one serving layer for models produced by TensorRT, ONNX Runtime, PyTorch, TensorFlow, Python, or another supported backend. It can serve several models and versions from one process. It can also run preprocessing, inference, and postprocessing as one ensemble. These features concentrate model-specific execution policy in a model repository instead of duplicating it in every client application.

The request path

A deployment begins with a model repository. Each top-level directory represents a model. Numeric subdirectories hold model versions, while config.pbtxt describes inputs, outputs, batching, instance placement, version policy, and scheduling behavior. Triton can infer part of this configuration for some backends, but an explicit configuration makes the serving contract reviewable.

When a request arrives, the server frontend validates the model name, version, tensor names, data types, and shapes. The request then enters that model's scheduler. A stateless model can use the default scheduler or dynamic batcher. A stateful model uses the sequence batcher so requests with the same correlation identity reach the correct model instance in order. An ensemble scheduler moves tensors between composing models without sending intermediate values back to the client.

The scheduler dispatches work to a model instance. An instance is one execution copy of a model assigned to a CPU or GPU. Multiple instances can run concurrently when memory and compute permit. The backend translates Triton's request into the runtime-specific operation, such as executing a TensorRT engine or an ONNX graph. Results travel back through the frontend to the client.

NVIDIA Triton Inference Server

The request path

Relevant careers