ML Model Deployment: Patterns and Architecture

A trained model is not a deployed model. The notebook that hit 0.91 AUC on a held-out split is an artifact; production is a contract about latency, freshness, rollback, and what happens at 3 a.m. when the feature pipeline that fed that 0.91 starts returning nulls. The result of underestimating that gap is the failure mode we keep getting called in to unwind: a good model that nobody can ship safely, scale predictably, or update without an outage.

This is a working guide to ml model deployment for the engineer who has to own the result. We will walk the three serving modes that cover almost every workload, the packaging and serving stack that turns a checkpoint into an endpoint, the rollout strategies that let you change a live model without holding your breath, and a reference architecture you can map onto your own infrastructure. The wrong default here is expensive in a way that does not show up until traffic does, so every decision is framed with its failure mode named.

The three serving modes: batch, online, streaming

Before you containerize anything, decide how predictions reach their consumer. This single choice cascades into every other part of the architecture: how you package the model, what you autoscale on, how you monitor it, and what it costs to run. There are three modes, and the deciding question is the latency budget: how fresh a prediction has to be at the moment of consumption.

Batch inference runs the model on a schedule over a bounded set of inputs and writes results to a store the application reads later. Think nightly churn scores, a daily recommendation refresh, a weekly propensity table. The model never sits behind a live request. You trade freshness for the cheapest, simplest, most reliable mode there is: no autoscaling, no tail-latency drama, just a job that either finished or did not. If a prediction computed at 2 a.m. is still useful when a user shows up at 9 a.m., batch is the right answer, and reaching for real-time serving instead is the most common over-engineering mistake we see.

Online inference, also called real-time serving, puts the model behind a synchronous request and returns a prediction inside the caller's latency budget: fraud scoring at checkout, search ranking, a recommendation that reflects the item you are looking at right now. This is the mode people picture when they say deploy machine learning model, and it earns every word of operational complexity in the rest of this guide. You are now on the hook for a p99, for autoscaling under bursty load, for graceful degradation when a dependency is slow.

Streaming inference scores events as they arrive on a continuous flow: a Kafka or Pulsar topic, a Flink job, a real-time feature pipeline. It looks like online serving but the trigger is an event, not a user request, and the consumer is usually another system, as in real-time anomaly detection on telemetry. Streaming buys continuous freshness at the cost of the hardest operational surface of the three: a stateful, always-on pipeline where backpressure, ordering, and exactly-once semantics are now your problem.

Model deployment patterns that survive contact with traffic

Once the serving mode is fixed, a small set of model deployment patterns describe how the model itself is wrapped and addressed. Pick the wrong one and you inherit either a tangle you cannot scale or an indirection layer you did not need. Four are worth naming, and the boundary between them is who owns the model's lifecycle relative to the application.

Model-embedded-in-service is the simplest: the model loads into the same process that serves requests, usually a FastAPI or gRPC app, and predicts in-process. Lowest latency, no extra network hop, trivial to reason about. The cost is coupling. The model's memory footprint and its release cadence are now welded to the application's: you redeploy the whole service to ship a new model, and a 4 GB checkpoint sets your pod size. Fine for one model and a small team; it stops scaling the day you have ten models on different cadences.

Model-as-a-service puts the model behind its own dedicated inference server (Triton, TorchServe, BentoML, KServe, or a vLLM instance for generative workloads) and the application calls it over the network. The model now scales, versions, and deploys on its own axis. You pay one network hop and the weight of a second tier, and you gain the ability to share a GPU across callers, batch requests server-side, and roll a new model version without touching the app. This is the default we reach for past a single model, and the pattern the reference architecture below assumes.

Two more patterns sit on top of those. Multi-model serving hosts many models behind one server with a registry lookup, so a single fleet serves a long tail of small models without a pod per model; Triton's model repository and BentoML's multi-model runners both do this. Serverless on-demand serving spins the model up per request on Modal or a similar platform, trading cold-start latency for zero idle cost, which is right for spiky, low-QPS internal models and wrong for anything latency-sensitive at steady load.

Packaging: from checkpoint to reproducible artifact

A model in production is three things bundled together: the weights, the exact preprocessing and postprocessing code, and a pinned dependency closure. Drift between any of these and the training environment is where the most maddening bugs live. The model is "the same" but a tokenizer minor version or a different NumPy shifts a feature by a rounding error and recall quietly drops. Packaging exists to make that drift impossible.

The reproducible unit is a container. You build a Docker image that pins the runtime, the framework, and the serving code, and that image is what gets tested, scanned, and promoted unchanged from staging to production. We keep the model weights out of the image layer: store them in a registry (MLflow's model registry, an S3 path under DVC, or a Hugging Face repo) and reference an immutable version, so the same image can serve model v3 today and v4 tomorrow without a rebuild. Pin everything. "latest" is how you get a different model than the one you evaluated.

Below the container sits the question of model format. A native PyTorch or TensorFlow checkpoint is portable but slow and heavy at serve time. Exporting to ONNX, TensorRT, or a Triton-native format unlocks graph optimization, quantization, and hardware-specific kernels, frequently a 2–4× throughput gain on the same GPU, which is the difference between three replicas and one. The trade is a brittle export step that has to be validated: an exported graph can silently diverge from the eager model on edge cases, so we re-run the eval set against the exported artifact, not just the original.

# Reproducible serving image. Weights are NOT baked in — they're pulled
# at startup from the model registry by immutable version, so this same
# image serves model v3 today and v4 tomorrow without a rebuild.
FROM nvcr.io/nvidia/tritonserver:24.05-py3

# Pin the dependency closure. 'latest' is how you serve a model you never evaluated.
COPY requirements.lock /tmp/requirements.lock
RUN pip install --no-cache-dir -r /tmp/requirements.lock

ENV MODEL_REGISTRY_URI="s3://paiteq-models"
ENV MODEL_NAME="fraud-scorer"
ENV MODEL_VERSION="3"          # immutable — change via deploy, never 'latest'

COPY serving/ /opt/serving/
COPY entrypoint.sh /opt/entrypoint.sh

# entrypoint pulls weights for $MODEL_VERSION, then launches Triton
ENTRYPOINT ["/opt/entrypoint.sh"]

The serving layer: turning the artifact into an endpoint

The serving layer is where ml model serving stops being a noun and becomes an operational system. Its job is to take the packaged artifact and expose it as something a caller can hit, with the throughput, batching, and concurrency the workload needs. The choice of server is one of the higher-leverage decisions in the stack, because it sets your ceiling on hardware efficiency.

For classical and deep-learning models, NVIDIA Triton is the throughput leader. It does dynamic batching out of the box, collecting incoming requests for a few milliseconds and running them as one GPU batch, plus concurrent model execution, multiple framework backends, and a model repository for multi-model hosting. TorchServe is the lighter PyTorch-native option, simpler to stand up when you are all-PyTorch and don't need Triton's full surface. BentoML sits a layer up: it packages model, code, and dependencies into a "Bento" with a Python-first authoring experience, which we like when the preprocessing is non-trivial Python.

Generative models are a different regime. An LLM endpoint is dominated by autoregressive decoding, and the right server is vLLM or its peers, because the win is PagedAttention and continuous batching: interleaving requests at the token level instead of the request level, which raises GPU utilization sharply under load. Putting an LLM behind a generic request-batching server leaves most of your GPU on the table. When we inherit a deployment that mixes classical models and an LLM, we expect two serving stacks, not one, and make that split explicit rather than forcing both into one server.

#!/usr/bin/env bash
# Dynamic batching + concurrent execution come from the model config,
# not flags. This just points Triton at the pulled model repository.
set -euo pipefail

tritonserver \
  --model-repository=/models \
  --model-control-mode=explicit \
  --load-model=fraud-scorer \
  --strict-model-config=false \
  --allow-metrics=true \
  --metrics-port=8002       # Prometheus scrape target for autoscaling

#!/usr/bin/env bash
# Continuous batching + PagedAttention are on by default. The knob that
# matters most under load is max-num-seqs (concurrent sequences in flight).
set -euo pipefail

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --port 8000

# The embedded pattern: model loads in-process, no second tier.
# Lowest latency, tightest coupling. Good for one small model.
from fastapi import FastAPI
from pydantic import BaseModel
import joblib, os

MODEL = joblib.load(os.environ["MODEL_PATH"])   # pinned version, pulled at startup
app = FastAPI()

class Req(BaseModel):
    features: list[float]

@app.post("/predict")
def predict(r: Req):
    score = float(MODEL.predict_proba([r.features])[0][1])
    return {"score": score, "model_version": os.environ["MODEL_VERSION"]}

Rollout strategies: shipping a new model without holding your breath

What separates a mature deployment from a fragile one is not the model; it is what happens when you change the model. A new version that looks better offline can be worse online for reasons your eval set never saw: a shifted input distribution, a slow new feature, a calibration that drifted. Rollout strategies exist so you find that out on a sliver of traffic instead of all of it. Four are worth knowing, and you usually stack them.

Shadow deployment runs the new model on live traffic but throws its predictions away: the old model still serves, the new one scores the same requests in parallel, and you compare. It is the safest possible test because users never see the candidate, and the only way to validate latency and prediction distribution under real load before a single user is affected. The cost is double inference compute for the window plus a logging path to capture both predictions.

Canary deployment routes a small slice of live traffic, starting at 1–5%, to the new model while the rest stays on the old one, watches the guardrail metrics, and ramps if they hold. This is where users do touch the new model, so the canary is gated on automated checks: error rate, p99 latency, and a proxy metric like prediction-positive-rate. If a guardrail trips, traffic snaps back to the old version automatically. Canary is the workhorse; most production model changes should go out this way.

Blue-green keeps two full environments (blue live, green holding the new version fully warmed) and cuts traffic over all at once, with rollback being an instant flip back to blue. It gives the fastest rollback of any strategy and the cleanest story for stateful or schema-coupled changes, at the cost of running two full fleets during the cutover. Underneath all three is the plain rolling update Kubernetes does by default, replacing pods a few at a time. It is fine for stateless infra changes but a poor fit for model changes on its own, because it gives you no traffic-percentage control and no model-aware guardrail.

Scaling and the economics of GPU serving

Scaling a model service is not the same problem as scaling a stateless web service, because the unit of work is expensive and often pinned to a GPU. Two things make it hard: idle GPU capacity is a budget line you will be asked about, and model containers are large enough that cold starts run tens of seconds, not milliseconds. Your autoscaling has to respect both.

Scale on the right signal. CPU utilization is the wrong one for a GPU service; it can read low while the GPU is saturated and the queue is backing up. Autoscale on request-queue depth or a latency SLO instead, which is what KServe and Knative-based serving expose. Set a minimum replica count above zero for anything latency-sensitive so you never pay a cold start on a user request, and scale to zero only for spiky internal models where a few seconds of cold start is acceptable.

Those bars are illustrative orders of magnitude, not a benchmark of your model; the real numbers depend on model size, sequence length, and hardware. The ordering is what holds: optimize the serving layer before you scale the fleet, because horizontal scaling multiplies whatever per-GPU efficiency you start with. In our 2026-Q1 throughput testing, vLLM continuous batching sustained 540% the requests-per-second of naive per-request inference on the same single GPU under concurrent load. Three replicas of a server we made twice as efficient beats six replicas of a naive one, at half the GPU bill.

A reference model deployment architecture

Here is a model deployment architecture that holds up for the online-serving case, drawn as the layers a request and a model version each travel. It is the model-as-a-service pattern, because that is the one that scales. Read it as two flows that meet at the serving layer: the request flow on the left, and the model-promotion flow coming down from the registry.

Walk the request flow. A call enters through an API gateway that handles auth and routing, hits a rollout router that applies the canary split, and lands on a model server (Triton or vLLM) that pulls online features from a feature store such as Feast or Tecton so serving features match the ones used at training. That last point is where a quiet class of bugs lives: training-serving skew, where the online feature differs subtly from the training-pipeline one and the model degrades for reasons no one can see in the code. A shared feature store is the structural fix.

Now the model flow. A new version lands in the model registry (MLflow is the common choice), which triggers CI to build the serving image, run the eval gate against the held-out set and the exported artifact, and only on a pass hand off to a guarded rollout that promotes through shadow and canary into the live router. Telemetry from the model server feeds a monitoring layer that watches latency, input drift, and prediction quality, and that monitoring closes the loop: a drift or quality alert schedules a retrain, which produces the next registry version, and the cycle repeats. This is the same loop that ml infrastructure is built to support, and the same loop that model monitoringsits

How to deploy a model to production, end to end

The architecture above implies a procedure. Here is the path we walk for an online model going live for the first time, in dependency order. Each step is a gate; we do not proceed until the previous one is green.

ML model deployment mistakes that cost the most

A handful of mistakes account for most of the production incidents we get called in on, and none of them are exotic. The most common is reaching for online serving when batch would do: a GPU endpoint, an autoscaler, and a monitoring stack for a model whose predictions are consumed once a day. The second is shipping a model change as a plain deploy with no canary, so the first signal that the new version is worse is a customer complaint. The third is no rollback path: the new model is the only model, the old image is gone, and getting back to a known-good state is a frantic rebuild instead of a flip.

Then there are the quiet ones. Training-serving skew, where serving features diverge from training features and the model degrades invisibly. Validating only the mean latency and getting paged on the p99 a week later. Baking weights into the image so every update is a full rebuild. Treating the exported artifact as identical to the eager model without re-running the eval. Each is cheap to prevent at design time and expensive to diagnose in production, which is the trade that makes deployment discipline worth the upfront cost.

Frequently asked questions