Model Serving

What it is

Model serving is the infrastructure and set of patterns used to expose trained ML models so that applications can request predictions. It is the production face of any ML system — the layer that connects data science work to real-world impact. Model serving encompasses synchronous online inference (returning a prediction in response to a request), asynchronous batch inference (scoring large datasets offline), and stream-based inference (scoring events in a continuous data stream).

Online serving exposes models as REST or gRPC endpoints. Requests arrive with feature vectors, the model computes a prediction, and the result is returned within a latency SLA — often single-digit milliseconds for simple models, or tens to hundreds of milliseconds for deep learning models. The serving infrastructure must handle request routing, load balancing, autoscaling, model version management, and health checking. Platforms like TensorFlow Serving, TorchServe, Triton Inference Server, and managed offerings like AWS SageMaker handle much of this plumbing.

Batch inference runs predictions over a large dataset — typically millions to billions of records — writing results to a data warehouse or file store for downstream consumption. This pattern is used when predictions don't need to be real-time: generating daily recommendation lists, pre-scoring all customers for a campaign, or applying a model to an entire document corpus. Batch inference can leverage CPU or GPU clusters at scale without the latency constraints of online serving.

Why it exists

Training a model is only half the work. A model sitting in a Jupyter notebook or a file on a researcher's laptop delivers no business value. Model serving bridges the gap between trained artifacts and production applications. Without it, every team integrating an ML model would need to write their own serving code, handle versioning, manage rollouts, and implement scaling — duplicating effort across the organization and creating a long tail of ad hoc, unmaintained serving solutions.

Production inference has non-trivial engineering requirements that differ from training. Latency percentiles (p99, p999) matter much more than throughput alone. Models must be versioned and replaced without downtime. Multiple model versions often need to run simultaneously for A/B testing or shadow deployments. Hardware accelerators (GPUs, TPUs, specialized inference chips) must be efficiently allocated and batched. These requirements justify a dedicated serving infrastructure rather than bespoke solutions per model.

Safe model deployment requires gradual rollout mechanisms. Unlike a traditional software deployment that can be rolled back in seconds, a model update may silently degrade accuracy in ways that aren't immediately apparent. Canary deployments — routing a small fraction of traffic to the new model while the majority continues using the old model — allow teams to compare prediction quality in production before committing to a full rollout. This requires serving infrastructure that supports traffic splitting and per-version observability.

When to use

• Models need to serve predictions in real time within strict latency budgets — use online serving with REST or gRPC endpoints.
• Large-scale predictions over historical or slowly-changing data sets where real-time latency is not required — use batch inference.
• Continuous event processing where predictions must be generated as data flows through a stream — use stream-based inference.
• Multiple versions of a model need to run simultaneously for A/B testing, gradual rollout, or shadow deployment evaluation.
• GPU or specialized accelerator hardware must be shared across multiple models to control infrastructure costs.

When NOT to use

• Ad hoc one-time batch scoring jobs that don't need to be operationalized — a simple script run manually may be sufficient.
• Very simple models (logistic regression, decision trees) that can be embedded directly in the application code without a dedicated serving layer — avoid operational overhead when the model is trivial.
• Contexts where prediction latency requirements are so tight (<1ms) that the network round trip to a serving endpoint is itself the bottleneck — consider embedding the model in the application process.

Typical architecture

 ┌─────────────────────────────────────────────────────────────┐
 │                     CLIENT APPLICATION                       │
 └──────────────────────────────┬──────────────────────────────┘
                                │ REST / gRPC request
                                ▼
 ┌─────────────────────────────────────────────────────────────┐
 │                  API GATEWAY / LOAD BALANCER                  │
 │           Rate limiting · Auth · Traffic splitting            │
 └─────────────┬──────────────────────────┬────────────────────┘
               │ 95% traffic              │ 5% canary traffic
               ▼                          ▼
 ┌─────────────────────┐      ┌─────────────────────────────┐
 │  MODEL SERVER v1.2  │      │    MODEL SERVER v1.3 (NEW)   │
 │  ┌───────────────┐  │      │  ┌───────────────────────┐  │
 │  │ Pre-processing│  │      │  │   Pre-processing       │  │
 │  │ Feature fetch │  │      │  │   Feature fetch        │  │
 │  │ Inference     │  │      │  │   Inference (GPU)      │  │
 │  │ Post-process  │  │      │  │   Post-process         │  │
 │  └───────────────┘  │      │  └───────────────────────┘  │
 └──────────┬──────────┘      └──────────────┬──────────────┘
            │                                │
            ▼                                ▼
 ┌──────────────────────────────────────────────────────────┐
 │                 PREDICTION STORE / CACHE                  │
 │      (Optional: cache predictions for hot entities)       │
 └──────────────────────────────────────────────────────────┘
            │
            ▼
 ┌──────────────────────────────────────────────────────────┐
 │               OBSERVABILITY PIPELINE                      │
 │  Request logs · Prediction distribution · Latency (p99)  │
 │  Shadow comparison · Ground truth capture                 │
 └──────────────────────────────────────────────────────────┘

 BATCH INFERENCE PATH:
 ┌──────────┐    ┌──────────────┐    ┌────────────┐    ┌──────────┐
 │ Scheduler│───▶│ Feature Prep │───▶│ Batch Score│───▶│  Output  │
 │ (Airflow)│    │ (Spark job)  │    │ (GPU cluster│    │  (S3/DW) │
 └──────────┘    └──────────────┘    └────────────┘    └──────────┘

Pros and cons

Implementation notes

Measure and optimize inference latency from the client's perspective, not just the model's compute time. End-to-end inference latency includes network I/O, feature retrieval from a feature store or cache, pre/post-processing steps, and model compute. Profile each step separately. Often the majority of inference latency is in feature retrieval, not in model compute — optimize accordingly. For deep learning models, batching multiple requests together (dynamic batching) significantly improves GPU utilization without proportionally increasing latency.

Implement shadow deployment as a first-class citizen of your serving architecture. In a shadow deployment, the new model version receives a copy of every request that the production model receives, but its predictions are logged rather than returned to the user. This allows you to compare the production model's predictions against the challenger model's predictions on real traffic — without any user-facing risk — before deciding to promote the challenger. Tools like Seldon, BentoML, and AWS SageMaker support this pattern natively.

Design your model API contracts defensively. Treat the input schema as a formal contract: version it, validate every incoming request against it, and emit clear errors for schema violations rather than silently returning garbage predictions. Similarly, define an output schema and include a confidence score or calibration signal alongside every prediction. Downstream consumers of predictions should never be surprised by a model returning an unexpected output format after an update.

Common failure modes

• Cold start latency: Container-based model servers with long initialization times (loading a large neural network from disk) cause severe tail latency for the first requests after a scale-out event. Mitigate with pre-warming strategies and keep-alive minimum replicas.
• GPU memory fragmentation: When multiple models are hosted on a shared GPU, memory fragmentation can prevent loading a new model even when nominal free memory exists. Use explicit memory allocation policies and model multiplexing frameworks designed for GPU sharing.
• Runaway batch jobs: Batch inference jobs that don't have circuit breakers or row-count limits can consume entire clusters for days if fed unexpectedly large datasets. Always partition batch jobs and implement progress checkpointing.
• Silent prediction degradation after rollout: A model update changes the prediction distribution in subtle ways that pass A/B tests but cause downstream issues (e.g., the ranking changes so steeply that user experience degrades). Monitor prediction distribution stability, not just aggregate accuracy, during and after rollouts.

Decision checklist

• What is the latency SLA for online predictions (p50, p99, p999)?
• Is GPU hardware required for inference, and how will it be provisioned and shared?
• What is the traffic splitting mechanism for canary deployments and A/B tests?
• How will prediction inputs and outputs be logged for monitoring and ground truth collection?
• What is the rollback procedure if a newly deployed model degrades production metrics?
• Is dynamic batching configured to maximize GPU utilization without violating latency budgets?

Example use cases

• Search ranking API: A deep learning ranking model is served via gRPC behind a load balancer; incoming queries retrieve candidate feature vectors from a feature store, the model scores candidates within 20ms p99, and results are returned ranked.
• Daily product recommendations: A batch inference job runs nightly, scoring every active user against the recommendation model using yesterday's behavioral features; results are written to a DynamoDB table used by the website as a pre-computed recommendations cache.
• Fraud detection stream scorer: Transactions are published to a Kafka topic; a Flink job enriches each event with entity features from an online store, scores it against a gradient-boosted model, and publishes the fraud score to a downstream topic within 50ms.

Related patterns

• Feature Stores
• Model Monitoring
• MLOps
• LLM Architecture
• Deployment Patterns

Further reading

• NVIDIA Triton Inference Server Documentation
• Deploy Models for Inference – Amazon SageMaker
• TensorFlow Serving Guide
• How to Deploy Machine Learning Models – Christopher Samiullah