ML System Design

What it is

ML system design is the discipline of architecting end-to-end machine learning platforms that reliably take data from raw ingestion through feature engineering, model training, deployment, and production monitoring. Unlike traditional software systems where the logic is explicitly programmed, ML systems are defined by the data they learn from — which means the architecture must account for the full lifecycle of data, models, and their interactions over time.

A production ML system comprises several interconnected subsystems: data pipelines that collect and transform raw data into a form suitable for learning; feature stores that compute, store, and serve derived signals; training infrastructure that orchestrates distributed compute; model registries that track versions and metadata; serving layers that expose predictions via APIs or batch jobs; and monitoring systems that detect when models degrade and trigger retraining workflows.

The hidden technical debt in ML systems is substantial. Research by Google showed that the actual ML code in a production system is a tiny fraction of the total codebase — surrounded by data collection, feature extraction, process management, serving infrastructure, monitoring, and configuration code. Designing these supporting systems correctly from the start is what separates reliable ML platforms from fragile research prototypes that can't survive real-world deployment.

Why it exists

The gap between a working notebook prototype and a robust production ML system is enormous. Data scientists can demonstrate impressive accuracy metrics in controlled experiments, but translating that into a system that runs continuously, serves millions of requests, handles data drift, and maintains accuracy over months requires deliberate architectural thinking. Without a designed system, teams end up with a graveyard of models that were successfully trained but never reliably deployed.

ML systems face unique challenges compared to traditional software. The behavior of the system is determined by training data, not just code — meaning a bad data batch can silently degrade model performance without any code change. Models become stale as the real world changes, requiring continuous monitoring and periodic retraining. Experiments must be reproducible, which demands tight versioning of data, code, hyperparameters, and environment. These requirements don't emerge naturally; they need architectural support.

Operationally, ML teams must move from ad hoc experimentation toward a discipline analogous to software engineering — with version control, automated testing, CI/CD, and observability. MLOps as a practice emerged precisely because teams learned the hard way that deploying a model once is straightforward, but keeping it healthy in production for years requires platform-level investment.

When to use

• Building a platform that will train and serve multiple ML models across multiple product teams.
• When model accuracy in production is business-critical and degradation has measurable user or revenue impact.
• Teams deploying more than a handful of models and needing standardized pipelines, not bespoke solutions per model.
• Organizations that need reproducibility and auditability — regulated industries, research institutions, or teams doing continuous experimentation.
• Systems where training data and serving data come from different pipelines, creating training-serving skew risks that must be managed structurally.

When NOT to use

• One-off analysis or batch reports — over-engineering a full ML platform for a single model that runs weekly is wasteful.
• Very early experimentation phases where the problem isn't yet defined; invest in platform after the value of ML is proven.
• When a simple rule-based or statistical system would suffice — not every prediction problem requires a neural network and a feature store.
• Teams without the operational maturity to maintain the infrastructure; a half-implemented ML platform is worse than no platform.

Typical architecture

 ┌─────────────────────────────────────────────────────────────────┐
 │                      DATA SOURCES                               │
 │   [Transactional DB]  [Event Stream]  [Data Lake]  [3rd Party]  │
 └──────────┬───────────────────┬──────────────────────────────────┘
            │                   │
            ▼                   ▼
 ┌──────────────────┐  ┌─────────────────────────────┐
 │  Batch Ingestion │  │   Stream Ingestion (Kafka)   │
 │  (Spark / Flink) │  │   Real-time feature compute  │
 └────────┬─────────┘  └──────────────┬──────────────┘
          │                           │
          ▼                           ▼
 ┌──────────────────────────────────────────────────┐
 │              FEATURE STORE                        │
 │  ┌─────────────────┐    ┌──────────────────────┐  │
 │  │  Offline Store  │    │   Online Store        │  │
 │  │  (S3 / Hive)    │    │   (Redis / DynamoDB)  │  │
 │  └────────┬────────┘    └───────────┬───────────┘  │
 └───────────┼─────────────────────────┼──────────────┘
             │                         │
             ▼                         │
 ┌───────────────────────┐             │
 │   TRAINING PIPELINE   │             │
 │  Data Split → Train   │             │
 │  Hyperparameter Tune  │             │
 │  Evaluation & Metrics │             │
 └──────────┬────────────┘             │
            │                          │
            ▼                          │
 ┌──────────────────────┐              │
 │   MODEL REGISTRY     │              │
 │  Versioning, Tags,   │              │
 │  Lineage, Artifacts  │              │
 └──────────┬───────────┘              │
            │                          │
            ▼                          ▼
 ┌───────────────────────────────────────────────┐
 │              SERVING LAYER                     │
 │   [Online API]   [Batch Job]   [Stream Scorer]  │
 └──────────────────────┬────────────────────────┘
                        │
                        ▼
 ┌───────────────────────────────────────────────┐
 │           MONITORING & OBSERVABILITY           │
 │  Data Drift · Prediction Drift · Latency       │
 │  Ground Truth Capture · Retraining Triggers    │
 └───────────────────────────────────────────────┘

Pros and cons

Implementation notes

Start with the serving and monitoring layers before building an elaborate training platform. Many teams make the mistake of investing heavily in AutoML and distributed training infrastructure before they understand how their models will be served and observed in production. Determine your latency and throughput requirements for inference first — this dictates the serving architecture, which in turn dictates what feature freshness is achievable, which constrains your pipeline design.

Adopt a metadata-first mindset. Every artifact — datasets, features, models, experiments — should be treated as a first-class versioned object with rich metadata: who created it, what data it was derived from, what metrics it achieved, where it is deployed. Tools like MLflow, Weights & Biases, or Vertex AI Experiments support this, but the discipline of recording metadata must be enforced culturally as well as technically. Without this, debugging production issues becomes archaeology.

Design for failure from the beginning. ML systems have many more failure modes than traditional software: data pipelines can silently produce corrupt data, model accuracy can degrade gradually rather than catastrophically, and serving infrastructure can experience latency spikes under load. Implement circuit breakers on prediction endpoints, fallback logic (serve a cached prediction or a rule-based fallback if the model is unavailable), and comprehensive canary deployment policies before any model reaches full production traffic.

Manage training-serving skew proactively. This is the single most common silent failure mode in ML production systems. It occurs when the features used to train a model are computed differently from the features computed at serving time. Using a feature store that serves both training and online prediction from a unified feature definition is the most reliable mitigation. Periodically compare training feature distributions against serving feature distributions as part of your monitoring regime.

Common failure modes

• Training-serving skew: Features are engineered differently in the training pipeline versus the serving pipeline, leading to models that perform well offline but poorly in production. Mitigate by using a shared feature store for both paths.
• Silent data quality degradation: An upstream data source changes its schema or data distribution without notifying the ML team, causing feature values to drift without raising alerts. Mitigate with data validation at pipeline ingestion boundaries.
• Feedback loop contamination: Model predictions feed back into the training data, creating a self-reinforcing bias loop. Common in recommendation and ranking systems. Design explicit feedback collection mechanisms that distinguish model predictions from ground truth.
• Model staleness: A model trained on historical data stops being accurate as the world changes but no retraining is triggered because monitoring thresholds are too loose. Implement both statistical drift detection and business-metric-based triggers for retraining.
• Experiment leakage: Failing to properly separate training, validation, and test sets — especially for time-series data where future data leaks into past windows — leads to overly optimistic offline metrics that don't translate to production performance.

Decision checklist

• Have you defined latency and throughput SLAs for model inference before designing the serving layer?
• Is there a feature store, or will training and serving compute features independently (creating skew risk)?
• Are all experiments, datasets, and model artifacts versioned with full lineage metadata?
• Is there a monitoring system that detects both data drift and prediction/output drift with alerting?
• Is there a defined retraining trigger policy — scheduled, drift-based, or performance-based?
• Do prediction endpoints have fallback logic for when the model is unavailable or returns unexpected outputs?
• Are model deployments gated behind canary or shadow deployment stages before full traffic rollout?

Example use cases

• E-commerce recommendation engine: User behavior events feed a stream processing pipeline that updates a feature store hourly; a daily training job produces a new ranking model; inference is served via a low-latency REST API with shadow testing to validate new models against live traffic.
• Fraud detection platform: Transaction features are computed in real time and written to an online feature store; a gradient-boosted model scores each transaction in under 50ms; monitoring tracks both feature distributions and the fraud-catch rate as ground truth labels arrive.
• Demand forecasting: Historical sales data and external signals (weather, holidays) are batch-processed; a time-series model is retrained weekly; predictions are written to a warehouse and surfaced in supply chain dashboards. Forecast accuracy is monitored against actuals after each cycle.

Related patterns

• Feature Stores
• Model Serving
• Model Monitoring
• Training Pipelines
• MLOps
• Data Pipeline Patterns

Further reading

• Hidden Technical Debt in Machine Learning Systems – Sculley et al., NeurIPS 2015
• MLOps: Continuous delivery and automation pipelines in machine learning – Google Cloud
• Designing Machine Learning Systems – Chip Huyen (O'Reilly)
• Continuous Delivery for Machine Learning – Martin Fowler