Big Data & Analytics AI & ML

ML Feature Engineering

Feature stores (Feast, Tecton, Hopsworks), online store vs offline store, point-in-time correct joins, feature pipelines, cross-team feature sharing, training-serving skew prevention, feature monitoring, and drift detection.

⏱ 12 min read

What it is

Feature engineering is the process of transforming raw data into numerical or categorical representations (features) that machine learning models can consume. At scale, managing features is not just a modelling problem — it is a data engineering and platform problem. A feature store is an infrastructure layer that centralises feature computation, storage, and serving for both model training (offline) and real-time model inference (online), ensuring that the same feature logic is used in both contexts, features are discoverable across teams, and features are served with low latency during inference.

Why it exists

Without a feature store, three common problems emerge: (1) training-serving skew — the Python code used to compute features during training differs from the Java/Go code used in the production scoring service, causing subtle differences in feature values and degraded model performance; (2) feature duplication — three different teams independently compute "user purchase count in the last 30 days" in three different pipelines, each with different logic and different bugs; and (3) data leakage — features joined without point-in-time correctness allow future information to leak into training data, producing models that perform well in training but fail in production.

Typical architecture


FEATURE STORE ARCHITECTURE
────────────────────────────
  Raw Data Sources (S3, Kafka, databases)
            ↓
    Feature Pipelines (Spark / Flink)
    - Compute feature values from raw data
    - Write to both offline and online stores
            ↓
  ┌───────────────────────────────────────┐
  │              Feature Store            │
  │                                       │
  │  Offline Store          Online Store  │
  │  (S3 Parquet,           (Redis /      │
  │   BigQuery, Hive)       DynamoDB /    │
  │   - Full history        Bigtable)     │
  │   - Point-in-time       - Latest      │
  │     correct reads         feature     │
  │   - Batch training        values per  │
  │     dataset generation    entity key  │
  │                         - P99 < 10ms  │
  └─────────────┬─────────────────────────┘
                │
  ┌─────────────┴──────────────────┐
  │             │                  │
  Training   Batch Scoring   Real-time Scoring
  Job        Pipeline        Service (ML API)
  (reads     (reads offline) (reads online store
   offline                    per request)
   with PITC
   joins)

ONLINE vs OFFLINE STORE
──────────────────────────
  Offline Store:
  Purpose:     Model training and batch inference
  Storage:     S3 Parquet, BigQuery, Hive, Delta Lake
  Access:      Bulk reads (millions of rows per training job)
  Latency:     Seconds to minutes acceptable
  History:     Full time-series (all historical feature values)
  Example:     user_features_2024-11-14.parquet

  Online Store:
  Purpose:     Real-time model inference (serving)
  Storage:     Redis, DynamoDB, Google Bigtable, Cassandra
  Access:      Single-key lookup per inference request
  Latency:     P99 < 10ms
  History:     Latest value only (or last N values)
  Example:     GET redis:user_features:{user_id}

  Materialisation: Feature pipeline writes to BOTH stores simultaneously
  (or offline store is synced to online store on a schedule)

POINT-IN-TIME CORRECT JOINS
─────────────────────────────
  Problem (data leakage):
  Training a fraud model. Label: was_fraudulent (set 2h after transaction)
  Feature: user_risk_score (computed daily, updated as events arrive)

  Naive join (WRONG - leaks future data):
  transactions JOIN user_features ON user_id
  → user_risk_score reflects risk computed AFTER the fraud was detected
  → model trains on information it won't have in production

  Point-in-time correct join (CORRECT):
  For each transaction at time T:
    Join to the feature value that existed AT time T
    (i.e., the latest feature value with timestamp ≤ T)

  SQL equivalent (AS OF JOIN):
  SELECT t.*, f.risk_score
  FROM transactions t
  ASOF JOIN user_features f
    ON t.user_id = f.user_id
    AND f.feature_ts <= t.transaction_ts  -- only past feature values

  Feature stores implementing PITC: Feast, Tecton, Hopsworks, Featureform
  Manual PITC in Spark (range join + window):
  from pyspark.sql.functions import last
  joined = transactions.join(
      features,
      on=["user_id"],
      how="left"
  ).filter(
      col("feature_ts") <= col("transaction_ts")
  ).groupBy("transaction_id").agg(
      last("risk_score").alias("risk_score")
  )

FEATURE STORE COMPARISON
──────────────────────────
  Feast (open-source, Google-donated):
  - Kubernetes-native; supports AWS, GCP, Azure
  - Feature registry (git-based), batch + streaming materialisation
  - Online: Redis, DynamoDB, Bigtable; Offline: S3, BigQuery, Snowflake
  - Best for teams wanting self-hosted, cloud-agnostic solution

  Tecton (commercial, ex-Uber Michelangelo):
  - Managed SaaS; enterprise feature platform
  - Strong streaming feature support (real-time aggregations)
  - Built-in monitoring, lineage, access control
  - Best for enterprise teams wanting minimal ops overhead

  Hopsworks (open-source + managed):
  - Full ML platform: feature store + model registry + deployment
  - HSFS Python API; supports Spark and Python transformation engines
  - Best for on-premise or AWS deployments needing integrated MLOps

TRAINING-SERVING SKEW PREVENTION
───────────────────────────────────
  Root cause: different code computes features in training vs serving

  Solution 1: Same feature definition for both (feature store's core value)
    Feast feature definition (Python):
    user_features = FeatureView(
      name="user_purchase_stats",
      entities=["user_id"],
      features=[
        Feature("purchase_count_30d", ValueType.INT64),
        Feature("total_spend_30d",   ValueType.DOUBLE),
      ],
      batch_source=S3Source(path="s3://features/user_stats"),
      online=True
    )
    → Feast uses this SAME definition to serve training data
      (offline) and inference data (online) — same logic, same result.

  Solution 2: Log actual feature values at inference time, join to labels
    Inference service logs {request_id, user_id, features, timestamp}
    to S3 → when labels arrive, join by request_id
    → Training on logged features (guaranteed same values as served)

FEATURE MONITORING AND DRIFT DETECTION
────────────────────────────────────────
  Monitor: mean, std, min, max, null rate, cardinality per feature per day
  Alert on: statistical drift (PSI > 0.2), sudden null rate spike,
            unexpected value range changes

  Population Stability Index (PSI):
  PSI < 0.1:  stable (no action)
  0.1–0.2:    moderate drift (investigate)
  > 0.2:      significant drift (retrain model or fix pipeline)

  Tools: Evidently AI (open-source), WhyLogs/whylabs, Fiddler, Arize

Pros and cons

Pros

  • Point-in-time correct joins eliminate data leakage, producing models that match offline evaluation performance in production.
  • Single feature definition for training and serving eliminates training-serving skew — the number one silent killer of ML model production performance.
  • Feature reuse across teams reduces duplication: a fraud team's 30-day purchase velocity feature can be shared with the recommendations team.
  • Low-latency online store (Redis) enables real-time feature serving for synchronous ML API calls without recomputing features per request.

Cons

  • Feature store adds infrastructure complexity: an offline store, online store (Redis), feature registry, and materialisation pipeline all require provisioning and maintenance.
  • Feature registry discipline requires organisational adoption: teams must register features, maintain definitions, and follow naming conventions — governance overhead.
  • Online store costs increase linearly with feature count × entity count × read QPS — at large scale (millions of users × hundreds of features) Redis costs can be substantial.
  • Point-in-time joins are computationally expensive at scale — large PITC training dataset generation can take hours in Spark.

Implementation notes

Feature naming conventions: Establish a naming convention early and enforce it via the feature registry. A useful pattern: {entity}_{source}_{aggregation}_{window} — e.g., user_transactions_count_30d, card_velocity_amount_sum_1h. Consistent names make features discoverable and reduce duplicates.

Streaming features vs batch features: Some features must be updated in near real-time (card transaction velocity in the last 1 hour for fraud detection). These require a streaming pipeline (Flink/Kafka Streams) writing to the online store on every event. Batch features (30-day purchase history) can be updated hourly or daily. Use streaming features sparingly — they add significant pipeline complexity compared to batch materialisation.

Common failure modes

  • Training-serving skew without realising it: A Python feature computed in a Jupyter notebook for training and a Scala feature computed in a production service produce subtly different values (different null handling, different time zones) — the model underperforms silently in production. Log and compare feature distributions at training vs serving time.
  • No point-in-time correctness on time-series features: Any feature that changes over time (e.g., credit score, account age, recent activity count) must use PITC joins in training; forgetting this for even one feature can leak future information and inflate model metrics by 5–20%.

Decision checklist

  • Are all time-varying features joined with point-in-time correct logic in training datasets?
  • Is the same feature definition (same code path) used for offline training and online serving?
  • Are feature values logged at inference time to enable actual-vs-predicted analysis?
  • Is feature drift monitored (PSI or equivalent statistical test) with alerts for significant changes?
  • Is there a feature registry where teams can discover and reuse existing features before building new ones?

Example use cases

  • Real-time fraud scoring: On each payment event → online store lookup for 50 user features (velocity counts, historical amounts) with P99 < 5ms → fraud model inference → decision returned in < 50ms total; training uses PITC joins over 90-day transaction history; feature drift monitored daily.
  • Recommendation engine: Batch Spark job nightly materialises 200 user and item features to offline store (S3 Parquet); training dataset generated with PITC joins over 6-month history; hot features (session-level) added as streaming features via Flink → Redis; shared feature definitions reused by search ranking team.

Further reading