IoT Data Pipeline

What it is

An IoT data pipeline ingests the continuous stream of telemetry from device fleets, routes it through enrichment and validation stages, stores it in appropriate stores for different query patterns, and enables real-time analysis and alerting. The challenge is scale: a fleet of 100,000 devices each sending one reading per second produces 100,000 messages per second — 8.64 billion messages per day — requiring distributed ingestion, efficient time-series storage, and stream processing that can detect conditions worth acting on without scanning every record.

Why it exists

IoT telemetry has a distinctive data profile: it is time-ordered, high-volume, high-frequency, and primarily queried by time range and device identity. Relational databases (OLTP) perform poorly at this workload — a table with 8 billion rows requires aggressive partitioning and indexing to serve real-time queries. Time-series databases (TSDBs) are purpose-built for this pattern: they compress repeated device IDs and timestamps efficiently, enable sub-second range queries, and support time-series specific operations (downsampling, gap filling, rate-of-change calculations).

When to use

• Any IoT system generating continuous sensor telemetry (temperature, pressure, vibration, power, location).
• Operational monitoring dashboards requiring real-time device status views.
• Predictive maintenance: anomaly detection on sensor streams to predict equipment failure.
• Energy management: aggregating power consumption data across large building or vehicle fleets.

Typical architecture

REFERENCE PIPELINE (AWS)
─────────────────────────
  Devices → MQTT → AWS IoT Core
                      │
              IoT Core Rules Engine
             /                    \
    Kinesis Data Streams          S3 (raw archive)
            │                      │
    Kinesis Data Analytics    Glue ETL → Parquet
    (Flink SQL / Apache Flink)         │
            │                     Athena (ad-hoc)
    ┌───────┴──────────┐
    │                  │
  Timestream          DynamoDB
  (recent 90d,        (device latest
   range queries)      state, fast lookup)
            │
      Grafana / QuickSight dashboards
            │
      SNS (alert on anomaly)

HOT PATH VS COLD PATH
──────────────────────
  Hot path (real-time):
  - Latency: sub-second to seconds
  - Answers: "is motor X currently vibrating above threshold?"
  - Storage: time-series DB (Timestream, InfluxDB, TimescaleDB)
  - Retention: 30–90 days of full-resolution data
  - Query pattern: SELECT last 1h WHERE deviceId = 'motor-42' AND metric = 'vibration'

  Cold path (historical / analytical):
  - Latency: seconds to minutes acceptable
  - Answers: "show me average daily power consumption for all sites in Q3"
  - Storage: S3 Parquet (columnar, compressed)
  - Retention: unlimited (object storage is cheap)
  - Query pattern: Athena SQL / Spark against S3 Parquet

TIME-SERIES DATABASE COMPARISON
─────────────────────────────────
  InfluxDB 3.0 (OSS, cloud):
  - Columnar storage (Apache Arrow/Parquet under the hood)
  - InfluxQL + Flux query language
  - Best-in-class compression for float series
  - Kubernetes-native clustering in v3

  TimescaleDB (PostgreSQL extension):
  - Familiar SQL; hypertables auto-partition by time
  - Continuous aggregates: materialized downsampled views
  - Full PostgreSQL ecosystem (PostGIS for spatial, rich indexing)
  - Best choice if team is already PostgreSQL-native

  Amazon Timestream:
  - Fully managed; no ops overhead
  - Magnetic tier (cold) + memory tier (hot) automatically tiered
  - Deep AWS integration: IoT Core, Kinesis, Grafana
  - Priced per written record + per scanned byte

  Azure Data Explorer (ADX):
  - Kusto Query Language (KQL); highly expressive time-series functions
  - Free-tier for IoT Hub integration
  - Best for telemetry + log analytics combined

STREAM PROCESSING
──────────────────
  Apache Flink (preferred for complex event processing):
  - Stateful stream processing; exactly-once semantics
  - Event time processing with watermarks (handles late-arriving data)
  - Use cases: rolling averages, anomaly detection, joins between streams
  - Managed: Kinesis Data Analytics for Apache Flink, Confluent Cloud

  Apache Spark Streaming:
  - Micro-batch (DStream) or continuous (Structured Streaming)
  - Better for batch-oriented pipelines that need some streaming
  - Larger ecosystem; easier to find Spark expertise

  Flink example — vibration anomaly (pseudo-SQL):
  SELECT deviceId, AVG(vibration) OVER (PARTITION BY deviceId
         ORDER BY event_time RANGE INTERVAL '5' MINUTE PRECEDING)
  FROM telemetry_stream
  WHERE avg_vibration > threshold

EDGE PRE-AGGREGATION
─────────────────────
  Problem: 1000 sensors each at 100Hz = 100k messages/sec = expensive ingestion
  Solution: aggregate at gateway before sending to cloud

  Edge aggregation patterns:
  Downsample:   Send 1-second avg instead of 100 raw readings
                100k msg/sec → 1k msg/sec (100x reduction)
  Threshold:    Only send if value changes by > N% (event-driven)
  Summary:      Send min/max/avg/count per 5-min window
  Edge ML:      Classify locally; send classification + confidence, not raw data

  Implementation: MQTT + Greengrass Stream Manager, Azure IoT Edge ASA module

Pros and cons

Implementation notes

Partitioning strategy: In time-series databases and Parquet on S3, partitioning strategy is the primary determinant of query performance. Partition by time first (day or hour), then by device group or sensor type. This allows queries for "all readings from today" to scan only today's partition. On S3, use a prefix structure like s3://bucket/year=2024/month=11/day=15/deviceType=thermostat/ to enable partition pruning in Athena and Spark.

Late-arriving data: IoT devices may buffer telemetry locally and send it in bulk when connectivity is restored — data may arrive 30 minutes to 24 hours late. Stream processors must use event time (the timestamp the device generated the reading) not processing time (when the message arrived at the broker). Configure Flink watermarks to allow late data within a tolerance window; Timestream and TimescaleDB both support inserting out-of-order data with past timestamps.

Common failure modes

• Using a relational DB for time-series: Storing telemetry in a standard PostgreSQL or MySQL table without time-series specific optimisations leads to table bloat, slow range queries, and expensive storage at even moderate scale.
• No data retention policy: Storing every reading at full resolution indefinitely; implement tiered retention: full resolution for 30 days, 1-minute aggregates for 1 year, hourly aggregates forever.
• Processing time instead of event time: Using the Kinesis ingestion timestamp instead of the device-generated timestamp causes incorrect time windows and missing data for devices that send late.

Decision checklist

• Is telemetry stored in a purpose-built time-series database rather than a relational DB?
• Is there a hot path (real-time, TSDB) and a cold path (historical, Parquet/S3)?
• Is stream processing using event time with watermarks to handle late-arriving data?
• Is there a data retention/tiering policy (full res → downsampled → cold storage)?
• Is edge pre-aggregation used to reduce ingestion volume and bandwidth?

Example use cases

• Factory predictive maintenance: 5,000 machines × 20 sensors × 1Hz = 100k msg/sec; gateway aggregates to 10-second windows (10x reduction); Kinesis → Flink detects rolling-average vibration anomalies → SNS alert → work order in CMMS; raw data archived to S3 Parquet; Timestream for 60-day rolling operations dashboard.
• Smart grid energy monitoring: 500k smart meters × 15-min readings = ~550 msg/sec; IoT Core rules → Timestream (daily usage, real-time billing); S3 Parquet (long-term, ML features for demand forecasting); Athena for monthly regulatory reporting.

Related patterns

• IoT Architecture — MQTT ingestion and IoT Core rules engine that feed this pipeline.
• Edge Computing — Edge pre-aggregation reduces the volume entering this pipeline.
• Lambda Architecture — IoT hot/cold path is an instance of the Lambda architecture pattern.
• Data Architecture — Time-series storage, data lakes, and stream processing patterns.

Further reading

• Amazon Timestream — Developer Guide
• InfluxDB 3.0 Documentation
• Apache Flink Documentation