IoT & Edge

Edge Computing

Bringing compute closer to where data is generated: edge categories, AWS Greengrass and Azure IoT Edge runtimes, on-device ML inference, edge-cloud synchronisation patterns, and orchestration challenges.

⏱ 11 min read

What it is

Edge computing moves data processing and application logic from a centralised cloud data centre to locations physically closer to where data is generated — the device itself, a local gateway, or a nearby micro data centre. By reducing the physical distance data must travel, edge computing lowers latency, reduces bandwidth consumption (by processing locally rather than sending raw data to the cloud), and enables operation during cloud connectivity loss.

Edge computing is not binary; it exists on a continuum from device edge (on the microcontroller itself) to near edge (a gateway in the same building) to far edge (a regional micro data centre or telco MEC point-of-presence a few milliseconds away) to the public cloud. Real architectures typically use several of these tiers for different classes of processing.

Why it exists

Three forces drive edge computing adoption. First, latency: a round trip to a cloud region takes 30–150ms; an autonomous vehicle, a robotic arm, or a real-time safety system cannot tolerate that. Second, bandwidth: a factory floor with 500 cameras each generating 4K video cannot economically send all raw video to the cloud — local processing must reduce data to events or anomaly clips. Third, connectivity: devices in remote locations (oil rigs, ships, mines) may have intermittent or no cloud connectivity; they need to operate autonomously and sync when connectivity resumes.

When to use

  • Latency requirements under 100ms that cannot be met by a cloud round trip.
  • Bandwidth constraints: video analysis, audio processing, or high-frequency sensor streams where sending raw data to cloud is cost-prohibitive.
  • Intermittent connectivity: systems that must operate during cloud outages.
  • Data residency requirements: regulations prohibiting certain data from leaving a facility or jurisdiction.
  • On-device ML inference: running TFLite or ONNX models where sub-100ms latency is required and a cloud API call is too slow.

Typical architecture


EDGE TIER CATEGORIES
──────────────────────
  Device edge:    On the MCU/MPU itself
                  Latency: sub-millisecond
                  Compute: very limited (MCU: 256KB RAM, Cortex-M)
                  Use: real-time control loops, sensor fusion, wake-word detection

  Near edge:      Local gateway / industrial PC / Raspberry Pi
                  Latency: < 5ms LAN round trip
                  Compute: moderate (4-16 cores, 8-32GB RAM, optional GPU/NPU)
                  Use: protocol translation, local analytics, edge ML inference

  Far edge:       Telco MEC, mini data centre, colocation at facility
                  Latency: 5-30ms
                  Compute: rack-scale, Kubernetes available
                  Use: video analytics at scale, multi-site aggregation

  Cloud:          AWS / Azure / GCP region
                  Latency: 30-150ms
                  Compute: unlimited (horizontal scale)
                  Use: ML training, long-term storage, global analytics

AWS GREENGRASS V2
──────────────────
  - Software runtime deployed on edge devices (Linux, rPi, industrial PCs)
  - Components: IoT core client, Lambda-on-device, stream manager, ML inference
  - Nucleus: core runtime managing component lifecycle
  - Component deployment via IoT Core Fleet Hub
  - Local pub/sub: inter-component messaging without cloud round trip
  - Stream Manager: local data buffering → S3 export when connected
  - Greengrass ML inference: run SageMaker Neo-compiled models locally

AZURE IOT EDGE
───────────────
  - Containerized runtime (Docker); modules = Docker containers
  - IoT Edge daemon: manages module lifecycle
  - Edge Hub: local message broker (MQTT/AMQP); routes between modules
  - IoT Edge Agent: reports module status to IoT Hub
  - Azure Stream Analytics module: run SQL-like queries on edge data stream
  - Azure ML module: run ONNX models at the edge
  - Deployment manifests: specify modules + routes + desired properties
  - Twin-based configuration: IoT Hub twin synchronises module config

EDGE ML INFERENCE
──────────────────
  Problem: cloud ML inference adds 100-500ms latency + cost per call
  Solution: deploy compressed model to edge device

  TensorFlow Lite (TFLite):
  - Quantised models (INT8): 4x smaller, 2-4x faster on ARM
  - Runs on Cortex-A (rPi), Coral Edge TPU, Android NPU
  - Delegates: GPU delegate, NNAPI delegate for hardware acceleration

  ONNX Runtime:
  - Cross-framework (PyTorch, TF, sklearn → ONNX format)
  - Execution providers: CPU, CUDA, TensorRT, OpenVINO, DirectML
  - Suitable for near-edge / far-edge (more compute available)

  Workflow: train in cloud → export to TFLite/ONNX → deploy via Greengrass/IoT Edge
            → inference on device → send only anomalies/results to cloud

EDGE-CLOUD SYNC PATTERNS
──────────────────────────
  Store-and-forward: buffer messages locally during cloud outage; drain on reconnect
  Delta sync: send only changed values since last sync (reduce bandwidth)
  Aggregate-and-send: compute min/max/avg at edge; send summaries not raw data
  Event-only: only send to cloud when threshold crossed (not all readings)
  Bidirectional sync: cloud pushes model updates / config; edge pushes telemetry/results

Pros and cons

Pros

  • Sub-millisecond to single-digit millisecond latency for control loops and safety-critical decisions.
  • Dramatic bandwidth reduction by filtering and aggregating data before transmission to cloud.
  • Continued operation during cloud connectivity loss — critical for industrial and safety systems.
  • Data sovereignty compliance — sensitive data can be processed locally without leaving a facility or jurisdiction.

Cons

  • Operational complexity: updating software on thousands of physically distributed edge devices is significantly harder than deploying to a cloud service.
  • Hardware heterogeneity: edge devices vary widely in CPU architecture, OS, available RAM, and accelerators — testing on all targets is challenging.
  • Security surface: edge devices are physically accessible and may be in unsecured locations, increasing physical attack vectors.
  • ML model lifecycle: retraining models and deploying updated versions to edge devices requires a robust MLOps pipeline for edge.

Implementation notes

Deployment strategy: Use staged rollouts for edge component deployments — deploy to 1% of fleet, monitor health metrics (error rate, latency, resource usage) for 24 hours, then expand to 10%, 50%, 100%. Greengrass and Azure IoT Edge both support deployment groups with rollback triggers. A bad deployment to all devices simultaneously can be catastrophic and may require physical access to recover constrained hardware.

Edge ML model compression: Full-precision (FP32) models trained in the cloud are typically too large for edge inference. Apply post-training quantisation (INT8) to reduce model size 4x and improve inference latency 2–4x on ARM CPUs. For microcontrollers (device edge), use TensorFlow Lite for Microcontrollers (TFLM) — models must fit in tens of kilobytes. Use knowledge distillation to train a small "student" model that approximates the large "teacher" model's behaviour.

Common failure modes

  • No offline operating mode: Systems designed as "cloud-first" with edge as an afterthought fail completely during cloud outages rather than degrading gracefully; offline operation must be a first-class design requirement.
  • Edge drift: Without automated deployment, edge devices run different software versions over time; configuration management and fleet-wide deployment tooling are essential from day one.
  • Resource exhaustion: Edge devices have limited RAM; containerised workloads that run fine in dev environments (with abundant memory) can OOM-kill on a 2GB gateway device; set resource limits on all containers.

Decision checklist

  • Is the edge/cloud processing split defined by concrete latency, bandwidth, or connectivity requirements?
  • Does the system have an offline operating mode that degrades gracefully during cloud connectivity loss?
  • Is edge software deployed via a managed runtime (Greengrass / Azure IoT Edge) with staged rollout and rollback?
  • Are resource limits (CPU, memory) configured for all edge containers?
  • Is edge ML model compression (quantisation, distillation) applied to meet device hardware constraints?

Example use cases

  • Manufacturing quality control: Camera at conveyor line → edge GPU runs defect detection model (ONNX + TensorRT) → sub-50ms detection → reject actuator triggered locally; only defect images sent to cloud for model retraining.
  • Remote oil field: SCADA sensors → Greengrass gateway with 4G modem → Greengrass Stream Manager buffers 48h of data locally → syncs to S3 during connectivity windows; anomaly detection component runs locally and triggers local alarm without cloud dependency.
  • IoT Architecture — Foundation: device/gateway/cloud tiers that edge computing extends.
  • Device Management — Fleet management, OTA firmware, and provisioning for edge devices at scale.
  • IoT Data Pipeline — Ingestion and processing pipeline for telemetry flowing from edge to cloud.

Further reading