Big Data & Analytics

Apache Spark Architecture

How Spark works internally: Driver, DAG Scheduler, Task Scheduler, executor model, RDD lineage, DataFrame/Dataset vs RDD, Catalyst query optimizer, Tungsten execution engine, wide vs narrow transformations, shuffle cost, and Spark on Kubernetes vs YARN.

⏱ 12 min read

What it is

Apache Spark is a unified analytics engine for large-scale data processing. It abstracts distributed computation behind a functional API (transformations and actions), compiles user code into a directed acyclic graph (DAG) of tasks, and executes those tasks in parallel across a cluster. Spark supports batch processing, interactive SQL (Spark SQL), streaming (Structured Streaming), ML (MLlib), and graph processing (GraphX) within a single framework, using in-memory processing to achieve 10–100x faster performance than Hadoop MapReduce for iterative algorithms.

Why it exists

Hadoop MapReduce writes intermediate results to HDFS after every map and reduce step, making it slow for iterative algorithms (ML training, graph computation) and interactive queries. Spark keeps intermediate data in memory across stages, dramatically reducing I/O for multi-pass workloads. It also provides a richer API — where MapReduce exposes only map and reduce, Spark provides join, groupByKey, window functions, UDFs, and DataFrames with SQL semantics — making complex ETL and analytics much simpler to express.

When to use

Large-scale ETL and data transformation (Bronze → Silver → Gold in Medallion architecture).
Distributed ML training (MLlib, or Spark as a data preparation layer before training in PyTorch/TensorFlow).
Complex aggregation queries over data lake (S3/ADLS Parquet) that require join, window, and aggregation at multi-TB scale.
Batch reprocessing of historical data in Kappa or Lambda architectures.

Typical architecture


SPARK CLUSTER ARCHITECTURE
────────────────────────────
  ┌─────────────────────────────────────────┐
  │              DRIVER (JVM process)       │
  │  SparkContext → DAG Scheduler           │
  │             → Task Scheduler           │
  │  SparkSession (Spark SQL entry point)   │
  └──────────────┬──────────────────────────┘
                 │ (sends tasks)
         Cluster Manager
      (YARN / Kubernetes / Standalone)
         /         |          \
  Executor 1   Executor 2   Executor N
  (JVM)        (JVM)        (JVM)
  - Task threads (cores)
  - In-memory storage (RDD cache, broadcast)
  - Shuffle write/read (disk)

EXECUTION FLOW
───────────────
  User code calls .count() or .write() (an Action)
       ↓
  Driver builds Logical Plan (DataFrame transformations)
       ↓
  Catalyst Optimizer optimises logical plan
  (predicate pushdown, column pruning, join reordering)
       ↓
  Physical Plan generated (merge sort join vs hash join)
       ↓
  Tungsten: code-generation for inner loops (JVM bytecode)
       ↓
  DAG Scheduler: splits plan at shuffle boundaries → Stages
       ↓
  Task Scheduler: dispatches Tasks to Executors
       ↓
  Executors run Tasks, read partitions, write shuffle files
       ↓
  Reduce stage reads shuffle files, produces result

RDD vs DATAFRAME vs DATASET
─────────────────────────────
  RDD (Resilient Distributed Dataset):
  - Low-level; distributed collection of JVM objects
  - Catalyst cannot optimise (opaque lambda functions)
  - Use when you need custom serialization or unsupported operations

  DataFrame (untyped, schema-on-read):
  - Distributed collection of Row objects; schema known at runtime
  - Catalyst can fully optimise: predicate pushdown, join reordering
  - Python/R users work primarily with DataFrames
  - 90%+ of Spark usage

  Dataset (typed, schema at compile time):
  - Type-safe DataFrame (Scala/Java only); compile-time errors
  - Same Catalyst optimisation as DataFrame
  - Preferred in strongly-typed Scala pipelines

NARROW vs WIDE TRANSFORMATIONS
────────────────────────────────
  Narrow: each input partition contributes to exactly one output partition
    map(), filter(), flatMap(), union()
    → No shuffle; fast; pipelined within one Stage

  Wide: each input partition may contribute to multiple output partitions
    groupByKey(), reduceByKey(), join(), repartition(), distinct()
    → Requires shuffle: write shuffle files to disk, transfer across network
    → Stage boundary; expensive; primary cause of Spark slowdowns

  SHUFFLE COST REDUCTION:
  - Use reduceByKey() instead of groupByKey() (pre-aggregates locally first)
  - Use broadcast joins when one side fits in executor memory (default < 10MB; tune spark.sql.autoBroadcastJoinThreshold)
  - Partition by join key before iterative joins: .repartition("userId")

SPARK ON KUBERNETES vs YARN
─────────────────────────────
  YARN:
  + Mature, well-tested in Hadoop ecosystem
  + Good resource isolation per queue
  - YARN cluster must be pre-provisioned
  - Poor cloud-native story (tied to HDFS cluster)

  Kubernetes:
  + Cloud-native; runs on EKS, GKE, AKS, or on-prem K8s
  + Dynamic resource provisioning (driver + executor pods on demand)
  + Better multi-tenancy via namespaces and resource quotas
  + Simpler on managed cloud: EMR on EKS, Dataproc on GKE
  - Slower pod startup vs YARN container (5–30s cold start per executor)
  - More complex networking and storage configuration

  Modern recommendation: Kubernetes for new cloud-native deployments;
  YARN only when already on Hadoop or using Cloudera/Hortonworks.

Pros and cons

Pros

Unified engine for batch, streaming, SQL, and ML — reduces the number of processing systems in the stack.
Catalyst optimizer and Tungsten code generation deliver performance competitive with hand-tuned C++ for many workloads.
Massive ecosystem: Delta Lake, Iceberg, Hudi, MLflow, Koalas (pandas-on-Spark); most data engineering tools integrate natively.
In-memory caching of hot datasets dramatically speeds up iterative workloads compared to MapReduce.

Cons

JVM overhead: memory management is complex — GC pauses, OOM errors with large executor heaps, serialization cost.
Shuffle performance is a major pain point: wide transformations cause network-intensive data exchange that can dominate job time.
Overhead for small jobs: Spark's driver startup, task scheduling overhead, and executor launch time make it a poor choice for sub-minute jobs.
Structured Streaming's micro-batch model adds latency compared to Flink's true streaming model for low-latency use cases.

Implementation notes

Partition sizing: Spark's parallelism is determined by the number of partitions. Too few partitions underutilises the cluster; too many creates scheduling overhead. Target 2–4 partitions per CPU core, with each partition 128–512 MB on disk. After a wide transformation (join, groupBy), partition count defaults to spark.sql.shuffle.partitions (default 200) regardless of data size — tune this value to the actual parallelism needed.

Data skew: When one key has significantly more records than others, the task handling that partition becomes the bottleneck, stalling the entire stage. Solutions: (1) salting — add a random suffix to the skewed key, then aggregate in two passes; (2) AQE (Adaptive Query Execution) in Spark 3.x automatically detects and coalesces skewed partitions at runtime.

Common failure modes

OOM on executors: Caused by large shuffle state, uncached RDDs being recomputed, or collecting too much data to the driver — profile memory usage with the Spark UI before tuning heap and memory fractions.
Default shuffle.partitions=200 on large joins: With 10TB of data, 200 partitions means each partition is 50 GB — tasks will OOM; set spark.sql.shuffle.partitions proportional to cluster size and data volume.

Decision checklist

Are DataFrames used instead of RDDs (unless there is a specific reason for RDDs)?
Is spark.sql.shuffle.partitions tuned for the actual data volume and cluster size?
Are broadcast joins configured for small tables to avoid expensive sort-merge shuffles?
Is Adaptive Query Execution (AQE) enabled (Spark 3.x default: true) for automatic skew handling?
Is the Spark UI reviewed for skewed stages and excessive GC time?

Example use cases

Data lake ETL (Medallion): S3 Bronze (raw JSON) → Spark job cleans, validates, writes Silver (Parquet, partitioned by date) → Spark SQL aggregates Silver to Gold (fact tables for BI tools); Delta Lake for ACID transactions and schema evolution.
Feature engineering at scale: 500 million user events per day → Spark computes 50 user-level features (30-day rolling counts, categorical aggregations) → writes to feature store offline store (S3 Parquet, partitioned by date) for ML training.

Lambda Architecture — Spark serves as both the batch layer and (via Structured Streaming) the speed layer.
Data Pipeline Patterns — Medallion architecture, incremental processing, and orchestration patterns for Spark pipelines.
ML Feature Engineering — Spark is the standard offline feature computation engine for large-scale feature stores.