Integration Data Engineering

Batch Integration

Scheduled bulk data movement using batch processing frameworks, windowed processing, checkpoint/restart patterns, and the tradeoffs between batch and streaming architectures.

⏱ 10 min read

What it is

Batch integration is the movement or transformation of data in discrete, scheduled bulk runs rather than record-by-record in real time. A batch job collects a defined window of data (all orders placed in the last 24 hours, all transactions since the last run), processes the entire dataset in sequence, and writes the results to a destination. The window is defined by time (nightly run), volume (every 10,000 records), or event (triggered when an upstream file lands).

Core batch processing frameworks include Spring Batch (Java, the dominant enterprise batch framework with built-in chunk-oriented processing, job repository, and restart/retry), Apache Spark (distributed batch processing for large-scale data transformation), AWS Glue (managed Spark on AWS with a visual ETL designer and serverless execution), Azure Data Factory (managed pipeline orchestration with 90+ connectors), and traditional cron-scheduled SQL scripts. Microbatching — running batch jobs on very short intervals (every 5–60 seconds) — bridges the gap between true batch and streaming.

Why it exists

Many data workloads are inherently batch in nature: payroll runs once a week, regulatory reports are submitted monthly, machine learning model retraining happens nightly on accumulated data. Real-time streaming architectures add significant complexity — stateful stream processing, exactly-once semantics, watermark management — that is unnecessary when the business requirement is satisfied by processing data once per hour or once per day. Batch is also significantly more efficient for certain operations: bulk SQL inserts, sorted aggregations, and columnar file writes benefit from processing large data sets in memory rather than individual records arriving one at a time.

Batch integration also plays a critical role in data reconciliation and audit: nightly batch reconciliation between an OLTP database and a data warehouse detects discrepancies that event-driven real-time replication might miss due to reordering or partial failures. The determinism of a batch window — "process exactly the records created between 00:00 and 23:59 UTC on 2026-01-15" — simplifies reasoning about data completeness.

When to use

  • Nightly data warehouse ETL/ELT loads where sub-hourly freshness is not required and bulk columnar writes are most efficient.
  • Regulatory and compliance reports that are submitted periodically (daily, monthly) and must be reproducible for a specific time window.
  • ML feature engineering pipelines that generate training datasets by scanning and transforming large historical datasets.
  • End-of-day financial settlement, ledger reconciliation, or billing runs that must process the complete closed period as an atomic operation.
  • Bulk data migrations between databases, file formats, or storage tiers where the entire dataset must be scanned and transformed.
  • Situations where the destination system has a maintenance window or bulk-load API that makes batching significantly more efficient than row-by-row streaming.

When not to use

  • When business requirements demand low-latency data freshness (under 1 minute) — streaming integration is the appropriate choice.
  • For event-driven triggers where the action must occur immediately after an event (a payment should trigger a notification instantly, not at the next batch run).
  • When data volumes are small and constant — the overhead of scheduling and managing a batch job is not justified for a 100-row/day dataset; a simple API call or CDC stream is more appropriate.
  • When failure recovery is extremely time-sensitive — a failed batch job processing 8 hours of data requires a restart and re-run that may take hours to complete.

Typical architecture


  Spring Batch / AWS Glue Batch Pipeline:

  Trigger           Job Orchestration         Processing Steps
  ┌──────────┐     ┌──────────────────────┐  ┌───────────────────┐
  │  Cron /  │────▶│  Job Scheduler       │  │  Step 1: Extract  │
  │  S3 Event│     │  (Airflow / EventBridge│─▶│  - Read source DB │
  │  /EventBridge   │  / Spring Batch)   │  │  - Read S3 files  │
  └──────────┘     └──────────────────────┘  └────────┬──────────┘
                            │                         │
                   ┌────────▼───────┐       ┌────────▼──────────┐
                   │  Job Repository│       │  Step 2: Transform│
                   │  (tracks state)│       │  - Validate rows  │
                   │  - step status │       │  - Enrich / join  │
                   │  - checkpoint  │       │  - Aggregate      │
                   │  - retry count │       └────────┬──────────┘
                   └────────────────┘                │
                                            ┌────────▼──────────┐
                                            │  Step 3: Load     │
                                            │  - Bulk INSERT    │
                                            │  - COPY to DWH    │
                                            │  - Write Parquet  │
                                            └───────────────────┘

Pros and cons

Pros

  • High throughput: bulk I/O operations (COPY, bulk insert, columnar writes) are vastly more efficient than record-by-record streaming for large datasets.
  • Simpler mental model: a batch job processes a well-defined window of data deterministically; there is no complex state management or late-arriving event logic.
  • Checkpoint/restart: frameworks like Spring Batch persist step state to a job repository; a failed job resumes from the last successful checkpoint rather than starting over.
  • Lower infrastructure cost: batch jobs run on demand, consuming compute only during the processing window; streaming requires always-on infrastructure.
  • Good fit for audit and compliance: a batch job produces a complete, reproducible snapshot of a time window that satisfies regulatory requirements.

Cons

  • High latency: data is only available at the batch frequency; a daily batch means downstream consumers work with data that is up to 24 hours stale.
  • Resource spikes: batch jobs cause large compute and I/O spikes when they run, potentially impacting production systems running on shared infrastructure.
  • Difficult partial failure handling: if a batch job fails midway, you must decide whether to retry from checkpoint or from the beginning, and whether partial writes to the destination must be rolled back.
  • Long recovery time: re-running a 4-hour batch job after a failure means 4+ hours of data delay, which may violate SLAs.
  • Scheduled bottleneck: all dependencies queue up waiting for the nightly batch to complete before they can proceed.

Implementation notes

Spring Batch's chunk-oriented processing model — read N items, process N items, write N items within a transaction, commit, repeat — provides natural checkpoint semantics. The JobRepository persists job and step execution state to a relational database, so jobs can be restarted from the last committed chunk after a failure. Configure chunk size based on memory constraints and transaction overhead: a chunk size of 1,000–10,000 rows is typical for database destinations; use larger chunks (50,000+) for object storage writes where each write has higher overhead. For AWS Glue, use Glue Job Bookmarks to track which S3 objects or DynamoDB records have been processed, preventing reprocessing on incremental runs without manual checkpoint management.

For microbatching (sub-minute intervals), Spark Structured Streaming in batch trigger mode (Trigger.Once() or Trigger.AvailableNow()) processes all available data as a single batch and exits. This provides the throughput efficiency of batch processing with near-streaming freshness when triggered frequently by a scheduler. Apache Airflow or AWS EventBridge Scheduler is the standard choice for cron-based batch orchestration; use DAG dependencies in Airflow to model multi-step batch pipelines with conditional branching and SLA monitoring.

Common failure modes

  • Job overlap: The nightly batch run takes longer than 24 hours and the next scheduled run starts before the previous one finishes, causing race conditions and duplicate writes.
  • Missing checkpoint configuration: A Spring Batch job is configured with an in-memory JobRepository; when the job fails and the JVM exits, all checkpoint state is lost and the entire job restarts from the beginning.
  • Source database contention: The batch job runs a full table scan on the production OLTP database during peak hours, consuming all available I/O bandwidth and degrading real-time user operations.
  • Unbounded memory growth: A batch step accumulates all records in memory before writing (rather than using chunk processing), causing an OOM error when the dataset grows beyond the JVM heap size.
  • Silent partial success: A batch job completes with status SUCCESS but skipped 15% of records due to validation errors; the skip count is logged but no alerting is configured, and the missing data is not discovered until downstream reports show anomalies.

Decision checklist

  • Is the maximum acceptable data latency compatible with the proposed batch window (hourly, daily, weekly)?
  • Have you configured checkpoint/restart so failed jobs resume from the last committed step rather than restarting from scratch?
  • Does the batch job run against a read replica or offline snapshot to avoid impacting the production OLTP database?
  • Is the batch schedule monitored with SLA alerts for jobs that run longer than expected?
  • Have you defined and tested a re-run strategy for cases where the destination already contains partial data from a failed run?
  • Is the skip/error threshold configured appropriately — should a job fail on 1 error or allow up to N validation errors before failing?

Example use cases

  • Nightly DWH load: An Airflow DAG runs at 01:00 UTC, triggers an AWS Glue job that reads changed records from the production RDS PostgreSQL database (using a high-water mark on updated_at), transforms them, and bulk-loads them into Redshift for BI reporting.
  • Monthly billing: A Spring Batch job runs on the first of each month, reads all usage records for the previous month from the metering database, calculates charges, applies discounts, writes invoice records to the billing database, and generates PDF invoices for email delivery.
  • ML feature pipeline: An Apache Spark job runs nightly on EMR, reads 90 days of user interaction events from S3, computes aggregated user features, and writes Parquet feature files to the feature store for model training.
  • Streaming Integration — The real-time alternative to batch; understand the latency/complexity tradeoff.
  • File-Based Integration — Batch jobs often read from or write to files; file-based integration is a common input to batch pipelines.
  • ETL vs ELT — The dominant data movement architectures are both batch-oriented; batch integration is the transport mechanism.

Further reading