File-Based Integration

What it is

File-based integration is the practice of exchanging data between systems by writing structured files to a shared location (a file server, SFTP drop zone, or cloud object store) that a receiving system periodically polls or is event-triggered to process. Despite the prevalence of API-first and event-driven architectures, file-based integration remains extremely common: B2B data exchange with trading partners, financial institution settlement files, healthcare HL7 batch feeds, and government EDI transactions routinely use the file transfer paradigm because it is simple, auditable, and requires no real-time connectivity between sender and receiver.

Common formats include CSV (simplest, widely supported), fixed-width flat files (legacy mainframe output), XML (SOAP, XBRL, HL7 v2 wrapped), JSON, EDI (X12 for US, EDIFACT for Europe), and Parquet or Avro for analytics-oriented file transfers. Transport mechanisms include SFTP (SSH File Transfer Protocol — the workhorse of B2B integration), AS2 (HTTPS-based with MDN acknowledgements, mandated by many retailers and healthcare payers), S3/Azure Blob/GCS with event notifications, and legacy FTP (avoid due to security concerns).

Why it exists

File-based integration persists because of its inherent decoupling: the producer and consumer don't need to be available simultaneously. The producer writes a file; the consumer processes it when convenient. This asynchronous, store-and-forward model is resilient to temporary outages on either side. It also creates a natural audit trail — every file transfer is an artifact that can be archived, reprocessed, and inspected. For regulated industries, an immutable audit log of "the exact file we received from the bank at 23:00 on 2026-01-15" is a compliance requirement that real-time API calls cannot easily provide.

Trading partner integration is another major driver: a retailer mandating AS2 or SFTP for EDI 850 purchase orders from thousands of suppliers cannot require every supplier to implement a real-time API integration. The file-based standard is the lowest common denominator that any business, regardless of technical sophistication, can support.

When to use

• B2B data exchange with external trading partners who use SFTP/AS2/EDI as their standard integration interface.
• Bulk data exports or imports that are logically batch operations — end-of-day reconciliation files, monthly billing extracts, regulatory reporting submissions.
• When an immutable, archivable record of the exact data transferred is required for compliance (financial clearing, healthcare claim submission).
• Legacy system integration where the legacy system's only output is a file drop and cannot be modified to support an API.
• Large file transfers (multi-gigabyte) where streaming via API is impractical and object storage (S3) is a better transport.
• Cross-network file transfers where real-time connectivity is not guaranteed and asynchronous delivery is acceptable.

When not to use

• When sub-minute latency is required — file polling intervals or S3 event notification delays make file-based integration unsuitable for near-real-time needs.
• Transactional operations requiring acknowledgement and immediate response (order placement, payment processing) — use APIs or messaging instead.
• High-frequency, small-record operations where writing and reading individual files per transaction creates huge overhead compared to message queues or event streams.
• When the data contains sensitive PII that would be better protected by in-transit encryption via API calls with TLS rather than files that may sit in landing zones.

Typical architecture

  SFTP / Cloud Object Store File Integration:

  Trading Partner          SFTP Server / S3 Bucket        Processing Service
  ┌────────────┐          ┌──────────────────────┐       ┌─────────────────┐
  │  Partner   │  SFTP    │  Inbound Drop Zone   │       │                 │
  │  System    │─────────▶│  /inbound/pending/   │       │  File Processor │
  │            │  or S3   │                      │       │                 │
  └────────────┘  upload  │  orders_20260115.csv │──────▶│  1. Validate    │
                          │  claims_batch_001.edi│       │  2. Parse       │
                          └──────────────────────┘       │  3. Transform   │
                                   ▲                     │  4. Load to DB  │
                                   │  S3 Event /         │                 │
                                   │  Cron Poller        └────────┬────────┘
                                   │                              │
                          ┌────────┴─────────┐          ┌────────▼────────┐
                          │  Archive Zone    │          │  Processed Zone │
                          │  /archived/      │◀─────────│  /processed/    │
                          │  (immutable)     │  move    └─────────────────┘
                          └──────────────────┘
          

Pros and cons

Implementation notes

For S3-based file integration, configure S3 Event Notifications to trigger an SNS topic or SQS queue on s3:ObjectCreated:* events. The processing Lambda or ECS task is invoked with the S3 object key. Implement idempotent processing by maintaining a processed-files registry (a DynamoDB table or Redis set keyed on the file's S3 ETag or SHA-256 checksum). If the same file is re-delivered, the processor detects it has already been processed and skips it. Move completed files to an archive prefix immediately after successful processing to prevent double-processing on polling-based systems.

To prevent partial-file processing with SFTP, use an atomic rename pattern: the producer writes to a temp file (orders.csv.tmp) and renames to the final name (orders.csv) only when the write is complete. The consumer only picks up files matching the final name pattern. Alternatively, use a sentinel file convention: the producer writes the data file and then writes a zero-byte control file (orders.csv.done); the consumer processes the data file only when the control file exists. For EDI integrations, use a managed EDI platform (SPS Commerce, TrueCommerce, or AWS B2B Data Interchange) rather than building raw X12/EDIFACT parsers.

Common failure modes

• Partial file processing: The consumer starts reading before the producer finishes writing; it processes a truncated file and loads corrupted records into the database.
• Duplicate processing: A redelivered file (from a retry or manual reprocess) is processed a second time, creating duplicate records with no deduplication logic in place.
• Silently stuck queue: Files accumulate in the drop zone because the consumer crashed; no alerting is configured for drop zone file age; data is hours or days stale before someone investigates.
• SFTP key expiry: A trading partner's SSH key expires or is rotated; the consumer's connection fails; no automated key rotation or monitoring detects the failure until a business user reports missing data.
• File size growth: A batch that normally produces 10MB files suddenly produces 10GB due to a business event; the file processor's memory allocation is insufficient and the job crashes without a meaningful error message.

Decision checklist

• Is your latency requirement compatible with file-based polling intervals (minutes to hours acceptable)?
• Does the trading partner require SFTP/AS2/EDI as their standard interface?
• Have you implemented idempotent file processing to handle redeliveries safely?
• Do you have monitoring for drop zone file age and consumer lag alerting?
• Have you defined a naming and archiving convention to distinguish pending from processed files?
• Is the file format versioned and validated at ingestion to catch schema changes before loading?

Example use cases

• Retail EDI order intake: A retailer receives EDI 850 purchase orders from 500 suppliers via AS2. Each order file is processed by an EDI translation service that converts X12 to internal JSON and publishes an OrderCreated event to the order management system.
• Bank statement reconciliation: A finance team receives daily BAI2 bank statement files via SFTP. An AWS Lambda triggered by S3 events parses the fixed-format file, reconciles transactions against the GL, and posts unmatched transactions to a review queue.
• Healthcare claim submission: A medical billing service generates EDI 837P claim files nightly and uploads them to the payer's SFTP server. The payer processes the file and responds with an EDI 277 acknowledgement and EDI 835 remittance in the return directory.

Related patterns

• Batch Integration — File-based integration is a subset of batch integration; batch jobs often process files as their input source.
• Idempotency — Essential for safe file reprocessing; all file processors must handle duplicate file delivery.
• Claim Check Pattern — Storing large payloads as files in object storage and passing references rather than the data in-band.

Further reading

• Amazon S3 Event Notifications documentation
• EDI Document Standards Reference — EDIBasics
• File Transfer Integration — Enterprise Integration Patterns