Cloud Storage Patterns

What it is

Cloud storage falls into four fundamental categories. Object storage (AWS S3, GCS, Azure Blob Storage) is a flat namespace of objects accessed via HTTP API, optimised for durability (eleven nines), massive scale, and concurrent read access; it is ideal for unstructured data: user uploads, logs, backups, ML datasets, static web assets, and data lake raw zones. Block storage (AWS EBS, GCP Persistent Disk, Azure Managed Disks) presents a raw block device formatted with a filesystem and mounted to a single VM instance; it is optimised for low-latency random I/O and is required for relational databases, NoSQL data directories, and virtual machine OS disks. File storage (AWS EFS/FSx, Azure Files, GCP Filestore) provides a shared filesystem mounted by multiple instances simultaneously via NFS or SMB; it is used for shared application state, rendering farm outputs, and legacy applications that require POSIX filesystem semantics across multiple nodes.

Archival storage (S3 Glacier, GCS Archive, Azure Archive) stores data at a cost of $0.001–0.004/GB/month — 5–40× cheaper than standard object storage — in exchange for retrieval latency measured in minutes to hours. Most organisations have the majority of their stored data in a cold state but leave it in Standard tier due to missing lifecycle automation; S3 Intelligent-Tiering and lifecycle rules address this by automatically transitioning objects based on access patterns or age rules without application changes.

Why it exists

Different workloads have fundamentally different storage access patterns that no single storage technology serves optimally. A relational database needs microsecond-latency random read/write access to 8-KB pages — only block storage on NVMe SSD (io2 Block Express) can provide sub-millisecond IO. A data lake accumulates petabytes of CSV and Parquet files that are batch-read by Spark jobs once per day — block storage would cost $1,000/TB/month vs S3 Standard at $23/TB/month. A content delivery origin bucket serves millions of small static files to a CDN — only object storage scales horizontally to millions of concurrent requests without capacity planning.

The economics of cloud storage tiers are compelling: data that is accessed less than once per month costs 10–40× less in Glacier or Archive tier than in Standard. For organisations with regulatory data retention requirements (7–10 years for financial records), the difference between paying $23/TB/month (Standard) and $0.4/TB/month (Glacier Deep Archive) for 100 TB of retained records is $2,300/month vs $40/month annually — a 98% reduction. Lifecycle automation with S3 lifecycle policies or GCS retention rules makes this reduction essentially free to implement.

When to use

• Object storage: Unstructured data, user uploads, application assets, logs, ML training datasets, data lake raw/curated zones, backup targets, or any data accessed via HTTP range requests.
• Block storage: Database data directories (MySQL, PostgreSQL, MongoDB), Kubernetes persistent volumes requiring ReadWriteOnce semantics, VM OS disks, or any workload requiring POSIX filesystem operations with sub-millisecond latency.
• File storage: Shared configuration files accessed by multiple pods, rendering workloads that produce output consumed by multiple downstreams, legacy POSIX applications requiring shared filesystem mounts, and home directories in VDI/remote desktop environments.
• Archival storage: Data past its active analysis window but required for legal/regulatory retention, log archives, backup long-term retention tiers, and infrequently accessed historical datasets.
• S3 lifecycle policies: Any bucket accumulating data over time where access frequency naturally decreases with age — logs after 30 days, backups after 90 days, user-generated content after a user churns.

When not to use

• Object storage for databases: S3 latency (first-byte ~5–100 ms) is incompatible with database random I/O requirements; some analytical databases (DuckDB, Trino) query S3 directly but buffer large columnar reads, which is a fundamentally different access pattern than transactional databases.
• Block storage for shared access: EBS volumes support only a single attached instance at a time (except Multi-Attach io1/io2 in limited configurations); use EFS or S3 for data that multiple pods or instances must write to concurrently.
• Glacier for unpredictably accessed data: If business requirements might demand retrieval within seconds (a legal hold, a production incident), Glacier Instant Retrieval is appropriate, but Deep Archive with multi-hour retrieval is not.
• EFS for database workloads: NFS latency (1–10 ms) is too high for database I/O patterns; always use block storage (EBS) for databases even in Kubernetes.

Typical architecture

Cloud Storage Decision Map
─────────────────────────────────────────────────────────
  User uploads images/videos
      │
      ▼ PUT via pre-signed URL
  S3 Standard (hot)
      │ S3 Event Notification
      ▼
  Lambda: generate thumbnail, extract metadata
      │
      ▼ Store metadata
  RDS PostgreSQL → EBS io2 (block, low-latency IOPS)
      │
  S3 Lifecycle Policy (object tier transitions)
      Day   0 → S3 Standard
      Day  30 → S3 Standard-IA  (46% savings)
      Day  90 → S3 Glacier IR   (68% savings, ms retrieval)
      Day 365 → S3 Glacier DA   (95% savings, hours retrieval)

  Kubernetes shared config:
      EFS (NFS ReadWriteMany) → multiple pods read same files

  Log archive pipeline:
      CloudWatch Logs → Kinesis Firehose → S3 Standard-IA
      → Glacier after 90 days → delete after 7 years

  Cross-region replication:
      S3 production bucket (us-east-1)
          → CRR → S3 replica (eu-west-1) for DR

Pros and cons

Implementation notes

For S3 lifecycle policies, enable them at the bucket level with an expiration date and transition rules. Use S3 Storage Lens in your AWS Organisation to find buckets with high Standard storage balances and no lifecycle rules — these are the highest-priority candidates for cost reduction. For objects that may be accessed unpredictably, S3 Intelligent-Tiering automates transitions at no per-object fee above the monitoring charge ($0.0025/1,000 objects/month). Enable S3 Versioning for production buckets and combine with lifecycle rules that expire previous versions after 30–90 days to prevent version accumulation from tripling storage costs.

For S3 event notifications triggering Lambda: configure s3:ObjectCreated:* event notifications on the source bucket with a Lambda function ARN as the destination. Use a filter prefix (uploads/) to restrict triggers to the relevant prefix. Note that Lambda invocations are asynchronous for S3 events — implement a dead-letter queue (SQS or SNS) to capture failed Lambda invocations and retry. For cross-region replication, enable S3 CRR with a replication rule specifying destination bucket ARN, IAM role, and optional KMS key. CRR does not replicate existing objects — use S3 Batch Operations to copy pre-existing objects to the replica bucket for DR initialisation.

Common failure modes

• S3 public access enabled: Misconfigured bucket policy or ACL exposing data publicly; enable S3 Block Public Access at the account level and use pre-signed URLs for customer-facing uploads/downloads.
• Missing encryption at rest: RDS instances created without encryption cannot be encrypted in place; EBS volumes similarly require snapshot + re-encryption + restore — enforce via Config rules and SCPs.
• EBS volume not in same AZ as instance: EBS volumes are AZ-specific; creating a volume in a different AZ from the EC2 instance causes attachment failures — automate with CloudFormation or Terraform selecting the same AZ.
• Glacier accidental mass restore: A script that inadvertently submits Bulk restore for all objects in a Glacier bucket triggers per-GB restore fees; review Bulk restore jobs before submission and set retrieve limits.
• S3 ListBucket DDoS amplification: Unthrottled S3 list operations by a buggy client listing millions of objects per minute can exhaust S3 request rate limits for the bucket prefix — implement rate limiting and pagination at the application layer.

Decision checklist

• Have you selected the correct storage type for each workload based on access pattern (random vs sequential, single vs multi-instance, latency requirements)?
• Are S3 lifecycle rules configured for all buckets that accumulate ageing data?
• Is server-side encryption enabled for all S3 buckets (SSE-S3 minimum; SSE-KMS for regulated data)?
• Are S3 Block Public Access settings enabled at the account level?
• Is versioning enabled with corresponding lifecycle rules to expire old versions and prevent cost accumulation?
• For EBS, are automated snapshots configured with Data Lifecycle Manager policies including retention limits?

Example use cases

• Media platform: User-uploaded videos stored in S3 Standard; S3 event notification triggers transcoding Lambda; transcoded files in S3 Standard-IA after 30 days; originals transitioned to Glacier IR after 90 days; CloudFront serves popular content from S3 with cache-hit rates above 95%.
• Financial audit trail: Transaction records written to S3 Standard via Kinesis Firehose; lifecycle policy transitions to Glacier Deep Archive after 1 year; 7-year retention enforced via S3 Object Lock in Compliance mode; annual compliance exports via S3 Batch Operations restore jobs.
• Rendering farm: Artists' project files on EFS (shared NFS mount, ReadWriteMany); render workers on Spot EC2 instances all mount same EFS filesystem; completed render outputs copied to S3 for client delivery; EFS used only for active projects, reducing file storage cost by keeping archive in S3 Standard-IA.

Related patterns

• Cloud Cost Optimization — storage lifecycle policies are one of the highest-ROI cost optimisation actions.
• Data Lake — S3/GCS object storage is the universal storage layer for data lake architectures.
• Serverless Architecture — S3 event notifications are a primary trigger source for Lambda-based serverless architectures.
• Disaster Recovery Patterns — S3 cross-region replication is a key mechanism for object storage DR objectives.

Further reading

• Managing your storage lifecycle — Amazon S3 Docs
• Storage classes — Google Cloud Storage
• Storage options for EC2 — Amazon EC2 Docs