Security Architecture

Encryption Patterns

Encryption in distributed systems: AES-256-GCM, envelope encryption, KMS key hierarchy, TLS 1.3, mutual TLS, field-level encryption, key rotation, HSMs, and BYOK.

⏱ 11 min read

What it is

Encryption protects data confidentiality and integrity by transforming plaintext into ciphertext that can only be read by holders of the appropriate key. In distributed systems, encryption is applied at two layers: encryption in transit (protecting data moving between systems over networks using TLS) and encryption at rest (protecting stored data using symmetric encryption of storage volumes, databases, or individual fields). The dominant algorithm for symmetric encryption at rest is AES-256-GCM (Advanced Encryption Standard, 256-bit key, Galois/Counter Mode) — AES provides confidentiality, GCM provides authenticated encryption with associated data (AEAD), ensuring both confidentiality and integrity in a single operation. For asymmetric operations (key exchange, digital signatures), RSA-4096 and ECDSA/P-256 are standard.

Envelope encryption is the pattern used by all cloud KMS providers: data is encrypted with a Data Encryption Key (DEK) generated per-object or per-record; the DEK itself is then encrypted by a Key Encryption Key (KEK) managed in a KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS). The encrypted DEK is stored alongside the ciphertext. This pattern solves the key management scaling problem: you never need to call the KMS for each data operation (the DEK is cached); you only call the KMS to unwrap the DEK, which limits KMS API calls and cost, while ensuring that rotating or revoking the KEK immediately invalidates all data encrypted under it. Bring Your Own Key (BYOK) is an enterprise pattern where the customer owns and controls the root KEK in a customer-managed KMS or HSM, providing cryptographic isolation from the cloud provider.

Why it exists

Encryption at rest ensures that physical theft of storage media, an overprivileged cloud support engineer, or a misconfigured storage bucket that becomes publicly readable cannot expose sensitive data as plaintext. Encryption in transit prevents man-in-the-middle attacks, network sniffing on untrusted networks, and passive data collection at network chokepoints. Together, these controls are mandated by virtually every data protection regulation (GDPR, HIPAA, PCI DSS) and are baseline requirements in SOC 2 Type II audits. Key management — generating, rotating, and controlling access to encryption keys — is the hardest part of encryption: a perfectly encrypted database with its encryption key stored in plaintext in the same system is effectively unencrypted.

When to use

All data in transit must be encrypted — TLS 1.2 minimum, TLS 1.3 preferred, with no fallback to insecure versions.
All data at rest in databases, object stores, block storage, and backups must be encrypted — provider-managed keys at minimum, customer-managed keys for sensitive or regulated data.
Mutual TLS (mTLS) for service-to-service communication in Zero Trust architectures.
Field-level encryption for the most sensitive fields (PAN, SSN, PHI) that require independent access control and audit separate from the database.
BYOK for regulated industries (financial, healthcare, government) where contractual separation of duties requires cryptographic proof the cloud provider cannot access customer data.

When not to use

Encrypting non-sensitive public data: Public static assets (CSS, JavaScript, public images) do not benefit from encryption at rest; the operational cost is not justified.
Rolling your own crypto: Never implement cryptographic primitives from scratch. Use well-audited libraries (libsodium, Bouncy Castle, Go's crypto/tls). Custom implementations introduce subtle vulnerabilities that are impossible to detect without cryptographic expertise.

Typical architecture

ENVELOPE ENCRYPTION PATTERN:

  Write path:
  1. Generate random 256-bit DEK (per-record)
  2. Encrypt plaintext with DEK (AES-256-GCM)
     → ciphertext + auth tag
  3. Call KMS: Encrypt(DEK) → EncryptedDEK
  4. Store: { ciphertext, auth_tag, EncryptedDEK,
              KMS_key_id, iv/nonce }

  Read path:
  1. Fetch record with ciphertext + EncryptedDEK
  2. Call KMS: Decrypt(EncryptedDEK) → DEK
     (KMS logs this call with caller identity)
  3. Decrypt ciphertext with DEK
  4. Return plaintext; discard DEK from memory

KEY HIERARCHY:
  Root of trust: HSM / cloud KMS root
       ↓ wraps
  Customer Master Key (CMK) — never exported
       ↓ wraps
  Data Encryption Key (DEK) — unique per record
       ↓ encrypts
  Actual data

KEY ROTATION:
  Automatic rotation: new CMK version generated
  Re-encryption: existing DEKs re-wrapped with
                 new CMK version (lazy re-encryption)
  Old CMK versions: retained for decryption of
                    data encrypted before rotation

mTLS SETUP:
  - Both client and server present certificates
  - Issued by internal CA (SPIFFE/SPIRE or cert-manager)
  - Short-lived certs (24h) auto-renewed
  - Mutual authentication: server proves its identity,
    client proves its identity
  - Used for: service mesh, microservice-to-microservice

Pros and cons

Pros

Encryption at rest limits the damage from storage media theft, misconfigured access controls, or cloud provider access.
Envelope encryption scales efficiently — KMS is called only for key operations, not for each data operation.
Key rotation provides forward secrecy: future key compromise does not expose past data if old keys are revoked.
mTLS provides cryptographic proof of both client and server identity, beyond what TLS alone provides.
Field-level encryption enables fine-grained access control at the field level, enforced cryptographically rather than only by access control policies.

Cons

Encryption adds latency and CPU overhead; AES-NI hardware acceleration on modern CPUs mitigates this, but KMS API calls add network round-trips.
Key management is operationally complex; incorrect key rotation procedures can result in data that cannot be decrypted.
BYOK adds significant operational burden: the customer is responsible for HSM availability and key backup procedures.
Encrypted fields cannot be indexed for equality or range queries without additional patterns (deterministic encryption or searchable encryption schemes).
mTLS certificate rotation must be automated; expired certificates cause outages as difficult to diagnose as other TLS errors.

Implementation notes

For TLS configuration, enforce TLS 1.2 minimum with TLS 1.3 preferred. Disable weak cipher suites (RC4, DES, 3DES, MD5, SHA-1). Use forward secrecy cipher suites (ECDHE) so that future compromise of the server private key does not expose past sessions. Set HSTS (HTTP Strict Transport Security) with a long max-age (1 year) and include subdomains. For databases, enable TLS on all connections (PostgreSQL: sslmode=require; MySQL: require_secure_transport=ON) and configure server certificate validation in the application client to prevent MITM. For S3, enable default bucket encryption with AWS-managed or customer-managed KMS keys; enforce encryption at upload with a bucket policy denying PutObject without x-amz-server-side-encryption.

For nonce/IV management with AES-GCM: never reuse a nonce with the same key. Generate a new random 96-bit IV for each encryption operation (12 bytes is standard for GCM). The risk of IV reuse is catastrophic for GCM — it leaks both the plaintext and the authentication key. Use a cryptographically secure random number generator (CSPRNG), not a general-purpose RNG. Store the IV alongside the ciphertext; it is not a secret.

Common failure modes

Encryption key stored alongside data: Storing the KMS key ID or DEK in the same database row as the ciphertext without further protection negates the security benefit — an attacker who can read the database can also read the key.
No certificate validation: Accepting any certificate (InsecureSkipVerify in Go, verify=False in Python requests) provides no protection against MITM and negates TLS entirely.
IV reuse in AES-GCM: Using a counter or a sequential IV that wraps around, or reusing the same IV on re-encryption of the same data, breaks GCM authentication.
Weak key derivation: Deriving encryption keys from passwords using MD5 or SHA-1 without key stretching (PBKDF2/bcrypt/Argon2) allows brute-force attacks on the derived key.
Certificate expiry: mTLS or internal service certificates that expire and are not auto-renewed cause immediate service outages. Automate certificate issuance and renewal with cert-manager or SPIRE.

Decision checklist

Is all data in transit protected by TLS 1.2+ with forward-secrecy cipher suites?
Is all sensitive data at rest encrypted with AES-256 (or equivalent) using KMS-managed keys?
Is envelope encryption used so that the KMS is not called for every data operation?
Is automatic key rotation configured on all KMS keys with appropriate rotation schedules?
For the most sensitive fields (PAN, SSN, PHI), is field-level encryption applied independently of database-level encryption?
Are TLS certificates auto-renewed with sufficient lead time before expiry?

Example use cases

Healthcare application: PHI fields (diagnosis codes, medical notes) encrypted with field-level AES-256-GCM using per-patient DEKs wrapped by a customer-managed CMK in AWS KMS; database-level encryption provides a second layer for all fields.
Payment processing: PCI DSS requires encryption of PANs at rest; tokenisation replaces PAN with a token in application databases; the actual PAN is stored only in an encrypted vault with independent access controls.
Service mesh: Istio/Linkerd configures mTLS between all pods in the cluster; SPIFFE SVIDs issued by SPIRE provide workload identity; certs rotate every 24 hours automatically.

Secret Management — KMS keys and certificates are secrets that require secure management.
Data Security — Field-level encryption and tokenisation for PII and regulated data.
Zero Trust Architecture — mTLS is a Zero Trust primitive for workload identity.