Secret Management

What it is

Secret management is the discipline of generating, storing, distributing, rotating, and revoking credentials (passwords, API keys, TLS certificates, SSH keys, database credentials, OAuth tokens) in a way that minimises the risk of unauthorised access. A secrets manager is a centralised, access-controlled, audited store for these credentials. Leading solutions include HashiCorp Vault (open-source and enterprise, self-hosted), AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager. Each provides a secure API through which applications retrieve credentials at runtime rather than having them baked into source code, environment variables in insecure locations, or configuration files committed to version control.

The most powerful capability offered by Vault is dynamic secrets: rather than storing a static database password, Vault generates a unique, short-lived credential for each requesting workload on demand, directly provisioning it in the target database (PostgreSQL, MySQL, MongoDB, AWS IAM, etc.). The credential has a configurable TTL (e.g., 1 hour) and is automatically revoked when it expires. This eliminates the risk of a stolen credential being reused long-term: by the time an attacker attempts to use an exfiltrated dynamic secret, it has already expired. Secret rotation is the periodic replacement of static secrets with new values — automated rotation through AWS Secrets Manager or Vault ensures that credentials are changed on schedule without human intervention, ensuring that stolen credentials have a bounded window of validity.

Why it exists

Credential exposure is consistently among the top causes of cloud security breaches. The 2023 Verizon DBIR reported that 49% of breaches involved credentials. The pattern is predictable: a developer hardcodes a database password in a config file, the file is committed to a GitHub repository (public or private), an attacker discovers the secret via automated scanning (tools like truffleHog, GitLeaks, or GitHub's secret scanning run continuously against public repos), and uses the credential to exfiltrate data. The credential exposure window between commit and discovery can be days, weeks, or never — many organisations have no mechanism to detect credential leakage in their codebases. Secret management eliminates the root cause: no static credentials in source code or environment variables because credentials are fetched dynamically from the secrets manager at runtime.

When to use

• Any application that connects to a database, third-party API, message queue, or other service using credentials.
• CI/CD pipelines that need credentials to deploy to infrastructure or call external services.
• Microservices architectures where many services each have their own set of credentials for downstream dependencies.
• Environments with compliance requirements (PCI DSS Requirement 8, SOC 2 CC6) mandating access controls and auditability for credentials.

When not to use

• Development local environments: A personal API key in a developer's local .env file (not committed) is acceptable; the overhead of a full secrets manager for local development is excessive. Use a local Vault dev server or dotenv with .gitignore.
• Public, non-sensitive configuration: Non-credential configuration values (feature flags, timeout values) do not need secrets manager storage; use a configuration management tool instead.

Typical architecture

VAULT DYNAMIC SECRET FLOW:

  Application startup:
  1. App authenticates to Vault using platform identity:
     - AWS: EC2 instance role / ECS task role (AWS auth method)
     - K8s: projected service account token (k8s auth method)
  2. Vault verifies identity with platform API
  3. Vault generates dynamic DB credential:
     CREATE ROLE app_xyz_ LOGIN PASSWORD '...';
     GRANT SELECT, INSERT ON orders TO app_xyz_;
  4. Vault returns: { username, password, lease_id, ttl: 3600 }
  5. App uses credential; agent renews lease before expiry
  6. At TTL expiry: Vault revokes
     DROP ROLE app_xyz_;

STATIC SECRET ROTATION (AWS Secrets Manager):
  Secret: { username: "app_user", password: "abc123" }
  Rotation Lambda:
    1. Generate new password
    2. SET PASSWORD for user in DB
    3. Update secret in Secrets Manager
    4. Test new secret
    5. Finalise rotation
  Schedule: every 30 days

SECRET INJECTION PATTERNS:
  ┌─────────────────────────────────────────┐
  │  Agent sidecar (Vault Agent):           │
  │  - Authenticates to Vault               │
  │  - Renders secrets to tmpfs file        │
  │  - App reads file (no Vault SDK needed) │
  └─────────────────────────────────────────┘
  ┌─────────────────────────────────────────┐
  │  Init container (K8s):                  │
  │  - Fetches secrets from Vault/CSI       │
  │  - Writes to shared emptyDir volume     │
  │  - App reads at startup                 │
  └─────────────────────────────────────────┘

Pros and cons

Implementation notes

For cloud-native deployments, use cloud-provider secrets managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) for static secrets requiring rotation — they integrate natively with IAM, CloudFormation, and managed rotation Lambdas with minimal operational overhead. For Kubernetes, the Secrets Store CSI Driver with provider plugins (Vault, AWS, Azure, GCP) mounts secrets as volumes into pods, avoiding the need to store secrets as Kubernetes Secrets (which are base64-encoded, not encrypted by default, and accessible to any pod with sufficient RBAC). Enable etcd encryption for Kubernetes Secrets if you must use them. Run Vault's secret scanning on your entire Git history with truffleHog or git-secrets as a one-time audit when adopting secret management, and set up pre-commit hooks or GitHub Advanced Security to prevent future leaks.

For the secrets manager itself, ensure it is highly available: Vault with Raft integrated storage requires 3 or 5 nodes for quorum; deploy across availability zones. Test the seal/unseal procedure regularly — a sealed Vault is as disruptive as a complete outage. Store the unseal keys and root token in a separate secure location (HSM, break-glass procedure) distinct from the secrets Vault protects.

Common failure modes

• Secrets in Git history: Removing a secret from the current commit does not remove it from Git history; the credential is still accessible via git log. Rotate the credential immediately and consider rewriting history with git-filter-repo.
• Secrets in CI/CD logs: Commands that echo environment variables or error messages that include connection strings leak secrets to CI/CD log streams, often accessible to all developers.
• Vault seal on restart: Auto-unseal (AWS KMS, Azure Key Vault) must be configured for production; a restarted Vault node that requires manual unseal causes service outages at 3am.
• Overly broad Vault policies: A policy granting read * on all secret paths provides no meaningful least-privilege; scope policies to the specific paths each application actually needs.
• Cached stale credentials: Applications that cache credentials in memory without renewal logic will fail when the TTL expires; always implement lease renewal or credential refresh in the application.

Decision checklist

• Are all secrets stored in a secrets manager and absent from source code, env files committed to Git, and CI/CD configuration?
• Do applications use platform identity (IAM role, Kubernetes service account) to authenticate to the secrets manager — never a static secrets-manager credential?
• Is automatic rotation configured for all static secrets with a rotation interval appropriate to the sensitivity?
• Are Vault policies or IAM permissions scoped to least privilege (specific paths, specific operations)?
• Is the secrets manager deployed in HA configuration across multiple availability zones?
• Is secrets scanning running on the Git repository to detect accidental leaks?

Example use cases

• Microservices on Kubernetes: Vault Agent sidecar injected into each pod authenticates with Kubernetes service account JWT, retrieves dynamic PostgreSQL credentials, renders them to a tmpfs volume; each service gets unique, short-lived DB credentials.
• CI/CD pipeline: GitHub Actions authenticates to AWS via OIDC (no stored AWS access keys), retrieves deployment credentials from AWS Secrets Manager, deploys to ECS, credential TTL expires after pipeline completion.
• Legacy application migration: Existing application uses static DB password; migrate to AWS Secrets Manager with 30-day rotation Lambda; application updated to fetch credential at startup via SDK rather than from environment variable.

Related patterns

• Encryption Patterns — KMS-managed keys and envelope encryption complement secret management.
• Identity and Access Management — Workload identity is the authentication mechanism secrets managers rely on.
• CI/CD Pipelines — Pipelines are a common secret injection point requiring careful management.

Further reading

• HashiCorp Vault Documentation
• AWS Secrets Manager User Guide
• Kubernetes Secrets Store CSI Driver