Deployment & Delivery

GitOps

Git as the single source of truth for cluster state, Flux and ArgoCD reconciliation loops, declarative desired state, pull vs push delivery models, and automatic drift detection and correction.

⏱ 10 min read

What it is

GitOps is an operational model for Kubernetes (and other infrastructure) where the desired state of the system is declared in Git, and an automated agent continuously reconciles the actual state to match the desired state. The four principles of GitOps (from the OpenGitOps project): (1) Declarative — the entire system state is described declaratively; (2) Versioned and immutable — desired state is stored in Git with full history; (3) Pulled automatically — approved changes are applied automatically; (4) Continuously reconciled — running agents ensure actual state matches desired state and alert on divergence.

The two primary GitOps tools are Flux and ArgoCD. Both run inside Kubernetes and implement a reconciliation loop: watch a Git repository, detect changes to manifests, and apply them to the cluster. Flux is a CNCF graduated project that uses a set of composable controllers (source-controller, kustomize-controller, helm-controller) and integrates tightly with Flagger for progressive delivery. ArgoCD provides a rich web UI, application health visualisation, and multi-cluster management; it is better suited for organisations that want visual oversight of deployment state across many applications. Both support Kustomize and Helm for manifest templating.

The pull vs push deployment model is a key distinction. Traditional CD pipelines "push" deployments by running kubectl apply from a pipeline runner that has credentials to the cluster. GitOps uses a "pull" model: the agent inside the cluster pulls from Git and applies changes. The pull model eliminates the need to grant external systems (CI runners) direct cluster access, reducing the attack surface. It also means the agent continuously corrects drift — if someone manually edits a resource in the cluster, the agent will restore it to the state declared in Git within the reconciliation interval (typically 1-5 minutes).

Why it exists

Before GitOps, Kubernetes deployments were often managed through a mixture of CI pipeline kubectl apply commands, manual helm upgrade invocations, and undocumented emergency changes. The result was configuration drift — the actual state of the cluster drifted from any documented or intended state, and no one had a reliable answer to "what version is running in production and why?" GitOps provides auditability (every change is a Git commit with author, timestamp, and message), recovery (rolling back is a Git revert), and consistency (drift is automatically corrected). It transforms Kubernetes operations from an imperative ("run these commands") to a declarative ("here is the desired state") model.

When to use

  • Kubernetes deployments — GitOps is the recommended deployment model for Kubernetes environments.
  • Teams wanting cluster auditability and change history — Git provides a complete audit trail for all infrastructure changes.
  • Multi-cluster environments — ArgoCD and Flux both support managing multiple clusters from a single Git repository with per-cluster configuration overlays.

When not to use

  • Non-Kubernetes infrastructure — GitOps as implemented by Flux/ArgoCD is Kubernetes-specific; for general infrastructure, Terraform with GitOps-style practices (Atlantis) is the equivalent.

Typical architecture

GITOPS REPOSITORY STRUCTURE:
clusters/
  production/
    apps/
      checkout/
        kustomization.yaml
        deployment.yaml
        service.yaml
        hpa.yaml
      payment/
        kustomization.yaml
    infrastructure/
      cert-manager/
      ingress-nginx/
  staging/
    apps/
      checkout/
        kustomization.yaml  # inherits base, overrides image tag

FLUX CONFIGURATION:
# GitRepository: defines what to watch
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: fleet-infra
spec:
  interval: 1m         # Check for changes every 1 minute
  url: https://github.com/org/fleet-infra
  ref: { branch: main }
  secretRef: { name: flux-system }

# Kustomization: applies manifests from GitRepository
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: checkout-production
spec:
  interval: 5m          # Reconcile every 5 minutes
  path: ./clusters/production/apps/checkout
  prune: true           # Remove resources deleted from Git
  sourceRef:
    kind: GitRepository
    name: fleet-infra
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: checkout
    namespace: production

ARGOCD APPLICATION:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-production
spec:
  project: production
  source:
    repoURL: https://github.com/org/fleet-infra
    targetRevision: main
    path: clusters/production/apps/checkout
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true       # Delete resources removed from Git
      selfHeal: true    # Correct manual changes (drift)
    syncOptions:
    - CreateNamespace=true

DEPLOYMENT WORKFLOW:
  1. CI pipeline builds image, pushes to registry
  2. CI updates image tag in Git (via PR or direct commit)
  3. Flux/ArgoCD detects Git change (within interval)
  4. Reconciler applies updated Deployment to cluster
  5. Kubernetes rolls out new pods
  6. Health check confirms rollout success

Pros and cons

Pros

  • Complete audit trail: every cluster state change is a Git commit with author, timestamp, message, and PR review history — invaluable for incident investigation and compliance requirements.
  • Self-healing: drift correction means manual changes are automatically reverted, preventing configuration drift accumulation over time and enforcing consistent cluster state.
  • Pull model security: CI systems no longer need cluster credentials; only the in-cluster agent accesses the cluster, significantly reducing the blast radius of CI system compromise.

Cons

  • Reconciliation delay: changes take 1-5 minutes to apply after merge (depending on polling interval); urgent hotfixes cannot be applied instantly — teams must either accept the delay or have an emergency escape hatch.
  • Secrets management integration requires additional tooling (Sealed Secrets, External Secrets Operator) since plaintext secrets cannot be stored in Git; this adds operational complexity to the GitOps setup.
  • Self-healing creates friction for debugging: manually scaling a deployment for investigation purposes is immediately reverted by the reconciler — teams must understand GitOps semantics before operating clusters managed this way.

Implementation notes

Separate application code repositories from deployment configuration repositories (often called "app repo" and "infra/fleet repo"). When a CI pipeline builds a new image, it opens a PR or directly commits to the infra repo updating the image tag. This separation of concerns means the infra repo is the single source of truth for "what is deployed where" and can be independently reviewed, versioned, and audited without coupling to application source code. Tools like Flux's image automation controllers or Argo Image Updater can automate the image tag update step in the infra repo.

For secrets, use External Secrets Operator (ESO) with a secrets backend (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager). ESO reconciles Kubernetes Secrets from an external store — the GitOps repository stores ExternalSecret resources (which reference secret names but not values), and ESO materialises the actual secrets at runtime. This keeps Git as the source of truth for secret references while secrets values remain in a secure external store and never in Git.

Common failure modes

  • Secrets in Git: Storing plaintext Kubernetes Secrets in the GitOps repository exposes credentials in version history permanently; use ESO or Sealed Secrets from day one.
  • Missing prune policy: Resources deleted from Git are not deleted from the cluster without prune: true; orphaned resources accumulate over time consuming cluster resources and creating confusion about the actual desired state.
  • Manual changes causing confusing reconciliation: Operators unfamiliar with GitOps make urgent manual changes (scale up a deployment) that are silently reverted; teams must understand that the Git repo is authoritative and communicate manual interventions clearly.

Decision checklist

  • Is the GitOps repository separate from application source code?
  • Are secrets managed via External Secrets Operator or Sealed Secrets (no plaintext in Git)?
  • Is prune: true configured to clean up resources deleted from Git?
  • Is selfHeal: true (ArgoCD) or equivalent configured to correct drift?
  • Is there a documented emergency procedure for urgent changes that cannot wait for reconciliation?
  • Are GitOps tool deployments (Flux/ArgoCD themselves) also managed via GitOps?

Example use cases

  • Audit trail for compliance: SOC2 audit requires evidence of change management for production deployments; GitOps provides a complete Git history of every image tag update, config change, and RBAC modification with PR review records; audit evidence generation is trivial.
  • Disaster recovery: Production cluster suffers catastrophic failure; new cluster provisioned in 20 minutes; GitOps agent installed and pointed at infra repo; entire workload state reconciled from Git within 10 minutes; total recovery time 30 minutes vs days of manual reconstruction from documentation.
  • Multi-environment promotion: Image tag update PR to infra repo updates staging environment first; after automated testing confirms stability, a second PR updates production; the promotion workflow is transparent, auditable, and consistent across all teams.

Further reading