CI/CD Pipelines

What it is

A CI/CD pipeline is the automated process that takes code from a developer's commit through build, test, and deployment to production. Continuous Integration (CI) is the practice of frequently merging code changes to a shared trunk (main/master), with automated build and test validation on every merge. CI aims to catch integration problems early, when they are cheap to fix, rather than at release time when conflicts have accumulated. Continuous Delivery (CD) ensures that every change that passes CI is in a releasable state — deployable to production at any time by a manual trigger. Continuous Deployment extends this further: every change that passes all automated checks is automatically deployed to production without human approval.

The standard pipeline stages are: (1) Source — trigger on code push or PR; (2) Build — compile, package, create container image; (3) Unit test — fast tests, <5 minutes; (4) Static analysis — SAST, linting, dependency vulnerability scan; (5) Integration test — tests requiring external dependencies (database, message broker); (6) Publish — push artifact to registry; (7) Deploy to staging — deploy and run smoke/E2E tests; (8) Deploy to production — manual or automatic, with deployment strategy (rolling, blue-green, canary). Fast feedback is the guiding principle: expensive, slow tests run after fast tests to minimise the time before a developer learns that their change broke something.

Trunk-based development (TBD) is the branching strategy that enables true CI. Developers commit directly to the main branch (or merge short-lived branches within a day), keeping the trunk always deployable. Feature flags gate incomplete features in production rather than in long-lived branches. This contrasts with Gitflow (develop/release/hotfix branches) which creates integration debt over weeks-long branch lifetimes and makes the "continuous" in CI a misnomer. DORA research consistently shows trunk-based development as a key differentiator between high and low performing delivery teams.

Why it exists

The DORA (DevOps Research and Assessment) research program tracked software delivery performance across thousands of organisations and identified four key metrics: Deployment frequency (how often code is deployed to production), Lead time for changes (time from commit to production), Change failure rate (percentage of deployments causing failures), and Mean time to restore (MTTR) (time to recover from failures). Elite performers deploy multiple times per day with lead times under an hour and change failure rates under 5%. CI/CD pipelines are the mechanism that makes this performance possible — automation eliminates manual, error-prone release processes and enables the feedback loops that drive quality improvement.

When to use

• All software delivery — CI/CD is a foundational practice, not an optional advanced technique. Every team producing software that reaches users should have an automated pipeline.
• Teams currently doing manual deployments — automation investment has the highest ROI for teams with slow, error-prone manual release processes.

When not to use

• Continuous deployment (automatic to production without approval) requires mature monitoring and rollback capabilities; teams without these should use continuous delivery (deployable on demand) while building that foundation.

Typical architecture

GITHUB ACTIONS PIPELINE (.github/workflows/deploy.yml):

name: CI/CD Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  # Stage 1: Build and fast tests (~3 min)
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with: { go-version: '1.22' }
      - name: Build
        run: go build ./...
      - name: Unit tests
        run: go test -short ./...
      - name: Lint
        uses: golangci/golangci-lint-action@v4

  # Stage 2: Security scan
  security:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Dependency scan
        uses: snyk/actions/golang@master
        env: { SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} }
      - name: Container scan
        uses: aquasecurity/trivy-action@master
        with: { image-ref: '${{ env.IMAGE }}' }

  # Stage 3: Build and push container image
  publish:
    needs: [build-and-test, security]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: |
            ghcr.io/org/checkout:${{ github.sha }}
            ghcr.io/org/checkout:latest

  # Stage 4: Deploy to staging and E2E test
  staging:
    needs: publish
    environment: staging
    steps:
      - name: Deploy to staging
        run: |
          kubectl set image deploy/checkout \
            app=ghcr.io/org/checkout:${{ github.sha }}
          kubectl rollout status deploy/checkout
      - name: Smoke tests
        run: npm run test:e2e -- --env staging

  # Stage 5: Deploy to production (manual approval)
  production:
    needs: staging
    environment:
      name: production
      # Requires manual approval via GitHub Environments
    steps:
      - name: Deploy to production (canary)
        run: ./scripts/deploy-canary.sh ${{ github.sha }}

DORA METRICS TARGETS:
  Elite performers:
    Deployment frequency: multiple per day
    Lead time: <1 hour
    Change failure rate: <5%
    MTTR: <1 hour

Pros and cons

Implementation notes

Pipeline speed is a first-class requirement. If a pipeline takes longer than 10 minutes, developers stop waiting for it and context-switch — the feedback loop is broken. Achieve fast pipelines through: parallelising independent stages (unit tests, lint, SAST can run in parallel); aggressive caching of dependencies and build artifacts; running expensive tests (E2E, integration) only on changes to affected services; and using matrix builds to distribute test execution across multiple runners. Set a "pipeline SLO" — e.g., PR pipeline must complete within 8 minutes, main pipeline within 15 minutes — and track it as a reliability metric.

Flaky tests are the most insidious pipeline problem. A 1% flakiness rate means that a pipeline with 500 tests will fail for random reasons 5% of the time. Track flakiness rates per test and automatically quarantine tests with flakiness above 1% into a "quarantine" suite that runs separately and does not block deployment. Fix quarantined tests within a sprint — they represent real intermittent failures that will eventually impact production if not addressed.

Common failure modes

• Slow pipelines: A 45-minute pipeline effectively stops CI from being "continuous" — developers push, go do other work, and merge without reviewing results. Target <10 minutes for most feedback.
• Ignoring failed pipelines: Teams that routinely bypass or ignore pipeline failures because "it's always flaky" have lost the quality signal — fix the root cause (flaky tests, environment instability) rather than training people to ignore failures.
• Snowflake environments: Staging environments that differ significantly from production (different config, different dependencies, different scale) provide false confidence; prod-like staging is essential for meaningful pre-production validation.
• No artifact promotion: Building a new artifact at each pipeline stage rather than promoting the same artifact (container image SHA) means testing different artifacts than what is deployed; always promote, never rebuild.

Decision checklist

• Does the PR pipeline complete in under 10 minutes?
• Is the same artifact (container SHA) promoted through staging to production rather than rebuilt?
• Are secrets stored in pipeline secrets management (not hardcoded)?
• Are DORA metrics tracked (deployment frequency, lead time, change failure rate, MTTR)?
• Is there a process for identifying and fixing flaky tests?
• Does the pipeline include security scanning (SAST, dependency vulnerabilities, container scan)?

Example use cases

• DORA improvement programme: Team with monthly releases and 4-hour deployment process; implements trunk-based development, automated pipeline, and feature flags; 6 months later deploys daily with 20-minute lead time; deployment-related incidents drop 70% due to smaller change sizes and automated validation.
• Security-gated pipeline: Financial services organisation adds mandatory SAST and dependency vulnerability scanning to pipeline; first month blocks 23 PRs with high-severity findings; developers learn to run checks locally first; findings in production code drop 80% year-over-year.
• Pipeline as documentation: New engineer joins team; pipeline configuration in version control serves as executable documentation of exactly how the service is built, tested, and deployed; onboarding to deployment process requires reading one YAML file instead of tribal knowledge.

Related patterns

• GitOps — CD using Git as the source of truth for Kubernetes deployment state.
• Feature Flags — Enabling trunk-based development by gating incomplete features.
• Canary Releases — Gradual traffic shifting as a production deployment stage.

Further reading

• DORA Research: State of DevOps
• Trunk Based Development
• GitHub Actions Documentation