Progressive delivery: canaries, blue-green and feature flags

Most outages start with a change: a deploy, a configuration push, a flag flip, a migration. Teams respond in one of two ways. Some slow down, batching changes into fewer, bigger releases that each carry more risk. Others make each change smaller and expose it gradually, so that a bad one reaches few users and is undone in seconds. The evidence favours the second, and progressive delivery is the set of techniques that makes it work.

This article covers the main deployment strategies and what each one protects against, how to run a canary that actually catches regressions, why the comparison should be automated, how feature flags separate deploying code from releasing it, and the changes that none of this can make safe.

Why changes cause outages

Google’s SRE book estimates that roughly 70% of its outages were caused by changes to a live system. The DORA research programme, behind the book Accelerate, found over several years that the teams that deploy most often also have the lowest change failure rates and recover fastest. Small, frequent changes are easier to review, easier to reason about when they fail, and cheaper to roll back.

Testing can’t catch everything because production differs from every test environment: real traffic patterns, real data, real scale, real dependencies. The only full test of a change is production itself. Progressive delivery accepts that and limits the blast radius: the share of users and time exposed before a bad change is detected and reversed.

Rolling, blue-green, canary

• Rolling update. Replace instances a few at a time; Kubernetes Deployments do this by default. Health checks stop a rollout that crashes on start, but a version that starts fine and returns wrong answers rolls out to everyone. Rolling back is another rollout, which takes minutes.
• Blue-green. Run the new version as a complete second environment, test it, then switch the router. Rollback is switching back, which takes seconds. But every user moves at once, and it needs double capacity during the switch.
• Canary. Send a small share of real traffic to the new version, compare it with the old, and increase the share in stages if it looks healthy. It is the only one of the three that tests the change on real traffic while limiting who sees it.

All three require old and new versions to be compatible with each other and with the data, because for some period both serve traffic against the same database. That is the expand-and-contract discipline from the API evolution article, and it is a precondition for everything here.

Canaries done properly

A canary is only as good as the details:

• Stages with bake time. A typical schedule is 1%, 5%, 25%, 50%, 100%, with a pause at each. Some bugs appear immediately; others need time, such as a memory leak, a cache that fills, or a daily batch job. Longer pauses at the later stages catch slower problems where they cost the most.
• Enough traffic to judge. At 1% of a service doing 300 requests a second, the canary sees 3 requests a second. A regression from 0.2% to 0.6% errors adds less than one error a minute, which is lost in the noise. Early stages catch crashes and gross failures; subtle regressions need the larger stages, or longer windows.
• Sticky assignment. Route by a hash of the user or session, not per request, so one user doesn’t bounce between versions and see inconsistent behaviour.
• Compare against a baseline, not against history. Compare the canary with the stable version over the same minutes, so time-of-day traffic and a dependency’s bad afternoon affect both equally. Better still, compare against a fresh baseline: a few instances of the old version deployed at the same moment as the canary, so that cold caches and warm-up affect both sides alike.

Automated analysis

A human watching dashboards during a rollout is slow, inconsistent, and not watching at 3 a.m. Automated canary analysis, as in Netflix and Google’s open-source Kayenta, Argo Rollouts or Flagger, runs the comparison on a schedule and decides:

• Choose a few metrics that reflect users: error rate, latency at p50 and p99, and saturation signals such as CPU and memory. Add a business metric where you can, such as checkouts completed per minute, since some regressions return 200 with the wrong content.
• Use relative thresholds. “Canary error rate no more than 1.5× stable” adapts to normal variation; “error rate below 1%” fires on a bad day for both versions, or misses a regression from 0.1% to 0.9%.
• Have three outcomes, not two. Pass, fail, and inconclusive when there isn’t enough data. Inconclusive should hold the rollout at its current stage, never promote it.
• Roll back automatically and promptly. The point of a canary is lost if a failed analysis only sends a notification.

The thresholds connect naturally to SLOs: a canary that burns the error budget much faster than stable is a failed canary, and a service whose budget is already spent should be deploying only fixes.

Feature flags

A canary controls which version of the code a request reaches. A feature flag controls which behaviour runs inside a version. Code ships “dark”, turned off, then is turned on separately, for internal users first, then a percentage, then everyone. That separates deploy, a technical event, from release, a product decision, and gives each its own rollback: a bad deploy rolls back the binary; a bad feature flips a flag in seconds, with no deploy.

Flags have costs that grow quietly:

• Every flag forks the code. Ten independent flags make over a thousand combinations, most never tested together. Keep flags short-lived and few.
• Flag debt. A flag that has been at 100% for months is dead code with a switch that could still be flipped by accident. Give every release flag an owner and an expiry date, and remove it once the feature is fully released.
• Flags are production changes. Flipping a flag for everyone at once is a big-bang deploy without the review. Roll flags out in percentages with the same monitoring as a canary, and keep an audit log of who changed what.
• Evaluation must not become a dependency. If the flag service is down, the application should keep working with its last known or default values. Evaluate flags locally from a cached rule set rather than with a network call per check.

Long-lived operational flags are a different and useful thing: kill switches that turn off an expensive feature under load, or route around a failing dependency. Name them differently from release flags, and test them.

What progressive delivery can’t protect

Some changes can’t be canaried, because they are global by nature:

• Database schema migrations. There is one schema, shared by every version. Make every migration backward compatible, deploy it separately from the code that needs it, and use expand and contract so it can be rolled out and back without touching data.
• Data changes and backfills. A bad backfill corrupts rows that every version reads. Run them in batches, verify as you go, and keep a way to undo them.
• Shared configuration. Some of the largest outages came from a configuration push that reached every server at once. Treat configuration as code: review it, validate it, and roll it out in stages, region by region, exactly like a binary.
• Irreversible side effects. Emails sent, payments taken and messages published to other systems can’t be rolled back. Put new side effects behind a flag, and turn it on slowly.

Making it routine

• Every change goes through the same pipeline: code, configuration, flags and infrastructure. An emergency path that skips the canary will eventually be used for a change that needed it.
• Roll out by region, then by percentage within a region, so a change never reaches every region at the same moment (see the multi-region article).
• Measure the pipeline. The DORA metrics are a good start: deployment frequency, lead time for changes, change failure rate, and time to restore service.

A canary with Argo Rollouts

A Kubernetes canary that steps from 1% to 50% with pauses, labels canary and stable pods so metrics can be split by version, and runs an analysis every minute that compares the canary’s error ratio with the stable version’s. Three failed measurements abort the rollout and return all traffic to stable.

# A canary rollout with automated analysis, using Argo Rollouts on Kubernetes.
# Each step shifts traffic, then pauses while the analysis runs in the background;
# a failed analysis aborts the rollout and sends all traffic back to the stable version.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 20
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2026-10-29.3
  strategy:
    canary:
      canaryMetadata: { labels: { track: canary } }   # so metrics can be split by version
      stableMetadata: { labels: { track: stable } }
      analysis:
        templates: [{ templateName: errors-vs-stable }]
        startingStep: 1
      steps:
        - setWeight: 1
        - pause: { duration: 10m }
        - setWeight: 5
        - pause: { duration: 15m }
        - setWeight: 25
        - pause: { duration: 30m }
        - setWeight: 50
        - pause: { duration: 30m }
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: errors-vs-stable
spec:
  metrics:
    - name: error-ratio-vs-stable
      interval: 1m
      failureLimit: 2            # the third failed measurement aborts the rollout
      # Compare the canary with the stable version over the same minutes, not with a fixed number.
      successCondition: result[0] <= 1.5
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            (sum(rate(http_requests_total{app="checkout", track="canary", code=~"5.."}[5m]))
              / sum(rate(http_requests_total{app="checkout", track="canary"}[5m])))
            /
            clamp_min(
              sum(rate(http_requests_total{app="checkout", track="stable", code=~"5.."}[5m]))
                / sum(rate(http_requests_total{app="checkout", track="stable"}[5m])),
              0.001)

References

1. Beyer, B., Jones, C., Petoff, J. & Murphy, N. R. (2016). Site Reliability Engineering, ch. 1: Introduction; ch. 8: Release Engineering. O’Reilly. sre.google/sre-book
2. Beyer, B. et al. (2018). The Site Reliability Workbook, ch. 16: Canarying Releases. O’Reilly. sre.google/workbook/canarying-releases
3. Forsgren, N., Humble, J. & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution. itrevolution.com
4. Graff, M. & Sanden, C. (2018). Automated Canary Analysis at Netflix with Kayenta. Netflix Technology Blog. netflixtechblog.com
5. Hodgson, P. (2017). Feature Toggles (aka Feature Flags). martinfowler.com
6. Argo Project. Argo Rollouts: Canary and Analysis. argo-rollouts.readthedocs.io
7. Humble, J. & Farley, D. (2010). Continuous Delivery. Addison-Wesley. continuousdelivery.com