Retries, timeouts and circuit breakers: how services fail together

Most large outages are not a single component failing. They are a small failure that the rest of the system amplifies: callers retry, queues fill, timeouts fire on work that was nearly done, and a dependency that was briefly slow is now permanently overloaded. The mechanisms meant to add resilience, retries above all, are usually what does the amplifying.

This article covers how that happens and the tools that stop it: timeouts and deadline propagation, backoff with jitter, retry budgets, idempotency, circuit breakers, load shedding and hedging. The figures show each one on the same small failure.

Why retries are dangerous

A retry is a bet that the failure was transient. When it was, the retry hides it from the user. When it wasn’t, and the dependency failed because it was overloaded, a retry adds load to the one component that can least afford it.

Retries also multiply through layers. If a request passes through d layers and each layer makes up to r attempts, one user request can become this many requests at the bottom:

That multiplier applies at exactly the wrong moment: when the bottom layer is failing, every layer above retries at once. The standard advice is to retry at one layer only, usually the one closest to the user, and have the others fail fast.

Timeouts and deadlines

Every remote call needs a timeout. Without one, a hung dependency holds a thread, a connection and memory in the caller forever, and a slow service turns into a stuck one. Many client libraries default to no timeout, or to minutes.

Set the timeout from the dependency’s measured latency, a little above its p99.9, not from a round number. Too long, and callers pile up waiting. Too short, and you abandon work that was about to succeed, then send it again.

Better than per-call timeouts is a deadline: an absolute time by which the whole request must finish, set once at the edge and passed down with every call. gRPC does this natively, propagating grpc-timeout to each server it calls. A service that receives a request with 50 ms left doesn’t start a 200 ms query, and nothing keeps working on a request whose user has already given up.

Backoff and jitter

If a dependency fails for a moment, every client sees the failure at the same moment. With a fixed retry delay they all come back at the same moment too, a synchronised wave that hits the dependency exactly as it is trying to recover.

Exponential backoff spaces the attempts out: wait base · 2^k before attempt k + 1, up to a cap. But it doesn’t desynchronise anything. Every client still retries at 1 s, 2 s, 4 s. What breaks the waves is jitter, randomising each delay. Marc Brooker’s analysis at AWS (2015) found “full jitter” works best for contended resources:

Jitter belongs anywhere many clients might act in step: retries, cache expiry times, cron jobs that every instance runs “at the top of the hour”, and reconnect loops after a deploy.

Retry budgets

A per-request limit (“at most 3 attempts”) bounds what one request can do, but not what all of them do together. During an outage every request fails, so every request retries, and load triples. A retry budget bounds retries as a fraction of total traffic instead. gRPC’s retry throttling is a token bucket: each failure costs a token, each success earns back a fraction of one, and retries stop when the bucket is half empty. Finagle’s default budget allows retries of about 20% of requests.

The effect is that retries behave as intended when failures are rare, smoothing over a lost packet or a restarting instance, and switch themselves off when failures are common, which is when they would only add load.

What is safe to retry

Not every failure should be retried, and not every operation can be:

• Retry connection errors, 503 (unavailable), 502/504 from a proxy, and 429 (too many requests), honouring the server’s Retry-After header.
• Don’t retry 4xx errors that describe the request: a 400 or 404 will fail the same way every time.
• Only retry writes that are idempotent. A timed-out POST /charges may already have charged the card. Retrying it safely needs an idempotency key: the client sends the same unique key on every attempt, and the server stores the result of the first and returns it for the rest.

Circuit breakers

A circuit breaker, popularised by Michael Nygard’s Release It! (2007), wraps calls to a dependency and tracks their failure rate. When failures cross a threshold it opens: calls fail immediately without touching the network. After a cooldown it goes half-open and lets a few probe calls through; if they succeed it closes, and if not it opens again.

A breaker does two jobs at once:

• It protects the caller: a request that would have waited 30 seconds for a timeout gets an error, or a fallback such as a cached value or a degraded page, in microseconds.
• It protects the dependency: it stops sending it traffic it can’t serve, giving it room to recover.

Breakers are per caller instance, so each one sees only its own slice of traffic. With hundreds of instances, set the thresholds on rates with a minimum number of calls, not on absolute counts.

Metastable failures

Bronson and colleagues (HotOS 2021) named the pattern behind many of the worst outages: a metastable failure. A system runs happily in a state that is efficient but fragile. A trigger such as a deploy, a cache flush or a brief network blip pushes it into an overloaded state. Then a sustaining effect, usually retries, keeps it there after the trigger is gone. Removing the trigger doesn’t fix anything; the load has to be removed.

The overloaded state is self-sustaining because overload makes work less efficient. Requests time out after the server has spent effort on them, so its useful capacity drops just as offered load rises. Every tool in this article works by breaking that loop: budgets cap the extra load, breakers remove it, jitter spreads it out, and deadlines stop wasted work.

Bulkheads and load shedding

A bulkhead gives each dependency its own bounded pool of threads or connections, so one slow dependency exhausts its own pool rather than the whole process. Without one, a service whose calls to a single slow backend block every worker becomes unavailable for everything else too. (How big each pool should be is a queueing question, covered in the article on queueing.)

Load shedding is the server-side counterpart: when a server is over capacity, it rejects work early and cheaply, at admission, rather than accepting it and timing out halfway through. Useful details from large systems:

• Shed by priority. Health checks and requests from paying users go first; prefetches and batch jobs go last.
• Bound queues by time, not length. Facebook’s “Fail at Scale” (Maurer, 2015) describes using CoDel-style queue management and switching to LIFO under overload: the newest request still has a user waiting, while the oldest has probably already timed out.
• Return 503 with Retry-After, so well-behaved clients back off instead of hammering.

Hedged requests

Retries handle failure; hedging handles slowness. A hedged request (Dean & Barroso, “The Tail at Scale”) sends a second copy of a read to another replica if the first hasn’t answered within, say, the p95 latency, and uses whichever reply comes first. It costs about 5% extra load and can cut tail latency dramatically, because a slow replica is usually slow for its own reasons, such as garbage collection or a noisy neighbour. Only hedge idempotent reads, and cancel the loser.

A careful retry loop

A retry wrapper with the pieces above: an overall deadline instead of fresh timeouts, retries only for statuses that invite them, full jitter, respect for Retry-After, and a shared budget that turns retries off when everything is failing.

// Retries with a deadline, capped exponential backoff with full jitter,
// and a token-bucket retry budget shared by every call to the same dependency.
class RetryBudget {
  private tokens: number;
  // Each success earns 'ratio' tokens; each retry spends one. With ratio 0.1, retries
  // can add at most ~10% to steady traffic, and stop entirely during an outage.
  constructor(private max = 10, private ratio = 0.1) { this.tokens = max; }
  onSuccess() { this.tokens = Math.min(this.max, this.tokens + this.ratio); }
  tryRetry() { if (this.tokens < 1) return false; this.tokens -= 1; return true; }
}

const RETRYABLE = new Set([408, 429, 502, 503, 504]);

export async function callWithRetries<T>(
  attempt: (signal: AbortSignal) => Promise<T>,
  { deadline, budget, base = 100, cap = 2_000, maxAttempts = 3 }:
    { deadline: number; budget: RetryBudget; base?: number; cap?: number; maxAttempts?: number },
): Promise<T> {
  for (let k = 0; ; k++) {
    const remaining = deadline - Date.now();
    if (remaining <= 0) throw new Error("deadline exceeded");
    try {
      // Each attempt gets what is left of the caller's deadline, not a fresh timeout.
      const result = await attempt(AbortSignal.timeout(remaining));
      budget.onSuccess();
      return result;
    } catch (err) {
      const status = (err as { status?: number }).status;
      const retryable = status === undefined || RETRYABLE.has(status); // network errors, or a status that says "try later"
      if (!retryable || k + 1 >= maxAttempts || !budget.tryRetry()) throw err;
      // Full jitter: sleep a uniform random time in [0, min(cap, base * 2^k)].
      const retryAfter = (err as { retryAfterMs?: number }).retryAfterMs ?? 0;
      const sleep = Math.max(retryAfter, Math.random() * Math.min(cap, base * 2 ** k));
      if (Date.now() + sleep >= deadline) throw err; // no point waking up after the caller gave up
      await new Promise((r) => setTimeout(r, sleep));
    }
  }
}

References

1. Brooker, M. (2015). Exponential Backoff And Jitter. AWS Architecture Blog. aws.amazon.com/blogs/architecture
2. Bronson, N., Aghayev, A., Charapko, A. & Zhu, T. (2021). Metastable Failures in Distributed Systems. HotOS. doi:10.1145/3458336.3465286
3. Huang, L. et al. (2022). Metastable Failures in the Wild. OSDI. usenix.org
4. Nygard, M. T. (2018). Release It! Design and Deploy Production-Ready Software, 2nd ed. Pragmatic Bookshelf. pragprog.com
5. Dean, J. & Barroso, L. A. (2013). The Tail at Scale. Communications of the ACM. doi:10.1145/2408776.2408794
6. Maurer, B. (2015). Fail at Scale: Reliability in the face of rapid change. ACM Queue. queue.acm.org
7. gRPC. A6: gRPC Retry Design (retry policy and retry throttling). github.com/grpc/proposal
8. Beyer, B. et al. (2016). Site Reliability Engineering, ch. 21–22: Handling Overload; Addressing Cascading Failures. O’Reilly. sre.google/sre-book