Resilience Patterns: Circuit Breaker, Retries, Backoff, Timeouts & Bulkheads

In a distributed system, every network call is a bet that another machine will answer quickly. Most of the time it does. The patterns in this guide exist for the rest of the time: the payment gateway that slows from 200 ms to 8 seconds during a sale, the recommendation service that falls over during an IPL final, the database replica that stalls for 40 seconds during failover. Without protection, one of those events becomes a platform-wide outage. With timeouts, careful retries, circuit breakers, bulkheads and load shedding, it becomes a degraded feature and a quiet postmortem.

How Cascading Failures Actually Happen

The intuitive picture of failure is "service B returns errors". The dangerous one is "service B gets slow". Errors are fast and free up resources; slowness holds them.

The thread pool arithmetic

Say an order service has a pool of 200 worker threads, and each request makes one synchronous call to a recommendations service. By Little's Law, throughput = concurrency ÷ latency:

If the order service normally receives 2,000 req/s, a 10× slowdown in a non-essential dependency drops its capacity below its load. Requests queue, the load balancer's health checks start timing out, pods are marked unhealthy, and the remaining pods get even more traffic. Now the API gateway's threads are stuck waiting on the order service, and the cascade climbs one level up. All because a "nice to have" widget got slow.

Retry storms make it worse

Now add retries. If each of four layers (app → gateway → order → inventory) retries up to 3 times (4 attempts), a failure at the bottom can produce up to 4⁴ = 256 calls for one user action. During partial outages, retries multiply load on exactly the component that is already failing. Public postmortems from large cloud providers have repeatedly described this shape: a component slows, clients time out and retry, the extra load prevents recovery. The Google SRE book's chapter on cascading failures is the classic reference on the dynamics.

Timeouts and Deadline Propagation

Every network call needs a timeout. Many clients default to none or to something absurd (some HTTP client and JDBC drivers default to infinite read timeouts). Set both:

• Connect timeout: short (100 ms to 1 s within a data center). A host that cannot accept a TCP connection quickly is probably down.
• Request/read timeout: based on measured latency, typically a bit above the dependency's p99 or p99.9. If inventory's p99 is 120 ms, a 300 ms timeout is reasonable; 30 s is not.

Deadlines, not just timeouts

Independent timeouts at each hop do not compose. If the user-facing budget is 1 s, but the gateway allows 1 s, order service allows 1 s per downstream call and inventory allows 1 s for its DB, work continues long after the user gave up. Instead, propagate a deadline: the gateway sets "must finish by T", each service passes the remaining time downstream and checks it before starting expensive work. gRPC supports this natively (deadlines propagate in request metadata); for HTTP, a header like X-Request-Deadline or a remaining-budget header works.

Retries with Exponential Backoff and Jitter

What to retry

Why jitter matters

Plain exponential backoff (100 ms, 200 ms, 400 ms, 800 ms) still synchronizes clients: 10,000 clients that failed at the same moment retry at the same moments, producing spikes every interval. Adding randomness spreads them out. Marc Brooker's AWS Architecture Blog analysis, "Exponential Backoff and Jitter", compared strategies and found full jitter, sleep = random(0, min(cap, base × 2attempt)), gives a good balance of fewer total calls and fast completion.

import random
import time

class RetryableError(Exception):
    """Raised for transient failures: timeouts, 503, connection reset."""

def call_with_retries(fn, *, max_attempts=4, base=0.1, cap=2.0, deadline=None):
    """Call fn(); retry RetryableError with exponential backoff and full jitter.

    deadline is an absolute time.monotonic() value; we never sleep past it.
    """
    for attempt in range(max_attempts):
        try:
            return fn()
        except RetryableError:
            if attempt == max_attempts - 1:
                raise
            sleep_for = random.uniform(0, min(cap, base * (2 ** attempt)))
            if deadline is not None and time.monotonic() + sleep_for >= deadline:
                raise
            time.sleep(sleep_for)

# attempt 0 fails -> sleep in [0, 0.1) s
# attempt 1 fails -> sleep in [0, 0.2) s
# attempt 2 fails -> sleep in [0, 0.4) s
# attempt 3 fails -> raise

import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Supplier;

public final class Retry {
    public static class RetryableException extends RuntimeException {
        public RetryableException(String msg, Throwable cause) { super(msg, cause); }
    }

    public static <T> T withBackoff(Supplier<T> call, int maxAttempts,
                                    long baseMillis, long capMillis) throws InterruptedException {
        for (int attempt = 0; ; attempt++) {
            try {
                return call.get();
            } catch (RetryableException e) {
                if (attempt == maxAttempts - 1) throw e;
                long expCap = Math.min(capMillis, baseMillis * (1L << attempt));
                long sleep = ThreadLocalRandom.current().nextLong(expCap + 1);   // full jitter
                Thread.sleep(sleep);
            }
        }
    }
}

Retry discipline

• Retry at one layer only, usually the one closest to the failing dependency, or at the edge. Never at every layer.
• Use a retry budget: allow retries only while they are a small fraction (for example 10%) of total requests to that dependency. When the dependency is broadly failing, the budget is exhausted and retries stop automatically. gRPC's retry throttling and Finagle's retry budgets implement this idea.
• Respect Retry-After on 429 and 503.
• Cap total time by the request deadline, not just attempt count.

Idempotency: What Makes Retries Safe

A timeout does not tell you whether the operation happened. The request might have failed before reaching the server, or succeeded with the response lost. Retrying "charge ₹499" blindly can charge twice. The fix is to make the operation idempotent: performing it N times has the same effect as once.

• Naturally idempotent: GET, PUT with a full resource, DELETE, "set status to SHIPPED".
• Not naturally idempotent: POST create order, "increment balance by 500", send SMS.

For the second kind, the client generates an idempotency key (a UUID) per logical operation and sends it on every retry. Stripe's API popularized the Idempotency-Key header; payment gateways and UPI flows rely on unique transaction references for the same reason.

# Table: idempotency_keys(key TEXT PRIMARY KEY, request_hash TEXT,
#                         status TEXT, response_json TEXT, created_at TIMESTAMPTZ)

def create_payment(db, idem_key, request):
    req_hash = sha256(canonical_json(request))
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO idempotency_keys (key, request_hash, status) "
            "VALUES (%s, %s, 'IN_PROGRESS') ON CONFLICT (key) DO NOTHING",
            (idem_key, req_hash)).rowcount
    if not inserted:
        row = db.fetchone("SELECT request_hash, status, response_json "
                          "FROM idempotency_keys WHERE key = %s", (idem_key,))
        if row.request_hash != req_hash:
            return http(422, "idempotency key reused with different request")
        if row.status == "IN_PROGRESS":
            return http(409, "request in progress, retry later")
        return http(200, row.response_json)          # replay the stored result

    result = charge(request)                          # the real side effect
    db.execute("UPDATE idempotency_keys SET status='DONE', response_json=%s "
               "WHERE key=%s", (to_json(result), idem_key))
    return http(201, result)

In a real payment system, the side effect and the status update should commit atomically, or the external call itself should carry the same key so the provider deduplicates. Deeper treatment in Design a Payment System. For message consumers, idempotency usually means deduplicating by event ID, because queues like Kafka and SQS deliver at least once.

The Circuit Breaker Pattern, State by State

• Closed: calls pass through; outcomes are recorded in a sliding window (count-based, e.g. last 100 calls, or time-based, last 60 s). Slow calls above a threshold can count as failures too.
• Open: calls fail immediately without touching the network. The caller's threads are freed instantly, and the dependency gets breathing room.
• Half-open: after the wait, a few trial calls go through. Success closes the breaker; failure reopens it.

import java.util.function.Supplier;

public final class CircuitBreaker {
    public enum State { CLOSED, OPEN, HALF_OPEN }

    private final int windowSize, minCalls, halfOpenPermits;
    private final double failureRateThreshold;
    private final long openNanos;

    private State state = State.CLOSED;
    private final boolean[] window;      // ring buffer: true = failure
    private int idx, count, failures;
    private long openedAt;
    private int halfOpenInFlight, halfOpenSuccesses;

    public CircuitBreaker(int windowSize, int minCalls, double failureRateThreshold,
                          long openMillis, int halfOpenPermits) {
        this.windowSize = windowSize;
        this.minCalls = minCalls;
        this.failureRateThreshold = failureRateThreshold;
        this.openNanos = openMillis * 1_000_000L;
        this.halfOpenPermits = halfOpenPermits;
        this.window = new boolean[windowSize];
    }

    public <T> T call(Supplier<T> action, Supplier<T> fallback) {
        if (!tryAcquire()) return fallback.get();
        try {
            T result = action.get();
            onSuccess();
            return result;
        } catch (RuntimeException e) {
            onFailure();
            return fallback.get();
        }
    }

    public synchronized State state() { return state; }

    private synchronized boolean tryAcquire() {
        if (state == State.OPEN) {
            if (System.nanoTime() - openedAt < openNanos) return false;
            state = State.HALF_OPEN;
            halfOpenInFlight = 0;
            halfOpenSuccesses = 0;
        }
        if (state == State.HALF_OPEN) {
            if (halfOpenInFlight >= halfOpenPermits) return false;
            halfOpenInFlight++;
        }
        return true;
    }

    private synchronized void onSuccess() {
        if (state == State.HALF_OPEN) {
            if (++halfOpenSuccesses >= halfOpenPermits) transition(State.CLOSED);
            return;
        }
        record(false);
    }

    private synchronized void onFailure() {
        if (state == State.HALF_OPEN) { transition(State.OPEN); return; }
        record(true);
        if (count >= minCalls && (double) failures / count >= failureRateThreshold) {
            transition(State.OPEN);
        }
    }

    private void record(boolean failed) {
        if (count == windowSize) {
            if (window[idx]) failures--;          // evict oldest outcome
        } else {
            count++;
        }
        window[idx] = failed;
        if (failed) failures++;
        idx = (idx + 1) % windowSize;
    }

    private void transition(State next) {
        state = next;
        idx = 0; count = 0; failures = 0;
        java.util.Arrays.fill(window, false);
        if (next == State.OPEN) openedAt = System.nanoTime();
    }
}

// usage: open after 50% failures over the last 20 calls (min 10), stay open 30 s
// CircuitBreaker cb = new CircuitBreaker(20, 10, 0.5, 30_000, 3);
// Recs recs = cb.call(() -> recsClient.fetch(userId), Recs::popularFallback);

In production you would use a library: Resilience4j in Java (Hystrix is in maintenance mode), Polly in .NET, or let a service mesh like Istio/Envoy do outlier detection and circuit breaking at the proxy. Interviewers care that you know the states, what counts as a failure, and that open must come with a fallback.

resilience4j:
  circuitbreaker:
    instances:
      recommendations:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 50
        minimumNumberOfCalls: 20
        failureRateThreshold: 50
        slowCallDurationThreshold: 500ms
        slowCallRateThreshold: 60
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
  timelimiter:
    instances:
      recommendations:
        timeoutDuration: 300ms
  bulkhead:
    instances:
      recommendations:
        maxConcurrentCalls: 40

Bulkheads: Isolating Failure Domains

Ships are divided into watertight compartments so one breach does not sink the vessel. In software, a bulkhead caps how many resources a single dependency or workload can consume.

• Per-dependency concurrency limits: at most 40 concurrent calls to recommendations, 80 to payments. If recommendations hangs, at most 40 of the 200 threads are stuck; checkout still works.
• Separate connection pools for different databases or for reads vs writes.
• Separate deployments for different traffic classes: run the admin/reporting API on different pods from the customer API, so a heavy report cannot starve checkout.
• Cell-based architecture at the largest scale: partition customers into independent cells (full stacks), so a bad deploy or poison request affects one cell's users only.

Revisit the thread-pool example: with a 40-call bulkhead on recommendations plus a 300 ms timeout, a recommendations meltdown costs at most 40 threads × 0.3 s, and the breaker opens within seconds. The order service keeps its 4,000 req/s capacity for everything else.

Load Shedding and Backpressure

When demand exceeds capacity, something must give. If nothing is shed, queues grow, every request's latency exceeds its deadline, and goodput (useful completed work) drops to near zero even though the servers are 100% busy. Shedding deliberately keeps goodput high.

• Reject early and cheaply: return 503/429 at the edge before doing expensive work. Rate limiting is one form; see Design a Rate Limiter.
• Bounded queues: an unbounded in-memory queue is a latency bomb. Bound it and reject when full.
• Drop requests that already expired: if the deadline passed while waiting in a queue, discard instead of processing.
• Prioritize: shed analytics beacons, prefetches and recommendations before checkout and payment. Tag requests with a criticality level at the edge.
• Adaptive concurrency limits: algorithms inspired by TCP congestion control (Netflix's concurrency-limits library) measure latency and shrink the allowed in-flight requests when latency rises.
• Waiting rooms for known spikes: virtual queues for ticket sales and Tatkal-style events admit users at a sustainable rate; see Design IRCTC Tatkal Booking.

Backpressure is the asynchronous cousin: consumers signal producers to slow down (bounded channels, reactive streams' request(n), Kafka consumer lag driving producer throttling) instead of buffering without limit.

Fallbacks and Graceful Degradation

A breaker that opens and then throws an error has only half solved the problem. Decide per dependency what the product does without it:

The last row matters: some dependencies are on the critical path and must fail closed. Classifying dependencies as critical vs optional is one of the most useful exercises a team can do, and a great thing to say in an interview.

Putting It Together: A Checkout Call Path

1. Edge: rate limit per user and globally; tag request criticality; set a 2 s deadline.
2. Checkout → inventory (critical): 300 ms timeout, 2 attempts with jittered backoff using a reservation ID as idempotency key, bulkhead of 100, breaker that fails the checkout clearly when open.
3. Checkout → payments (critical): idempotency key per payment attempt; no blind retry on timeout, instead query payment status; breaker per PSP with failover to a secondary PSP.
4. Checkout → offers/recommendations (optional): 150 ms timeout, no retries, bulkhead of 30, fallback to no offers.
5. Async side effects (SMS, email, analytics) via a queue with at-least-once delivery and idempotent consumers, so their outages never block checkout.
6. Observability: breaker state changes, retry counts and shed counts as metrics, so you can see the patterns working. See Observability Explained.

How Resilience Comes Up in Interviews

Almost every system design interview has a moment where the interviewer points at an arrow and asks "what if this service is slow or down?" A strong answer names specific patterns with numbers instead of saying "we'll add retries". It is also a staple of microservices interview questions.

Questions and what they test

• "Why not just retry until it works?" Retry amplification, synchronized retries, non-idempotent side effects. Answer with budgets, jitter and idempotency keys.
• "Explain the half-open state." Why it exists (probe recovery without a thundering herd), how many trial calls.
• "What timeout would you set here?" Derive from p99 and the end-to-end deadline, not from a guess.
• "The payment call timed out. Did the user get charged?" Unknown; query status with the idempotency key or transaction reference, reconcile, never blindly re-charge.
• "How do you protect a service during a 20× spike?" Pre-scaling, admission control/waiting room, load shedding by priority, caching, degrading optional features.
• "Where do these patterns live: library or mesh?" Libraries give app-level context (fallbacks, idempotency); a mesh gives uniform timeouts, retries and outlier ejection without code. Many teams use both, carefully avoiding double retries.

Red flags

• Retries at every layer, or retries on 4xx errors.
• No timeouts, or uniform 30 s timeouts everywhere.
• A circuit breaker with no fallback or no clear critical/optional classification.
• Assuming exactly-once delivery from queues instead of designing idempotent consumers.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.