Circuit Breakers, Retries, and Idempotency: A Survival Guide for Production APIs

Your API is only as reliable as the least reliable thing it calls. And the things it calls will fail — not "might" fail, will fail. The question is whether you've designed for that.

Three patterns, used together, are how production systems survive flaky dependencies: retries with backoff, circuit breakers, and idempotency keys. Each is incomplete alone.

The problem retries alone create

Naive retry: a downstream service slows down, your requests time out at 5s, you retry 3 times. Now you've sent 3x the load to a service that's already struggling. You've become the attack.

This is called retry amplification and it's killed more systems than outages themselves. The 2015 AWS DynamoDB incident is the canonical example: a downstream slowdown caused client retries, which amplified load, which caused more slowdown, which caused more retries. Cascade failure.

1. Retries with exponential backoff and jitter

The fix is to retry, but slowly and randomly:

attempt 1: immediate
attempt 2: wait 1s +/- jitter
attempt 3: wait 2s +/- jitter
attempt 4: wait 4s +/- jitter
attempt 5: give up

Two things matter:

• Exponential backoff — each retry waits longer than the last. This gives the downstream service room to recover.
• Jitter — randomize the wait within a range. Without jitter, every client that failed at the same moment retries at the same moment, re-creating the load spike. Jitter spreads them out.

Also: only retry on transient errors. Retry on 503, 504, network timeout. Don't retry on 400, 401, 403, 404, 422 — those won't change with another attempt, and retrying just wastes resources and can get you rate-limited.

2. Circuit breakers

Retries handle transient failures. But what about sustained failures? If a downstream is down for 5 minutes, retrying every request 5 times means 5x the wasted work for 5 minutes.

A circuit breaker is a state machine that tracks failure rate and "opens" to fail fast:

• Closed (normal): requests flow through. Failures are counted.
• Open (tripping): when failures exceed a threshold (e.g. >50% of last 100 requests), the circuit opens. Requests now fail immediately — no retry, no wait. This stops retry amplification.
• Half-open (probing): after a cooldown, the circuit lets a few requests through to test if the downstream has recovered. If they succeed, close. If they fail, open again.

The circuit breaker's job is to fail fast when failing fast is the right answer. It protects both you (don't waste resources) and the downstream (don't pile on a struggling service).

3. Idempotency keys

Retries create a subtle problem: did the request fail, or did it succeed but the response got lost? If you retry a payment, you might charge the customer twice.

Idempotency keys solve this. The client includes a unique key (a UUID) with each request. The server records the key + result. On retry with the same key, the server returns the cached result instead of re-executing.

POST /payments
Idempotency-Key: 7c8d3f1e-...
Body: { amount: 100, currency: "USD" }

-> first attempt: charges $100, returns { id: "pay_123", status: "succeeded" }
-> network error, client retries with same key
-> server sees key already used, returns { id: "pay_123", status: "succeeded" } without charging again

Stripe popularized this pattern. Every payment API should have it. Every mutating endpoint your service exposes should consider it.

How they fit together

A robust outbound call looks like:

1. Generate an idempotency key (client-side UUID, stable across retries).
2. Check the circuit breaker. If open, fail fast — don't even attempt the call. Return a fallback or a 503.
3. Make the call with a timeout.
4. On transient failure, retry with exponential backoff + jitter, up to N times.
5. On non-transient failure, don't retry. Return the error.
6. Track failures for the circuit breaker. If the threshold trips, the next call fails fast.
7. The idempotency key ensures retries are safe even if the server partially executed.

Omit any one of these and you have a hole:

• Retries without backoff → retry amplification, cascade failures.
• Retries without a circuit breaker → you keep hammering a dead service.
• Retries without idempotency → duplicate writes, double charges, corrupted state.
• Circuit breaker without retries → you trip too aggressively on transient blips.

Fallbacks: the fourth pattern

When the circuit is open or all retries fail, what does the caller get? Options:

• Cached/stale data — "I couldn't reach the inventory service, here's the last known stock count, labeled stale."
• Default value — "I couldn't verify the user's premium status, treat them as free tier for now."
• Degraded feature — "Recommendations are unavailable, show popular items instead."
• Honest error — "This feature is temporarily unavailable." Better than a 500 with a stack trace.

Graceful degradation is what separates "down" from "degraded but usable." Users tolerate degraded. They don't tolerate 500s.

Run the checklist

The ShipReady Reliability section covers all of this: "Retry logic with backoff + idempotency keys," "Circuit breakers & fallback behavior," "Timeouts on all external calls," "Graceful shutdown handling." Each is a separate box because each is a separate failure mode.

Run the checklist and make sure all four are checked before you ship. Your future self, at 3am with a flaky third party, will thank you.