NOTE
Designing Timeouts and Retries
Deadline propagation, timeout budgets, retry safety, exponential backoff/jitter, retry storms, idempotency, hedging, and observability.
This is a historical learning note and may contain outdated or incomplete understanding.
1. Every Remote Call Needs a Deadline
Without a deadline, slow dependencies consume threads/connections until the caller itself collapses.
Prefer end-to-end deadlines propagated downstream so every hop knows how much useful time remains.
2. Timeout Budget
A downstream timeout must fit within the caller’s remaining deadline and leave time for response processing/recovery.
Do not give each of five sequential dependencies the full user request timeout.
3. Retry Only Useful Failures
Retries help transient errors. They do not fix invalid requests, deterministic failures, or persistent overload.
Retry only operations that are idempotent or protected by idempotency keys/deduplication.
A timeout often means unknown outcome, not “the server did nothing.”
4. Backoff and Jitter
Use exponential/decorrelated backoff with jitter so thousands of clients do not retry simultaneously.
5. Retry Budget
Limit attempts and aggregate retry traffic. If every layer retries three times, fan-out can explode multiplicatively.
Usually one layer should own retry policy for a given call.
6. Hedged Requests
For read-only/idempotent operations with long-tail latency, a delayed duplicate request can reduce tail latency, but it consumes extra capacity and needs strict budgets.
7. Observe
Track original attempts separately from retries, timeout stage, final status, and added load. A rising retry rate is often an early overload signal.