Post-Mortem: The Retry Storm That Looked Like a Capacity Problem
A failure analysis of a dependency outage where well-intended retry logic amplified load and delayed recovery.
What Happened
A downstream service began timing out after a deploy increased database query latency. Upstream callers retried failed requests three times with identical timing. The dependency recovered briefly, then collapsed under synchronized retry traffic.
Root Cause
The incident was not caused by retries alone. It was caused by retries without a budget, without jitter, and without a system-level understanding of how many clients would retry at the same time.
What Went Right
The team had dependency-level metrics, so the overload pattern became visible after the first investigation pass.
What Changed
Retries now have deadlines, jitter, and bounded budgets. The calling service sheds noncritical work when dependency latency crosses a threshold.
Lesson
A reliability mechanism can become a failure mechanism when its second-order effects are ignored.