retry storm
A flood of retries from many clients at once that hits an already struggling service harder than the original traffic.
In more detail
If every layer retries three times, one user action can become dozens of calls to the failing service. Retry budgets, backoff with jitter and circuit breakers prevent it.
Example
A client retrying three times over a library that retries three times makes nine attempts per request.Related terms
- retry — Sending a request again after it failed or timed out, in the hope that it succeeds the second time.
- circuit breaker — A wrapper around calls to another service that stops making calls for a while after repeated failures, so the failing service can recover.
- cascading failure — A failure in one part of a system that overloads or breaks the parts around it, spreading until much more is down.
- jitter — A random amount added to or taken from a retry delay so that many clients do not all retry at the same moment.
- retry budget — A cap on retries measured as a fraction of normal traffic, so retries cannot overwhelm a failing dependency.
- thundering herd — A flood of requests that arrives at once when many clients react to the same event, such as all retrying after an outage.
Taught in
- Resilience Engineering: Circuit Breakers & Retries — Production Full-Stack Engineering
- Retries, Backoff And Jitter — Scalable System Design & Distributed Architecture
- Capacity Planning As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture