thundering herd
A flood of requests that arrives at once when many clients react to the same event, such as all retrying after an outage.
In more detail
It can knock over a service that was just recovering. Backoff with jitter, retry limits and rate limiting all spread the load out.
Related terms
- jitter — A random amount added to or taken from a retry delay so that many clients do not all retry at the same moment.
- backoff — Waiting longer and longer between successive retries, often doubling the delay each time, so a struggling service has room to recover.
- rate limiting — Capping how many requests a client may make in a period of time, and rejecting or delaying the rest.
- singleflight — A technique that merges many simultaneous identical requests into one, so only one does the work and they all share its result.
- mutex — A lock that lets only one piece of code at a time run a critical section or do a particular piece of work.
- single flight — Letting only one request fetch a missing value while other requests for the same key wait for its result.
- admission control — Deciding at the entrance whether a request may start, based on how much work the system can still take.
- retry storm — A flood of retries from many clients at once that hits an already struggling service harder than the original traffic.
Taught in
- Error Handling And Safe Retries — Modern API Architecture: REST, GraphQL & gRPC
- Rate Limiting Strategies — Modern API Architecture: REST, GraphQL & gRPC
- Bidirectional WebSockets: Reconnection & Heartbeats — Production Full-Stack Engineering
- Distributed Cache Invalidation: Redis & Tagged Purging — Production Full-Stack Engineering
- Cache Invalidation And The Three Hard Cases — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture