cascading failure
A failure in one part of a system that overloads or breaks the parts around it, spreading until much more is down.
In more detail
Typical causes are slow calls tying up every caller, retries multiplying load on a struggling service, and health checks that restart healthy instances. Timeouts, circuit breakers, load shedding and careful probes contain them.
Example
A liveness probe that checks the database restarts every pod when the database blips.Related terms
- circuit breaker — A wrapper around calls to another service that stops making calls for a while after repeated failures, so the failing service can recover.
- retry storm — A flood of retries from many clients at once that hits an already struggling service harder than the original traffic.
- load shedding — Deliberately rejecting some requests when overloaded so the requests that are accepted can still be served well.
- liveness probe — A health check that answers only whether a process is alive enough to keep running; if it keeps failing, the orchestrator, such as Kubernetes, restarts it.
- timeout — A limit on how long a caller waits for an answer before giving up.
- bounded concurrency — Capping how many requests a caller will have in progress to a dependency at once.
Taught in
- Resilience Engineering: Circuit Breakers & Retries — Production Full-Stack Engineering
- Health Probes & Graceful Shutdown: Kubernetes Lifecycles — Production Full-Stack Engineering
- Full-Stack System Verification: Capstone Integration — Production Full-Stack Engineering
- Timeouts, And Avoiding Cascaded Failure — Scalable System Design & Distributed Architecture