failover
Switching work from a failed or unavailable system to a standby copy so the service keeps running.
In more detail
Failover can be automatic, driven by health checks, or started by people, and it usually involves pointing users at the standby, for example by changing a DNS record. Failback is the return trip once the original system is fixed. Failover protects availability, not data: a standby database still contains any bad data the primary had.
Examples
"hot": {"replica already current": 0, "switch DNS and test": 0.5}Probes and upstream timeouts on the Ingress let it stop sending users to dead pods.Related terms
- multi-AZ — Running a system in two or more availability zones, often as a primary plus a standby copy, so losing one zone does not take it down.
- high availability — Designing a system so it keeps working when one part fails, usually by running redundant copies with no single point of failure.
- read replica — A copy of a database that follows the primary’s changes and can serve read traffic or be promoted to take over.
- hot site — A fully equipped recovery site running the same systems as the main site with continuously replicated data, ready to take over within minutes.
- recovery site — An alternative location where an organisation can run its critical systems and carry on working if its main site becomes unusable.
- disaster recovery — The planned, tested procedure for getting a system and its data back after a serious failure, such as lost storage or a deleted cluster.
- tabletop exercise — A practice session where the response team talks through a realistic incident scenario step by step, to test the plan without touching systems.
- backing service — Any service an application uses over the network, such as a database, cache, queue, mail relay or payments provider.
- resource handle — The address and credentials an app uses to reach a backing service, such as a database URL, kept in config.
- replica — One of several identical copies of a service or database running at the same time.
- readiness probe — A check that decides whether a pod should receive traffic right now; failing it removes the pod from its Service without restarting it.
- load balancing — Spreading requests or connections across several backend servers so no single one is overloaded and failed ones can be avoided.
- leader election — The process by which the members of a distributed system choose one of themselves to act as leader.
- fencing — Cutting a failed or doubtful member of a replicated system off from its data so it cannot keep writing while you recover.
- split brain — A failure in which two nodes both believe they are the leader and both accept writes.
Taught in
- One Database Doing Five Jobs — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- The Public Subnet That Was Not — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Two Hundred Accounts And One Bad Deploy — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- High Availability Across Tiers: What Fails Over By Itself? — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Scaling Patterns: When Capacity Arrives Too Late — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Recovery To RTO And RPO: Backups Nobody Could Delete — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Testing Resilience: The Experiment That Would Have Been The Outage — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Incident Tooling: Eighteen Minutes To Find A Human — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- Long-Stem Clinic: When Two Answers Both Work — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- The Full Mock And Your Review Plan — AWS Certified DevOps Engineer – Professional (DOP-C02) Exam Prep
- The Morning An Availability Zone Went Dark — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Load Balancers And Auto Scaling: Capacity That Arrives On Time — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Databases That Survive: Standbys, Replicas And Global Tables — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Disaster Recovery By The Numbers — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Route 53 Routing: Where Does A Fan In Lagos Land? — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Database Performance: Caches, Proxies And Capacity Units — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Network Performance: Peering, Transit Gateway And The Edge — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- The Performance Clinic: Five Slow Systems, Five Fixes — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Long-Stem Clinic: The Phrase That Decides — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- The Full Mock And Your Review Plan — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Would The X-Rays Survive? Redundancy, Encryption And Tools — Microsoft Azure Administrator (AZ-104) Exam Prep
- Ninety-Nine Percent Full: Disks, Sizes And Availability — Microsoft Azure Administrator (AZ-104) Exam Prep
- Restore It From Last Tuesday: Backup Vaults, Policies And Site Recovery — Microsoft Azure Administrator (AZ-104) Exam Prep
- Seventy-Two Percent, Now What? The Full Mock And Your Review Plan — Microsoft Azure Administrator (AZ-104) Exam Prep
- Storage Accounts, Redundancy And Tiers — Microsoft Azure Fundamentals (AZ-900) Exam Prep
- Look-Alike Services Sprint — Microsoft Azure Fundamentals (AZ-900) Exam Prep
- Managed Relational Databases (RDS) — Cloud Infrastructure & Hyperscaler Architecture
- When to Pick Which Database — Cloud Infrastructure & Hyperscaler Architecture
- Disaster Recovery Sites: Hot, Warm & Cold — Cybersecurity Fundamentals, GRC & Cryptography
- Config And Backing Services — DevOps & Site Reliability Engineering
- Kubernetes Ingress, Routing & TLS Termination — Production Full-Stack Engineering
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
References
- Failover for Multi-AZ DB instances
- NIST glossary: failover
- Kubernetes: Liveness, readiness and startup probes
- PostgreSQL documentation: High Availability, Load Balancing, and Replication
- Azure Architecture Center: Leader Election pattern
Certification courses
- AWS Cloud Practitioner course and practice exam: CLF-C02
- AWS DevOps Engineer Professional course and practice exam: DOP-C02
- AWS Solutions Architect Associate course and practice exam: SAA-C03
- AZ-104 course and practice exam: Microsoft Azure Administrator
- AZ-900 course and practice exam: Microsoft Azure Fundamentals