retry
Sending a request again after it failed or timed out, in the hope that it succeeds the second time.
In more detail
Retry only when the operation is safe or idempotent, or protected by an idempotency key, because a timed-out create may already have happened. Retry 5xx and 429, not other 4xx. Bound retries with a maximum count or deadline and space them with backoff and jitter, and count them separately so a rising retry rate is visible.
Examples
In AWS: Trying a failed operation again, automatically or by hand, in the hope that the failure was temporary.A Kafka producer retry after a lost acknowledgement writes the record twice unless the producer is idempotent.KafkaProducer(retries=0)A retry that eventually succeeds is logged at WARN, not ERROR.In this course: Retries are a real design choice for network calls and belong in review questions: what happens on timeout, retry and duplicate input. Retrying a flaky test until it passes is different: it hides the cause and spends build time.Related terms
- backoff — Waiting longer and longer between successive retries, often doubling the delay each time, so a struggling service has room to recover.
- jitter — A random amount added to or taken from a retry delay so that many clients do not all retry at the same moment.
- idempotency — The property of an operation that gives the same result whether you run it once or many times, because a repeat run finds nothing left to do.
- idempotency key — A unique value a client sends with a non-idempotent request, such as a create, so the server can recognise a retry and return the original result instead of acting twice.
- thundering herd — A flood of requests that arrives at once when many clients react to the same event, such as all retrying after an outage.
- dead letter queue — A separate queue where messages that keep failing are moved, so they stop blocking others and can be inspected and replayed later.
- throttling — A service refusing or delaying requests because a rate or concurrency limit has been reached.
- idempotent producer — A producer mode in which the broker detects and discards duplicate batches caused by retries, so each record is written once per partition.
- duplicate record — A second copy of a record that was meant to be written or processed only once.
- retry topic — A topic that holds records whose processing failed so they can be tried again later without blocking the main topic.
- exponential backoff — Waiting progressively longer between retries, for example 1, 2, then 4 seconds, so a struggling service gets room to recover.
- transient fault — A temporary failure, such as a dropped connection or a brief overload, that is likely to succeed if tried again shortly.
- circuit breaker — A wrapper around calls to another service that stops making calls for a while after repeated failures, so the failing service can recover.
- retry storm — A flood of retries from many clients at once that hits an already struggling service harder than the original traffic.
- transient failure — A failure caused by a temporary condition, such as a brief network glitch, that may succeed if simply tried again.
- timeout — A limit on how long a caller waits for an answer before giving up.
- WARN — The log level for a problem the system recovered from by itself, such as a retry that succeeded or a fallback that was used.
- exporter — A component that sends telemetry out to a backend; in the collector it is the last stage of a pipeline.
- retry budget — A cap on retries measured as a fraction of normal traffic, so retries cannot overwhelm a failing dependency.
- flaky test — A test that sometimes passes and sometimes fails on the same code, because it depends on something it does not control.
Taught in
- What An AI Engineer Does — AI Engineering: LLM Apps, RAG & Agents
- Streaming Responses — AI Engineering: LLM Apps, RAG & Agents
- When The Call Fails — AI Engineering: LLM Apps, RAG & Agents
- Structured Output — AI Engineering: LLM Apps, RAG & Agents
- Designing Tools A Model Can Use — AI Engineering: LLM Apps, RAG & Agents
- The Agent Loop — AI Engineering: LLM Apps, RAG & Agents
- Workflows Before Agents — AI Engineering: LLM Apps, RAG & Agents
- Build An MCP Server — AI Engineering: LLM Apps, RAG & Agents
- Connect An Agent To MCP — AI Engineering: LLM Apps, RAG & Agents
- Excessive Agency — AI Engineering: LLM Apps, RAG & Agents
- Working With AI Coding Assistants — AI Engineering: LLM Apps, RAG & Agents
- Capstone: The Tasks Assistant — AI Engineering: LLM Apps, RAG & Agents
- Error Handling And Safe Retries — Modern API Architecture: REST, GraphQL & gRPC
- Idempotency Keys — Modern API Architecture: REST, GraphQL & gRPC
- Webhooks Versus Polling — Modern API Architecture: REST, GraphQL & gRPC
- API Gateways And Proxies — Modern API Architecture: REST, GraphQL & gRPC
- Four Hundred Missing Labels — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Decoupling: SQS, SNS, EventBridge And Step Functions — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Serverless And Container Patterns: Where The Limits Bite — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Databases That Survive: Standbys, Replicas And Global Tables — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Storage Performance: IOPS, Throughput And The Right Volume — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Compute Cost: Commit, Interrupt Or Pay As You Go — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- The Full Mock And Your Review Plan — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Serverless Compute (Lambda) — Cloud Infrastructure & Hyperscaler Architecture
- Comparing Compute Options — Cloud Infrastructure & Hyperscaler Architecture
- Auto Scaling Groups — Cloud Infrastructure & Hyperscaler Architecture
- How Bills Actually Get Large — Cloud Infrastructure & Hyperscaler Architecture
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twelve Lines Instead Of Four Hundred — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retry and Circuit Breaker Policies with Polly — Modern .NET Core, C# & Enterprise Microservices
- Bidirectional WebSockets: Reconnection & Heartbeats — Production Full-Stack Engineering
- Asynchronous Event Processing: Outbox Pattern & Queues — Production Full-Stack Engineering
- Resilience Engineering: Circuit Breakers & Retries — Production Full-Stack Engineering
- Full-Stack System Verification: Capstone Integration — Production Full-Stack Engineering
- Checked Versus Unchecked Exceptions — Modern Java: Virtual Threads & High-Throughput Services
- The Cost Of Checked Exceptions — Modern Java: Virtual Threads & High-Throughput Services
- Acknowledgements: What Acks Means For Durability — Apache Kafka & Event Streaming
- Idempotent Producers And Retries — Apache Kafka & Event Streaming
- Duplicates Are The Default — Apache Kafka & Event Streaming
- Retries With Backoff Topics — Apache Kafka & Event Streaming
- Log Levels Done Properly — Full-Stack Observability: Metrics, Tracing & Logging
- Why A Histogram Beats An Average — Full-Stack Observability: Metrics, Tracing & Logging
- Prometheus Scraping And The Pull Model — Full-Stack Observability: Metrics, Tracing & Logging
- The OpenTelemetry Collector — Full-Stack Observability: Metrics, Tracing & Logging
- Alerting On Symptoms Versus Causes — Full-Stack Observability: Metrics, Tracing & Logging
- Latency Numbers Every Engineer Should Know — Scalable System Design & Distributed Architecture
- The Storage Decision: Key Value, Search And Blob — Scalable System Design & Distributed Architecture
- Partitioning, Sharding And Hot Keys — Scalable System Design & Distributed Architecture
- Eventual Consistency And What It Costs A Product — Scalable System Design & Distributed Architecture
- Caching Strategies And Topology — Scalable System Design & Distributed Architecture
- Idempotency, And Exactly Once As A Fiction — Scalable System Design & Distributed Architecture
- Load Balancing And The Failure Modes It Introduces — Scalable System Design & Distributed Architecture
- Timeouts, And Avoiding Cascaded Failure — Scalable System Design & Distributed Architecture
- Retries, Backoff And Jitter — Scalable System Design & Distributed Architecture
- Backpressure And Load Shedding — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
- Capacity Planning As A Design Input — Scalable System Design & Distributed Architecture
- Worked Design One: Ticket Booking, Correctness First — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Two: A Global Chat System — Scalable System Design & Distributed Architecture
- Worked Design Two: Scaling Chat Fan Out — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Flaky Tests And How To Kill Them — Automated Testing, TDD & Quality Engineering
- What To Look For In A Review — Automated Testing, TDD & Quality Engineering
- Writing Actionable Review Comments — Automated Testing, TDD & Quality Engineering
References
- Amazon Builders’ Library: Timeouts, retries and backoff with jitter
- AWS Prescriptive Guidance: Retry with backoff pattern
- Apache Kafka producer configs: retries
- Retry pattern
- Polly: retry strategy
- Java Tutorials: Exceptions
- RFC 9110: HTTP semantics
- Google SRE book: handling overload (retries)
- pytest: flaky tests