duplicate record
A second copy of a record that was meant to be written or processed only once.
In more detail
Duplicates come from producer retries after a lost acknowledgement, from consumers that crash before committing, and from deliberate replays. Kafka's default at-least-once delivery accepts them, so consumers must deduplicate or be idempotent.
Related terms
- at-least-once delivery — A delivery guarantee under which every message arrives, but some may arrive more than once.
- deduplication — Detecting and discarding repeated copies of the same message or request, usually by remembering identifiers already processed.
- idempotency — The property of an operation that gives the same result whether you run it once or many times, because a repeat run finds nothing left to do.
- idempotent producer — A producer mode in which the broker detects and discards duplicate batches caused by retries, so each record is written once per partition.
Taught in
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Idempotent Producers And Retries — Apache Kafka & Event Streaming
- Duplicates Are The Default — Apache Kafka & Event Streaming
- Idempotent Consumers And Deduplication Keys — Apache Kafka & Event Streaming