idempotent producer
A producer mode in which the broker detects and discards duplicate batches caused by retries, so each record is written once per partition.
In more detail
The broker gives the producer a producer ID, and the producer stamps every batch with a sequence number. A retried batch with a sequence the broker has already written is acknowledged but not stored again. It protects only against retries from one producer session, not against your code sending the same data twice.
Example
KafkaProducer(acks='all', enable_idempotence=True, max_in_flight_requests_per_connection=1)Related terms
- producer ID — A number the broker assigns to an idempotent or transactional producer so it can recognise that producer's batches.
- sequence number — An increasing number attached to each message so the receiver can detect gaps, reorder messages and say where to resume.
- retry — Sending a request again after it failed or timed out, in the hope that it succeeds the second time.
- max.in.flight.requests.per.connection — The producer setting for how many unacknowledged requests may be outstanding on one broker connection at the same time.
- idempotency — The property of an operation that gives the same result whether you run it once or many times, because a repeat run finds nothing left to do.
Taught in
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Idempotent Producers And Retries — Apache Kafka & Event Streaming
- Duplicates Are The Default — Apache Kafka & Event Streaming