partitioner
The producer component that chooses which partition each record is sent to.
In more detail
The default partitioner hashes a record's key with murmur2 and takes the result modulo the partition count. kafka-python sends unkeyed records to a random available partition, while the Java client uses sticky partitioning. You can pass your own function, for example to route by region.
Example
KafkaProducer(partitioner=region_router)Related terms
- murmur2 — A fast, non-cryptographic hash function that Kafka clients use to turn a record key into a partition number.
- record key — An optional value attached to a record that decides which partition it goes to, so all records with the same key stay together and in order.
- modulo — The operation that gives the remainder after division, written % in many programming languages, so 17 % 5 is 2.
- sticky partitioning — A strategy for unkeyed records that keeps sending to one partition until the current batch is full, then moves on to another.
- round robin — A way of spreading work by taking turns in a fixed order, giving the next item to the next recipient in the list.
Taught in
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Topics, Partitions And Where A Record Lands — Apache Kafka & Event Streaming
- Keys And Ordering Per Partition — Apache Kafka & Event Streaming