producer
The program that creates and sends messages or events.
In more detail
A fast producer can outrun a slow consumer, which is why queues need bounds and backpressure. Producers should give each message an identifier so consumers can detect duplicates.
Example
Kafka producers pick a partition per record and wait for the acknowledgement their acks setting asks for.Related terms
- consumer — A program that receives and processes messages or events, or more broadly any client that uses an API.
- event — A record of something that has already happened, such as task completed, with an identifier, a time, a type and its data.
- backpressure — A slow consumer signalling a fast producer to pause, so data does not pile up in memory faster than it can be handled.
- KafkaProducer — The kafka-python class that buffers records, batches them and sends them to Kafka in the background.
- acks — The producer setting that decides how many brokers must confirm a record before the send counts as successful: 0, 1 or all.
- partitioner — The producer component that chooses which partition each record is sent to.
- record batch — A group of records sent to the broker together in one request, instead of one request per record.
- message queue — A store that holds messages from producers until consumers are ready to process them, decoupling the rate of arrival from the rate of work.
Taught in
- WebSockets — Modern API Architecture: REST, GraphQL & gRPC
- Event-Driven Architecture And Queues — Modern API Architecture: REST, GraphQL & gRPC
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Acknowledged Is Not Always Safe — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twenty Seconds Of Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Field The Registry Refused — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Hours Of Scans At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Setting That Came Back — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The New Broker That Did Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Allowed To Read, Refused Anyway — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- A Producer In Python — Apache Kafka & Event Streaming
- Keys And Ordering Per Partition — Apache Kafka & Event Streaming
- Acknowledgements: What Acks Means For Durability — Apache Kafka & Event Streaming
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- Backpressure And Load Shedding — Scalable System Design & Distributed Architecture
References
- Apache Kafka: Introduction
- Apache Kafka design: the producer
- kafka-python KafkaProducer
- Apache Kafka documentation
- Amazon SQS Developer Guide