consumer
A program that receives and processes messages or events, or more broadly any client that uses an API.
In more detail
Message consumers must be idempotent, because at-least-once delivery sends some messages more than once. In contract testing, a consumer is a caller whose needs the provider must keep meeting.
Example
Kafka consumers pull records after their current offset and store progress as committed offsets.Related terms
- producer — The program that creates and sends messages or events.
- idempotent consumer — A message or webhook handler that produces the same result whether it sees a message once or several times.
- consumer-driven contract testing — A contract-testing approach in which each consumer records the requests and responses it depends on, and the provider verifies it still satisfies all of them.
- KafkaConsumer — The kafka-python class that reads records from Kafka topics, joins consumer groups and commits offsets.
- consumer group — A set of consumers sharing one group ID that divide a topic's partitions between them so each partition is read by only one member at a time.
- committed offset — The offset a consumer group has saved on the broker for a partition, marking the next record it should read after a restart.
- poll loop — The loop in which a consumer repeatedly asks the broker for new records and processes them.
- offset — The permanent position number of a record inside its partition, starting at zero and increasing by one for each record.
Taught in
- Contract Testing — Modern API Architecture: REST, GraphQL & gRPC
- Event-Driven Architecture And Queues — Modern API Architecture: REST, GraphQL & gRPC
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Acknowledged Is Not Always Safe — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twenty Seconds Of Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Field The Registry Refused — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Blank Names After Every Deploy — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Hours Of Scans At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twelve Lines Instead Of Four Hundred — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Setting That Came Back — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The New Broker That Did Nothing — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Allowed To Read, Refused Anyway — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty One Under-Replicated Partitions — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Data Centre That Went Dark — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- Offsets: Position, Not Acknowledgement — Apache Kafka & Event Streaming
- A Consumer In Python — Apache Kafka & Event Streaming
- Consumer Groups And Partition Assignment — Apache Kafka & Event Streaming
- Queues, Events And Asynchronous Work — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Idempotency, And Exactly Once As A Fiction — Scalable System Design & Distributed Architecture
- Backpressure And Load Shedding — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture
References
- Apache Kafka: Introduction
- Pact docs: Consumer-driven contracts
- Apache Kafka design: the consumer
- kafka-python KafkaConsumer
- Apache Kafka documentation
- Amazon SQS Developer Guide