stream-table duality
The idea that a stream of changes and a table of current values are two views of the same data: fold the stream to get the table, and record the table's changes to get the stream.
In more detail
It explains why replaying a topic can rebuild any view derived from it, and why a compacted topic can act as a table. Stream processing frameworks such as Kafka Streams are built around it.
Related terms
- fold — Combining a sequence of records one by one into a single result, such as the latest value or a running total per key.
- materialised view — A stored table of results computed from other data, kept so queries do not have to recompute them.
- log compaction — A cleanup mode in which the broker keeps at least the latest record for every key and removes older records with the same key.
- Kafka Streams — A Java library, part of Apache Kafka, for building applications that read, transform and write Kafka streams.
Taught in
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Works On A Laptop, Fails In Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Blank Names After Every Deploy — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Hours Of Scans At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Green Tests, Red Production — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Twelve Lines Instead Of Four Hundred — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Connector That Was Created Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Log Compaction: A Topic As A Table — Apache Kafka & Event Streaming
- Materialised Views From A Stream — Apache Kafka & Event Streaming