log compaction
A cleanup mode in which the broker keeps at least the latest record for every key and removes older records with the same key.
In more detail
A compacted topic behaves like a table of the latest value per key, so reading it from offset zero rebuilds the current state. Compaction runs in the background on closed segments, so older values stay visible for a while.
Related terms
- cleanup.policy — The topic property that chooses how old records are removed: delete by retention, compact by key, or both.
- record key — An optional value attached to a record that decides which partition it goes to, so all records with the same key stay together and in order.
- min.cleanable.dirty.ratio — The topic property that sets how much of a log must be uncompacted before the cleaner compacts it.
- log segment — One file in the series of files that together make up a partition's log on disk.
- stream-table duality — The idea that a stream of changes and a table of current values are two views of the same data: fold the stream to get the table, and record the table's changes to get the stream.
- append-only log — A data structure where new entries are only ever added at the end and existing entries are never changed in place.
- write amplification — When one logical write causes the storage engine to write the data several times over.
- retention — How long data such as backups, logs or snapshots is kept before it is deleted.
Taught in
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Connector That Was Created Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Log Compaction: A Topic As A Table — Apache Kafka & Event Streaming
- Materialised Views From A Stream — Apache Kafka & Event Streaming
- Back Of The Envelope Estimation: Storage And Bandwidth — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
References
- Apache Kafka design: log compaction
- Redpanda compaction settings
- Apache Kafka documentation: Log compaction
- RocksDB wiki: Leveled Compaction