append-only log
A data structure where new entries are only ever added at the end and existing entries are never changed in place.
In more detail
Event streaming platforms such as Kafka, database write-ahead logs and audit trails are all built on append-only logs. Because reading never removes anything, many readers can replay the same entries at their own pace, unlike a queue that deletes a message once it is taken. Old entries are usually removed only in bulk, by a retention or compaction policy.
Related terms
- message queue — A store that holds messages from producers until consumers are ready to process them, decoupling the rate of arrival from the rate of work.
- partition — One of several independent ordered logs a message stream is split into, with each message routed to one partition by its key.
- offset — The permanent position number of a record inside its partition, starting at zero and increasing by one for each record.
- retention — How long data such as backups, logs or snapshots is kept before it is deleted.
- immutable — Something that cannot be changed after it is created; to change it you make a new one.
- partition — One slice of a larger dataset or message stream, owned and processed independently of the others.
- replay — Delivering or returning something again: a stored result for a repeated request, or past events for a client that missed them.
Taught in
- Ninety Minutes, No Question Count — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where A Record Lands — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Retry That Overtook — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Alert That Came An Hour Late — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Stuck On One Record — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Connector That Was Created Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- A Sink That Died On Record Nine — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Setting That Came Back — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Brokers Down — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Certificate That Expired On A Sunday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Where Is The Password Kept? — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Upgrade That Could Not Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Controllers At Once — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Seventy One Percent — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- Offsets: Position, Not Acknowledgement — Apache Kafka & Event Streaming
- Log Compaction: A Topic As A Table — Apache Kafka & Event Streaming
- Materialised Views From A Stream — Apache Kafka & Event Streaming
- Back Of The Envelope Estimation: Storage And Bandwidth — Scalable System Design & Distributed Architecture
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Monolith Versus Services: The Honest Trade Offs — Scalable System Design & Distributed Architecture
- Observability As A Design Input — Scalable System Design & Distributed Architecture
References
- Apache Kafka design: persistence
- Apache Kafka implementation: log
- Apache Kafka documentation: Design
- Patterns of Distributed Systems: Write-Ahead Log