retention
How long data such as backups, logs or snapshots is kept before it is deleted.
In more detail
Retention is set by recovery and compliance needs, and it drives cost: kept data is billed every month. Write the period down and enforce it with lifecycle rules or retention settings.
Example
Kafka topics use retention.ms and retention.bytes and delete whole closed segments, never single records.Related terms
- lifecycle rule — A rule on a bucket or other store that automatically moves data to cheaper storage or deletes it after a set age.
- log retention — How long a log group keeps its entries before deleting them.
- backup and restore — Keeping recoverable copies of data from past points in time, and being able to bring them back.
- retention.ms — The topic property that sets the maximum age, in milliseconds, of records before their segment becomes eligible for deletion.
- retention.bytes — The topic property that caps how many bytes each partition may keep before the oldest segments are deleted.
- log segment — One file in the series of files that together make up a partition's log on disk.
- cleanup.policy — The topic property that chooses how old records are removed: delete by retention, compact by key, or both.
- append-only log — A data structure where new entries are only ever added at the end and existing entries are never changed in place.
- replay — Delivering or returning something again: a stored result for a repeated request, or past events for a client that missed them.
- tiered retention — Keeping recent data in fast storage and older data in cheaper, slower, often downsampled storage.
Taught in
- Seven Ways To Leave A Rack — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Ten Terabytes Nobody Reads — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Technology Sprint: The Design Review — AWS Certified Cloud Practitioner (CLF-C02) Exam Prep
- Protecting Data: Versions, Object Lock, Backups And Secrets — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Decoupling: SQS, SNS, EventBridge And Step Functions — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Databases That Survive: Standbys, Replicas And Global Tables — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Ingest And Transform: Streams, Lakes And Partitions — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Database Cost: Serverless, Capacity Modes And Reservations — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Long-Stem Clinic: The Phrase That Decides — AWS Certified Solutions Architect – Associate (SAA-C03) Exam Prep
- Object, Block, and File Storage — Cloud Infrastructure & Hyperscaler Architecture
- S3: Buckets and Objects — Cloud Infrastructure & Hyperscaler Architecture
- S3: Permissions, Tiers, and Policies — Cloud Infrastructure & Hyperscaler Architecture
- Elastic Block Store (EBS) — Cloud Infrastructure & Hyperscaler Architecture
- Elastic File System (EFS) — Cloud Infrastructure & Hyperscaler Architecture
- Managed Relational Databases (RDS) — Cloud Infrastructure & Hyperscaler Architecture
- When to Pick Which Database — Cloud Infrastructure & Hyperscaler Architecture
- Observability with CloudWatch — Cloud Infrastructure & Hyperscaler Architecture
- Cost, Budgets, and Tagging — Cloud Infrastructure & Hyperscaler Architecture
- How Bills Actually Get Large — Cloud Infrastructure & Hyperscaler Architecture
- Infrastructure as Code (Handover to Terraform) — Cloud Infrastructure & Hyperscaler Architecture
- The Payment That Was Charged Twice — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Queue That Forgot — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Error Storm That Fixes Itself — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Replay Everything Since Eleven — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Two Copies Of One Service — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Too Slow Between Two Polls — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Count Again — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Eleven Thousand Scans That Never Arrived — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Forty Minutes To Start — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Full Disks On Black Friday — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- The Night Before The Administrator Exam — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- Why A Log And Not A Queue — Apache Kafka & Event Streaming
- Retention: Time, Size And Why Data Stays — Apache Kafka & Event Streaming
- Log Compaction: A Topic As A Table — Apache Kafka & Event Streaming
- Back Of The Envelope Estimation: Storage And Bandwidth — Scalable System Design & Distributed Architecture
- The Difference Between A Queue And A Log — Scalable System Design & Distributed Architecture
- Worked Design Three: A Metrics Ingestion Pipeline — Scalable System Design & Distributed Architecture
- Worked Design Three: Scaling The Metrics Pipeline — Scalable System Design & Distributed Architecture
References
- AWS Well-Architected Cost Optimization pillar: manage data lifecycle
- Apache Kafka topic configs: retention.ms
- Redpanda: manage disk utilization
- Apache Kafka documentation: Design
- Prometheus documentation: Storage