partitioning
Splitting a dataset into separate pieces, each owned by a different machine or process, so the work can spread out.
In more detail
Partitioning moves a limit rather than removing it: each piece still has a ceiling, and work that touches every piece becomes a fan out. The partition key decides which records live together, and therefore what stays fast and ordered.
Related terms
- partition — One slice of a larger dataset or message stream, owned and processed independently of the others.
- partition key — The DynamoDB attribute whose value decides which storage partition an item lives in, and which every query must name.
- shard — One piece of a database that has been split across machines, holding a subset of the data.
- hot partition — A partition that gets far more traffic than the others because many requests use the same key, causing throttling even when total capacity is free.
Taught in
- Back Of The Envelope Estimation: Traffic And Throughput — Scalable System Design & Distributed Architecture
- Replication, Failover And Leader Election — Scalable System Design & Distributed Architecture
- Partitioning, Sharding And Hot Keys — Scalable System Design & Distributed Architecture
- Load Balancing And The Failure Modes It Introduces — Scalable System Design & Distributed Architecture
- Worked Design One: Scaling The Ticket Booking System — Scalable System Design & Distributed Architecture
- Worked Design Two: A Global Chat System — Scalable System Design & Distributed Architecture