5.1 Partitioning: Splitting Data

The goal is to map keys to partitions so load and data are distributed acceptably while important queries remain efficient.

1. Choosing a Partition Key

A primary ID often distributes uniformly, but queries by another business field may need scatter/gather. A business key such as user_id can colocate related data and make common queries efficient, but skewed users or tenants can create hot partitions.

Choose the key from the access pattern and skew distribution, not only from uniqueness.

2. Partitioning Strategies

2.1 Explicit Assignment

The caller or application chooses a partition directly. This offers control but pushes placement logic upward.

2.2 Random Partitioning

Writes spread evenly but point reads cannot derive a target partition from the key, so lookup becomes expensive unless another index exists.

2.3 Range Partitioning

Each partition owns a contiguous key range. Range scans are efficient, but monotonically increasing keys can create a hot newest range.

2.4 Hash Partitioning

hash(key) % N

Simple and usually distributes keys well, but changing N remaps a large fraction of keys.

Consistent Hashing

Map both keys and nodes into a stable hash space. Adding/removing a node remaps only portions of the space. Virtual nodes improve balance by assigning many hash positions to each physical node.

Hash Slots

Introduce a fixed logical slot space between keys and physical nodes. Redis Cluster, for example, maps keys into 16,384 hash slots and assigns slots to cluster nodes. Rebalancing moves slots without changing the key-to-slot function.

3. Range vs. Hash

	Range	Hash
Range scans	Efficient	Usually expensive across partitions
Uniformity	Depends strongly on key distribution	Usually better
Hotspot risk	High for sequential/skewed ranges	Still possible for hot keys
Rebalancing	Move/split ranges	Move hash ranges/slots

Consistent hashing improves remapping behavior; it does not solve a single extremely hot key by itself.