5.4 Partition Assignment and Rebalancing

After keys are mapped to logical partitions, those partitions must be placed on physical machines or brokers.

1. Placement Models

1.1 Pre-Created Logical Partitions

Create substantially more logical partitions/slots than physical machines and assign many partitions to each machine. Scaling moves existing partitions between machines instead of redefining the key space.

This makes placement flexible, but the partition count and metadata overhead must be chosen carefully.

1.2 Dynamic Partitioning

The system can split, merge, or create partitions as data and load change. This gives more elasticity but makes rebalancing, metadata, and routing logic more complex.

2. Assignment Goals

Placement commonly considers:

• even storage and traffic distribution;
• replica separation across failure domains;
• machine capacity differences;
• movement cost;
• locality constraints.

3. Rebalancing

When machines join, leave, or become imbalanced, partition ownership changes. Rebalancing must move state while preserving service correctness.

Manual vs. Automatic

Manual rebalancing gives operators precise control but increases operational work. Automatic rebalancing reacts faster but needs safeguards against excessive movement, cascading overload, and unstable placement.

A mature system usually keeps serving at least part of the workload while state migrates rather than treating rebalancing as an all-or-nothing cluster outage.