NOTE
Designing High-Concurrency Systems
A practical path from reducing work per request to caching, horizontal scaling, partitioning, asynchronous processing, backpressure, and hotspot control.
This is a historical learning note and may contain outdated or incomplete understanding.
1. High Concurrency Is Not One Technique
Start by identifying the bottleneck: CPU, database, storage, network, lock contention, downstream dependency, or queueing.
The same QPS can require very different architectures depending on request cost.
2. Reduce Work
Before scaling out:
- avoid unnecessary calls/queries;
- batch operations;
- eliminate N+1 patterns;
- use efficient indexes/data structures;
- move non-critical work off the synchronous path.
The cheapest request is the one you do not execute.
3. Cache Repeated Reads
Cache data/computation with good reuse and a clear freshness model.
Caching shifts load but introduces invalidation, hot keys, stampedes, and memory cost.
4. Horizontal Scale Stateless Work
Stateless application servers scale behind load balancing when sessions/state live in shared or partitionable stores.
Keep an eye on the downstream bottleneck: adding application instances can simply overload the database faster.
5. Partition Stateful Bottlenecks
Shard by a key that distributes load while preserving required locality/order.
Handle skew and hot partitions explicitly.
6. Async and Buffer Bursts
Queues can absorb spikes and decouple producer speed from consumer capacity, but backlog means latency. Measure queue age/lag, not just queue length.
7. Backpressure and Load Shedding
Bound queues, connections, workers, and retries. Once capacity is exhausted, fail fast or degrade rather than allowing unbounded waiting to collapse the entire service.
8. Optimize Tail Latency
At high concurrency, p99/p999 matters more than averages. Watch lock contention, GC, noisy neighbors, fan-out amplification, and slow dependencies.