Designing a Rate Limiter

1. Rate vs. Concurrency

A rate limiter controls work over time, such as requests/second. A semaphore/concurrency limiter controls simultaneous in-flight work.

Many systems need both.

2. Fixed Window

Count requests in a time bucket. Simple and cheap but allows boundary bursts approaching roughly twice the nominal rate over a short interval.

3. Sliding Window

Track or approximate activity over the moving time interval. More accurate at boundaries but more expensive.

4. Token Bucket

Tokens accumulate at a configured refill rate up to a burst capacity. Requests consume tokens.

This is often a good default because it enforces long-term rate while allowing controlled bursts.

5. Leaky Bucket / Queue Shaping

Queue work and drain at a controlled rate. This smooths output but converts excess load into waiting latency until the queue limit is reached.

6. Distributed Limiting

A global limiter needs shared/partitioned state, often implemented with Redis/another coordinator and atomic operations.

At very high QPS, use hierarchical limits:

local fast limiter → regional/global quota reconciliation

This reduces shared-store traffic.

7. Choose the Dimension

Limit by the resource you protect:

• user/tenant/API key;
• endpoint;
• IP/device;
• expensive downstream operation.

Weighted requests may consume different token amounts.

8. Failure Policy

Decide whether limiter failure is fail-open or fail-closed. Security/financial limits often prefer closed; protective overload limits may prefer carefully bounded open behavior.

Return clear retry metadata such as Retry-After where applicable.