Home › Scenarios › API rate limiting
System design scenario
Design an API rate limiter
Intermediate
A burst of client traffic pushes the API past capacity. Queues grow, requests time out, clients retry, and the whole API collapses even though it is only ~10% short.
▶ Simulate this designThe goal
Keep p99 < 300 ms and error rate < 20.00% at 100,000 req/s for 10s.
- Peak load
- 100,000 requests per second
- Traffic mix
- 80% reads, 20% writes
- p99 latency
- under 300 ms
- Error rate
- under 20.00%
The starting architecture
The simulator opens with this design. It is deliberately too weak for the goal: run it and watch which component saturates first.
- API gateway (Load Balancer, 2 instances): API gateway: the right place to enforce rate limits and shed excess load early.
- Public API (Service, 30 instances): Public API with a deep queue: overload turns into long waits, timeouts and retries.
- API data (Database): Backing data store with plenty of headroom.
How to approach it
The hints unlock one at a time in the app. Try the design yourself before reading them.
Show the hints
- Watch what happens to latency and retries when the spike arrives: this is congestion collapse.
- Failing some requests fast is better than failing all of them slowly: add a rate limit at the gateway.
- Retries make overload worse: give clients a retry budget and backoff.
What the simulator shows you
SimLoad models every component as a queue with real capacity, latency, timeouts and retries, so you see utilisation, p50/p99 latency, errors and monthly cost change as you scale. Insights name the bottleneck in plain language, and the capacity planner sizes the fix. A reference solution that meets the goal is verified by automated tests.
▶ Try it in the simulator