Rate Limiting
Deciding how much traffic each caller is allowed — and rejecting the rest gracefully.
Introduction
Rate limiting caps how many requests a client can make in a given time window. It protects a system from abuse, runaway clients, and accidental overload, and it enforces fair usage across many callers. When the limit is exceeded, the system responds with 429 Too Many Requests rather than falling over.
Why it exists
Shared services have finite capacity. A single buggy client in a retry loop, a scraper, or a credential-stuffing bot can consume all of it and starve everyone else. Rate limiting exists to protect availability, ensure fairness, control cost (especially for expensive endpoints), and provide a first line of defense against abuse and denial-of-service.
Analogy
It's a nightclub with a capacity and a bouncer. The bouncer lets people in at a controlled rate; when it's full, newcomers wait or are turned away. The club stays safe and enjoyable instead of dangerously overcrowded. Different clients (VIPs, general admission) can have different limits.
How it works
A limiter maintains a counter per key (user, API key, or IP) and decides allow/deny on each request. The algorithm determines the shape of the limit:
Fixed window counts requests per calendar window (e.g. per minute); simple but allows a 2x burst across the window boundary. Sliding window counts over the trailing period; smoother but needs more state. Token bucket refills tokens at a steady rate and lets a caller spend a burst up to the bucket size — the most common choice because it permits bursts while bounding the average. Leaky bucket processes at a constant rate, smoothing output like a queue.
In a distributed system the counter must be shared and atomic so limits hold across all instances — typically Redis with INCR/Lua scripts. On rejection, return 429 with a Retry-After header so well-behaved clients back off. Layered limiting (per-IP at the edge, per-user at the app, per-endpoint for expensive routes) is common.
- 1
A request arrives; the limiter computes the key (user / API key / IP).
- 2
It atomically updates the shared counter or token bucket for that key.
- 3
If under the limit, the request proceeds and the remaining quota is returned in headers.
- 4
If over the limit, it returns 429 with Retry-After and does not touch downstream services.
- 5
The window/bucket refills over time, restoring capacity.
Interactive
Architecture
Distributed limiter with a shared counter
Edge
App instances (share one counter)
Atomic counter
The counter lives in Redis so the limit is enforced across every app instance, not per instance.
In code
// Refill 'rate' tokens/sec up to 'capacity'; each request costs 1 token.
const TOKEN_BUCKET = `
local key = KEYS[1]
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local bucket = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(bucket[1]) or capacity
local ts = tonumber(bucket[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * rate) -- refill since last check
local allowed = tokens >= 1
if allowed then tokens = tokens - 1 end
redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', key, math.ceil(capacity / rate) + 1)
return allowed and 1 or 0
`
export async function allow(userId: string) {
const res = await redis.eval(TOKEN_BUCKET, 1, `tb:${userId}`, "5", "100", String(Date.now() / 1000))
return res === 1
}export async function middleware(req: Request) {
const key = getClientKey(req)
if (!(await allow(key))) {
return new Response("Too Many Requests", {
status: 429,
headers: { "Retry-After": "5", "X-RateLimit-Limit": "100" },
})
}
// ...proceed
}In the real world
Where you've seen this
Public APIs (GitHub, Stripe, Twitter) publish rate limits and return 429 with X-RateLimit-Remaining/Retry-After headers. Login endpoints are limited per-IP and per-account to blunt credential stuffing. A CDN/WAF layer rate-limits at the edge before traffic ever reaches the origin, shedding abusive load cheaply.
When to use it
Reach for it when
- Protecting shared or expensive endpoints from overload and abuse.
- Enforcing per-plan quotas on a public or internal API.
- Blunting brute-force and credential-stuffing on auth endpoints.
- Providing backpressure so a spike degrades gracefully instead of collapsing.
Avoid it when
- Internal, trusted, low-volume calls where limiting adds latency for no benefit.
- As a substitute for actual capacity when demand is legitimately high (scale instead).
- When strict global exactness would cost more than the abuse it prevents (approximate is fine).
Trade-offs
Advantages
- Preserves availability and fairness under load and abuse.
- Controls cost on expensive operations.
- First-line defense against brute force and basic DoS.
- Gives clients predictable, cooperative backpressure (429 + Retry-After).
Disadvantages
- A shared atomic counter can become a hot key / bottleneck at scale.
- Distributed exactness vs performance is a real trade-off.
- Misconfiguration can block legitimate users (false positives).
- Adds a dependency (e.g. Redis) whose failure needs a fail-open/closed policy.
The central trade-off is accuracy vs performance in a distributed setting: a single shared counter is exact but a potential hot spot; sharded or local counters scale but only approximate the global limit. Algorithms trade burst-friendliness vs smoothness (token bucket allows bursts; leaky bucket smooths). Where you enforce it trades cost vs precision: edge limiting is cheap and coarse; app-level limiting is precise but later and more expensive. And the Redis dependency forces an explicit fail-open vs fail-closed decision.
| Rate-limiting algorithms | Fixed window | Sliding window | Token bucket | Leaky bucket |
|---|---|---|---|---|
| Allows bursts | At edges (2x) | No | Yes (up to capacity) | No (smooths) |
| State/memory | Tiny | Higher (timestamps) | Small | Small (queue) |
| Accuracy | Coarse | High | Good | High (rate) |
| Best for | Simple quotas | Precise rolling limits | APIs that want bursts | Steady downstream rate |
Common mistakes
Watch out for
- Counting per instance instead of using a shared counter, so the real limit is N× intended.
- Fixed windows allowing double the limit across the boundary when that matters.
- Limiting by IP only, punishing users behind shared NAT/proxies.
- Returning a generic error instead of 429 with Retry-After.
- No decision for 'what if Redis is down' — silently failing open or closed.
Failure thinking
What breaks it
Your rate limiter's Redis becomes unreachable. What should happen?
You must decide fail-open (allow traffic, risking overload) or fail-closed (reject, risking a self-inflicted outage) — and it should be deliberate, not accidental. Often the answer is fail-open with a local in-process fallback limiter so you keep coarse protection without a hard dependency, plus alerting. The wrong outcome is discovering the behavior during an incident.
One global counter for a hugely popular API becomes a hot key. What breaks?
All limiter traffic hits one Redis slot/thread, bottlenecking throughput. Mitigate by sharding the limit across keys (approximate, e.g. per-region sub-limits), doing local pre-checks with periodic reconciliation, or accepting slightly looser accuracy for far higher throughput. Perfect global accuracy and extreme scale are in tension.
Think like a senior
Senior Engineer Insight
Layer your limits: cheap per-IP shedding at the edge, precise per-user/per-endpoint limits at the app. One global limiter is both a hot key and too blunt for real policies.
Senior Engineer Insight
Always specify the failure behavior. 'What happens to the limiter when Redis is down?' is the question that separates someone who copied a snippet from someone who has run this in production.
Remember
In a distributed system the counter must be shared and atomic, or your limit is really N× per-instance.
Remember
Reject with 429 + Retry-After so clients back off cooperatively.
Interview questions
Compare fixed window, sliding window, and token bucket.
Active recall
Check yourself
Which algorithm allows short bursts up to a capacity while enforcing a steady average rate?
Practical challenges
In-memory limiter, then Redis, then compare algorithms
Build a limiter three ways and observe their differences under a burst.
Requirements
- Implement an in-memory fixed-window limiter.
- Move the counter to Redis so it works across instances.
- Add sliding-window and token-bucket variants behind one interface.
Constraints
- Redis operations must be atomic (INCR or Lua).
- Return 429 with Retry-After on rejection.
Acceptance criteria
- The in-memory version fails to limit correctly across two instances; the Redis version succeeds.
- A burst spanning a window boundary passes fixed-window but is caught by sliding-window.
- Token bucket allows a bounded burst then throttles to the refill rate.
Edge cases
- Redis unavailable — pick and implement fail-open or fail-closed.
- Clock skew for sliding window.
Bonus
- Add per-tier limits and per-endpoint overrides.
Reflection
- Which algorithm best matches real API traffic, and why?
Architecture challenge
Design rate limiting for a public API with free (100 req/min) and paid (10,000 req/min) tiers, plus a very expensive /export endpoint (5/hour) and a login endpoint that must resist credential stuffing. Choose algorithms and keys per case, decide where each limit is enforced, and define behavior when the shared counter store is unreachable.
Flashcards
Summary
Rate limiting caps per-caller traffic to protect availability, enforce fairness, and blunt abuse, rejecting excess with 429 + Retry-After. Token bucket is the common default (bursts up to a capacity, bounded average); distributed correctness requires a shared, atomic counter (Redis). The production details — layered limits, hot-key mitigation, and an explicit fail-open/closed policy — are what make it real.
Key takeaways
- Token bucket allows bursts up to capacity while bounding the average rate.
- Distributed limits need a shared atomic counter or they're really per-instance.
- Reject with 429 + Retry-After; layer edge (per-IP) and app (per-user) limits.
- Decide fail-open vs fail-closed for when the counter store is down.
Your notes
Saved to this device