Labs/Distributed Systems/rate-limiting
Distributed SystemsIntermediate
~25 min

Rate Limiting

Control ingress throughput and defend upstream services against resource exhaustion, DDoS, and cascading failure.

#Token Bucket#Leaky Bucket#Sliding Window Counter#Distributed Redis Lua#Race Conditions#HTTP 429 Headers
01

Overview

The Problem
“Traffic is bursty. Servers are finite.”
Failure Cascade Progression

Normal Operating Window

1. Inbound Requests
450 req/s
Within limits
2. Worker Queue
4 queued
Processing buffer
3. p99 Tail Latency
12 ms
Optimal (SLO < 50ms)
4. Service State
Healthy
Thread pool intact
Incoming traffic matches provisioned worker pool capacity. Zero queue buildup.

Unregulated ingress forces requests into unconstrained memory queues. When queue depth saturates thread pools, latency breaches timeouts and client retries trigger cascading outages.

Core Invariant:For any client C and sliding time window W, the number of admitted requests must never exceed threshold N, while rejected requests terminate at the perimeter with minimal compute cost.
02

Why It Exists

Simulated Production Incident

Black Friday Flash Sale: 50,000 RPS vs 500 Max Database Connections

50,000 req/svs500 DB Conns
+0 ms
Flash Sale Begins
DB Pool: 45 / 500 active connections

Traffic spikes instantaneously from 800 req/s to 50,000 req/s.

+180 ms
Queue Saturation
DB Pool: 500 / 500 (100% saturated)

Application threads lock waiting for Postgres connections. TCP accept backlog fills.

+350 ms
Upstream Timeouts
Database is now executing queries for sockets that are already abandoned.

Clients hit their 300ms HTTP read timeout. Gateway drops waiting sockets.

+600 ms
The Retry Storm
CPU spikes to 100%. Thread context-switching consumes all CPU cycles.

50,000 failed clients simultaneously retry requests, doubling the load to 100k RPS.

+900 ms
Total Cascading Outage
Global outage. Complete service restart required.

PostgreSQL crash under connection exhaustion. Entire cluster health checks fail.

03

How It Works

Token Bucket

O(1) time

Tokens refill into a bucket of capacity B at rate r. Each request consumes 1 token. When empty, requests are rejected. Allows short bursts up to capacity B.

Strengths
  • Naturally absorbs sudden traffic bursts up to capacity
  • Extremely light memory footprint (2 numbers per key)
  • Industry standard: used by AWS, Stripe, Cloudflare
Trade-offs
  • Can allow momentary spikes into downstream databases
  • Requires careful capacity sizing to avoid downstream overload
Algorithm Specification
Burst Tolerance:YES (Up to Capacity)
Memory Per Key:~16 bytes (tokens count + last timestamp)
Atomic Primitive:Lua / Mutex / INCR
Engineering Prediction Challenge

Prove It: Predict the System Behavior

Reason before simulating
System State Given

Bucket Capacity = 100 tokens, Refill Rate = 20 tokens/sec. The bucket is currently full (100 tokens).

Prediction Question:

An instantaneous flash burst of 250 requests arrives in the exact same millisecond. What happens to the traffic?

A

All 250 requests are accepted because the system autoscales token generation.

B

Exactly 100 requests are admitted (HTTP 200) and 150 requests are immediately rejected (HTTP 429).

C

The first 100 requests pass, and the remaining 150 wait in an internal thread queue for 7.5 seconds.

D

All 250 requests are rejected because the burst size exceeded the bucket capacity.

04

Interactive Visualizer

Adjust capacity, refill rates, and packet arrival speeds. Trigger instant bursts or continuous streams to observe token consumption, bucket depletion, and HTTP 429 rejection branches.

Interactive Token Bucket Laboratory

Real-time packet arrival, leak/refill concurrency, and 429 rejection dynamics.

Bucket Fill: 10 / 10 TokensRefill: +3 tokens/sec
LIVE INBOUND PACKET STREAMGREEN: 200 OK | RED: 429 REJECT
Send requests or stream traffic...
p99 Latency Telemetry7.0 ms
Allowed (200)0
Rejected (429)0
Current RPS0 req/s
429 Rejection Rate0.0%
Capacity (B):10
Refill Rate (r):3/s
Incoming RPS:4/s
Burst Size:8 req
Speed:1x
05

Build It Step-by-Step

The engineering blueprint. Expand each sequential milestone to examine requirements, architectural impacts, common pitfalls, and blueprint pseudocode.

Why This Matters

Identify tenant identification key (API key vs IP) and bucket parameters (capacity B, refill rate r).

Architecture Impact

Determines state cardinality and whether storage fits entirely in L1 memory cache.

Blueprint Pseudocode:
type Bucket struct {
  capacity  int64     // maximum token capacity
  tokens    int64     // current available tokens
  refillRate int64    // tokens generated per unit time
  lastRefill time.Time // monotonic timestamp of previous evaluation
  mu        sync.Mutex
}
Common Implementation Pitfalls:
  • Using client IP without proxy awareness (breaks for NAT/universities)
  • Floating point token counts that cause precision drift
Implementation Hint: Always store integer counts of tokens (or scaled nanos) rather than floats.
06

Code Implementation

Production-ready, idiomatic reference implementations across Go, TypeScript, Python, and Java with sticky language switching, copy-to-clipboard, and source download.

Core Architectural Design Decisions

Invariants & Mechanics
Decision 01

Used time.monotonic() instead of time.time() to protect against clock drift and leap seconds.

Decision 02

Used context manager with self._lock to ensure locks always release even on unexpected exceptions.

Algorithmic Complexity:O(1) execution time. Negligible lock acquisition overhead for low-to-medium contention.
import time
import threading
from typing import Tuple

class TokenBucket:
    """Thread-safe Token Bucket Rate Limiter with lazy refill."""

    def __init__(self, capacity: float, refill_rate_per_sec: float):
        self.capacity = float(capacity)
        self.refill_rate = float(refill_rate_per_sec)
        self.tokens = float(capacity)
        self.last_refill = time.monotonic()
        self._lock = threading.Lock()

    def allow(self, cost: float = 1.0) -> Tuple[bool, float, float]:
        """
        Attempts to consume tokens.
        Returns: (is_allowed, remaining_tokens, retry_after_seconds)
        """
        with self._lock:
            now = time.monotonic()
            elapsed = now - self.last_refill
            self.last_refill = now

            # Replenish tokens
            self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)

            if self.tokens >= cost:
                self.tokens -= cost
                return True, self.tokens, 0.0

            deficit = cost - self.tokens
            retry_after = deficit / self.refill_rate
            return False, self.tokens, retry_after
Implementation Strategy: Python implementation using time.monotonic() to be immune to system clock adjustments (NTP sync shifts). Protected with threading.Lock to ensure thread-safe operations in multithreaded WSGI/ASGI servers.
O(1) execution time. Negligible lock acquisition overhead for low-to-medium contention.
07

Edge Cases & Failure Modes

Investigate real-world production incidents. Expand each failure mode to analyze symptoms, root causes, and defensive engineering mitigations.

08

Production Considerations

Production checklist covering observability metrics, horizontal sharding, reliability fail-open patterns, and client identity security.

rate_limiter_requests_total{status='allowed|rejected'}

Why it matters: Tracks total accepted vs throttled volume to calculate error budget burn rate.

Example: Prometheus Counter: alert when rejected / (allowed + rejected) > 0.05 for 5m.
rate_limiter_tokens_available{tier='standard|premium'}

Why it matters: Exposes real-time reservoir depth to detect systemic under-provisioning.

Example: Prometheus Gauge: visualize reservoir percentiles in Grafana dashboard.
rate_limiter_eval_latency_seconds

Why it matters: Ensures the limiter check itself does not introduce tail latency overhead.

Example: Histogram p99 latency SLA: must execute within < 1.2ms via Redis pipeline.
09

Trade-offs

Select an algorithm below to dynamically compare memory footprints, burst tolerance, boundary precision, and distributed suitability.

PrimitiveMemory FootprintBurst ToleranceDistributed Readiness
Token Bucket~16High Excellent
Leaky Bucket~64Zero Moderate
Fixed Window~8Flawed High
Sliding Window~32Moderate High (weighted) / Low (sorted set ZADD overhead).
Token BucketDefault choice for 90% of microservice rate limiting applications.
Best Real-World Use Case

General-purpose REST/GraphQL public APIs and user operations (AWS, Stripe).

Precision & Boundaries

High — Continuous monotonic time resolution without window quantization.

Burst Handling

High — Gracefully accommodates bursts up to bucket capacity.

Distributed Coordination

Excellent — Trivial to implement in atomic Redis Lua script.

10

Further Reading

Authoritative industry references, IETF RFC standards, and systems engineering papers.

Engineering BlogStripe Engineering

Scaling your API with rate limiters

How Stripe protects its payment API using token buckets and tiered load shedders.

Read Original Source
RFCIETF Standards

RFC 6585: Additional HTTP Status Codes (HTTP 429)

The canonical standard defining 429 Too Many Requests and Retry-After response semantics.

Read Original Source
Academic PaperACM & Network Working Group

Token Bucket Algorithm and Traffic Policing

Original mathematical formalization of leaky and token bucket traffic shaping mechanisms.

System ArchitectureCloudflare Systems Engineering

Rate Limiting Architecture at Cloudflare

Techniques for globally distributed rate limiting across hundreds of edge data centers without central sync.

Read Original Source