Labs/Reliability/circuit-breaker

ReliabilityIntermediate

~25 min

Circuit Breaker

Prevent cascading failure by detecting downstream distress, fast-failing traffic, and orchestrating controlled probe recoveries.

#Cascading Failure#CLOSED / OPEN / HALF-OPEN#Failure Thresholds#Recovery Timeout#Fallback Strategies#Thread Pool Isolation

Overview

The Problem Statement

When a remote downstream service experiences degradation, high latency, or crashes, upstream callers wait on network sockets until connection or read timeouts occur. Threads block, thread pools exhaust, upstream queues backlog, and the entire system collapses in a domino cascade.

System Invariant:A failing dependency must never consume upstream caller resources; when a failure threshold is crossed, the circuit breaker opens immediately and fails fast without touching the network.

✓ When To Use

Any synchronous remote procedure call across network boundaries (HTTP, gRPC, DB drivers).
Calling third-party SaaS APIs with unpredictable latency profiles.
Protecting downstream dependencies already in distress from being crushed by repeated incoming calls.

✕ When NOT To Use

Local in-memory function invocations where failures do not consume socket resources.
Asynchronous message consumption where message lag can naturally buffer in a durable queue.
Idempotent read-after-write operations within the same transactional boundary.

Why It Exists

Catastrophic Outage Scenario

An e-commerce order service depends on an external fraud-scoring API. The fraud vendor suffers a DDoS attack and response time increases from 50ms to 30,000ms (the HTTP socket timeout). Every checkout thread in the order service blocks. Within 12 seconds, all 1,000 server worker threads are stuck in socketRead0. The health-check endpoint stalls, Kubernetes kills the pods, restarts them, and the new pods instantly saturate their thread pools upon boot, causing permanent cluster downtime.

Downstream System Degradation:

Total upstream thread and file-descriptor exhaustion.
Unbounded latency propagation across the call graph.
Denial of service for unrelated healthy routes sharing the same runtime pool.
Inability of degraded downstream services to recover due to unremitting request pressure.

How It Works

The circuit breaker pattern wraps dangerous network operations in a stateful finite state machine with three primary states: CLOSED, OPEN, and HALF-OPEN.

In the CLOSED state, requests flow normally through to the remote service. The breaker monitors execution outcomes over a sliding time or count window. If consecutive failures or error rate exceeds the configured failure threshold, the breaker trips to OPEN.

In the OPEN state, the breaker immediately returns an error or fallback response without placing a network call. This gives the degraded downstream service breathing room to recover.

After a configured sleep window (reset timeout), the breaker transitions to HALF-OPEN. It permits a limited canary quota of trial requests through. If the trial requests succeed, the breaker resets to CLOSED. If any trial fails, it immediately re-opens for another timeout period.

Single-Node vs Distributed Reality:

Single-node circuit breakers keep failure counts in local process memory (e.g., using atomic counters). Distributed circuit breakers aggregate failure metrics across an entire service fleet (e.g., using Redis or Envoy mesh stats) to trip globally when a shared database or third-party API is down.

Count-based Sliding Window

Evaluates the outcome of the last N calls (e.g., last 100 requests). Trips if failure rate exceeds X%.

+ Smooth, statistically robust; immune to low-volume anomalies.

- Slower to trip during sudden catastrophic failure if window size N is large.

Time-based Sliding Window

Maintains a rolling ring buffer of 1-second buckets over the last T seconds (e.g., last 10 seconds). Trips if failure rate in the window exceeds threshold.

+ Adapts to changing traffic volume; reflects current system health accurately.

- Requires memory for ring buffer buckets; slightly higher synchronization overhead.

Consecutive Failure Counter

Trips immediately after K consecutive errors without statistical averaging.

+ Extremely simple; rapid reaction to sudden total blackouts.

- Brittle under momentary network blips; can trip prematurely under high volume.

Engineering Prediction Challenge

Prove It: Predict the System Behavior

Reason before simulating

System State Given

Failure threshold = 5 consecutive 500 errors. Sleep window = 10 seconds. State = CLOSED.

Prediction Question:

The downstream payment provider has a network outage and fails 5 requests in a row. What happens to the 6th incoming request?

The 6th request attempts downstream network I/O and times out after 30 seconds.

The breaker transitions to OPEN, and the 6th request fails fast locally in ~0.1ms with an ErrCircuitOpen error without touching the network.

The 6th request is buffered in an in-memory queue until the provider recovers.

Interactive Visualizer

Trip consecutive failures to watch the circuit state machine transition from CLOSED to OPEN, start the recovery sleep timer, and send single canary probes in HALF-OPEN mode.

Circuit Breaker Finite State Machine

Failure Threshold: 3 consecutive errors | Recovery Timeout: 5s

STATE 01ACTIVE

CLOSED

Normal operation. Requests pass to downstream service.

Failure Counter:0 / 3

STATE 02

OPEN

Fast-fails incoming calls immediately without network call.

Sleep Timer:0s remaining

STATE 03

HALF-OPEN

Allows trial canary requests. Success resets to CLOSED; failure re-opens.

Canary Status:Awaiting trial call

LAST EVENT:System initialized in CLOSED state.

Build It Step-by-Step

The engineering blueprint. Expand each sequential milestone to examine requirements, architectural impacts, common pitfalls, and blueprint pseudocode.

Define the State Machine & Invariants

Model the three canonical states. In CLOSED, errors are tracked. In OPEN, network calls are short-circuited. In HALF-OPEN, a restricted number of canary probes are allowed.

enum State { CLOSED, OPEN, HALF_OPEN }
struct CircuitBreaker {
  state: State
  failureThreshold: int      // e.g. 5 failures
  recoveryTimeout: Duration  // e.g. 10 seconds
  consecutiveFailures: int
  lastStateChange: Timestamp
}

Define Failure Criteria

A 404 Not Found or 400 Bad Request indicates client error, not downstream infrastructure failure. Only 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout, and network connection drops should increment the breaker's failure counter.

Implement Fast-Fail Logic in OPEN State

Before invoking the target function, inspect the current state. If OPEN, check if recoveryTimeout has elapsed. If not elapsed, return ErrCircuitOpen immediately.

function Execute(fn):
  state = getState()
  if state == OPEN:
    if now() - lastStateChange > recoveryTimeout:
      transitionTo(HALF_OPEN)
    else:
      return ErrCircuitOpen
      
  // Proceed with execution...

Execute Wrapped Operation with Error Trapping

Invoke the user-provided closure. Record whether the execution succeeded or failed. In Go, trap panics with recover(). In Java/TypeScript, wrap in try-catch blocks.

Orchestrate HALF-OPEN Canary Probing

When entering HALF-OPEN, only allow 1 (or a small configured limit) request to test the downstream dependency. Other requests arriving during this window should either fail-fast or execute the fallback.

Provide Fallback Hooks

Allow developers to supply an optional fallback function. For example, if the recommendation service breaker is OPEN, return the top 10 globally trending items from a static cache.

Add Observability & State Change Alerts

Every transition to OPEN is a critical system signal. Emit a metric (circuit_breaker_state{name} = 1) to alert site reliability engineers.

Code Implementation

Production-ready, idiomatic reference implementations across Go, TypeScript, Python, and Java with sticky language switching, copy-to-clipboard, and source download.

Core Architectural Design Decisions

Invariants & Mechanics

Decision 01

Protects internal state transitions with threading.Lock.

Decision 02

External function called outside lock to prevent blocking parallel threads.

Algorithmic Complexity:Thread-safe with sub-microsecond lock contention.

circuit_breaker.py•60 LOC

import time
import threading
from enum import Enum
from typing import Callable, Any, Optional

class CircuitState(Enum):
    CLOSED = "CLOSED"
    OPEN = "OPEN"
    HALF_OPEN = "HALF_OPEN"

class CircuitBreakerOpenException(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, recovery_timeout_sec: float = 10.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout_sec
        self.state = CircuitState.CLOSED
        self.consecutive_failures = 0
        self.last_state_change = time.monotonic()
        self._lock = threading.Lock()

    def call(self, func: Callable[..., Any], *args, fallback: Optional[Callable] = None, **kwargs) -> Any:
        with self._lock:
            now = time.monotonic()
            if self.state == CircuitState.OPEN:
                if now - self.last_state_change > self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                    self.last_state_change = now
                else:
                    if fallback:
                        return fallback()
                    raise CircuitBreakerOpenException("Circuit is OPEN: fast-failing")

        # Execute remote call outside the lock
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            if fallback:
                return fallback()
            raise e

    def _on_success(self):
        with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
                self.consecutive_failures = 0
                self.last_state_change = time.monotonic()
            else:
                self.consecutive_failures = 0

    def _on_failure(self):
        with self._lock:
            self.consecutive_failures += 1
            if self.state == CircuitState.HALF_OPEN or self.consecutive_failures >= self.failure_threshold:
                self.state = CircuitState.OPEN
                self.last_state_change = time.monotonic()

Implementation Strategy: Thread-safe Python implementation using threading.Lock and monotonic clock. Ensures lock is only held during fast state reads and counter mutations, never during network execution.

Thread-safe with sub-microsecond lock contention.

Edge Cases & Failure Modes

Investigate real-world production incidents. Expand each failure mode to analyze symptoms, root causes, and defensive engineering mitigations.

Scenario: Canary Probe Flood in HALF-OPEN

Consequence: If hundreds of threads check the breaker at the exact millisecond it enters HALF-OPEN, all of them send requests simultaneously, crushing the recovering downstream service before it stabilizes.

Engineering Solution: Use an atomic reservation counter (e.g. CAS token) allowing strictly 1 (or N) concurrent trial probe while forcing other threads to continue fast-failing until the probe resolves.

Scenario: Intermittent Spurious Network Glitches

Consequence: Consecutive failure counter trips on 3 isolated TCP resets across 1,000,000 successful requests, causing unnecessary 10-second downtime.

Engineering Solution: Transition from simple consecutive counters to rolling percentage failure rate windows (e.g., trip only if failure rate > 50% across at least 20 samples).

Scenario: Hung Connections Bypassing Breaker

Consequence: The circuit breaker wraps the function call, but the HTTP client inside has an infinite or 60-second read timeout. Worker threads remain blocked for a minute before reporting failure.

Engineering Solution: A circuit breaker cannot fix missing socket timeouts. Always enforce strict connection and socket timeouts (e.g., 500ms - 2s) at the HTTP client driver level.

Production Considerations

Production checklist covering observability metrics, horizontal sharding, reliability fail-open patterns, and client identity security.

Scaling to 10x / 100x Traffic:

Combine local in-process circuit breakers with Envoy/service mesh sidecar egress routing.
Share aggregated breaker states via lightweight gossip or Redis pub/sub if a global fleet-wide outage is detected.
Employ adaptive concurrency limits (TCP Vegas / Little's Law) alongside circuit breakers.

Telemetry & Observability:

PROMETHEUS METRICS:

➔ circuit_breaker_state{name} (0=Closed, 1=HalfOpen, 2=Open)

➔ circuit_breaker_failures_total{name}

➔ circuit_breaker_fast_failures_total{name}

DISTRIBUTED TRACES & LOGS:

➔ Span tag 'circuit_breaker.state' added to distributed traces to visualize fast-fails.

Trade-offs

Select an algorithm below to dynamically compare memory footprints, burst tolerance, boundary precision, and distributed suitability.

Approach	Advantages	Disadvantages	Use When
Consecutive Count Breaker	Lowest CPU/memory overhead; instant reaction to complete outages.	Cannot handle fluctuating error rates; prone to false trips on isolated blips.	Resource-constrained environments or services with strict zero-tolerance thresholds.
Rolling Window Percentage Breaker	Statistically sound; handles high traffic gracefully without false trips.	Consumes more memory for ring buffers; requires minimum request volume to evaluate.	High-throughput production microservices with hundreds of requests/sec.
Mesh-Level Breaker (Envoy / Istio)	Language-agnostic; centralized policy enforcement without code changes.	Adds sidecar proxy latency; cannot run custom in-process fallbacks easily.	Kubernetes microservice clusters standardizing across multiple languages.