Labs/Reliability/idempotency

ReliabilityIntermediate

~25 min

Idempotency

Guarantee that repeating an identical mutating operation produces the exact same system state and response without duplicate side effects.

#Idempotency Keys#At-Least-Once Delivery#Network Partitions#Concurrent Duplicate Requests#Database Unique Constraints#Payload Hash Verification

Overview

The Problem Statement

In distributed computing, networks are unreliable (the Two Generals' Problem). When a client sends a mutating request (like charging a credit card or transferring funds) and the connection drops before receiving the response, the client cannot know whether the server executed the mutation or failed before execution. Re-issuing the request risks catastrophic double-mutations.

System Invariant:f(f(x)) = f(x): Executing an operation multiple times with the same idempotency key must yield identical database state and identical client response as executing it once.

✓ When To Use

Financial transactions, payment gateways, and balance deductions.
Order creation, inventory reservation, and reservation dispatch.
Webhook delivery receivers and message queue consumers subject to at-least-once delivery.
Any REST POST / PUT mutating endpoint exposed over public or unreliable mobile networks.

✕ When NOT To Use

Naturally idempotent operations like GET, PUT with complete resource replacement, or DELETE by primary key.
Append-only telemetry and log ingestion pipelines where high throughput outweighs rare duplicates.
Read-only queries without state mutations.

Why It Exists

Catastrophic Outage Scenario

During an automated subscription renewal run, a worker issues 100,000 billing calls. An upstream gateway times out on 20,000 requests. The worker's retry policy fires, rebilling the 20,000 customers. Because the billing endpoint was not idempotent, 20,000 bank accounts suffer double debits, triggering thousands of overdraft fees, bank dispute chargebacks, and regulatory compliance fines.

Downstream System Degradation:

Double financial debits and erroneous ledger imbalances.
Duplicate stock allocations causing phantom negative inventory.
Loss of consumer trust and severe compliance penalties.
Inability to deploy aggressive automatic retries safely.

How It Works

To achieve idempotency on non-idempotent operations (like HTTP POST), the caller generates a cryptographically random unique identifier—an Idempotency Key (typically a UUID v4) sent in the request header.

When the server receives the request, it checks whether this key already exists in a durable transaction log or key-value store within an atomic transaction.

If the key is found and its status is 'COMPLETED', the server bypasses business logic completely and returns the cached HTTP status code and response body from the original execution.

If the key exists with status 'IN_PROGRESS', a concurrent duplicate is currently executing. The server either waits or rejects with HTTP 409 Conflict.

If the key does not exist, the server inserts a lock record ('IN_PROGRESS'), executes the business logic inside a database transaction, stores the serialized response body, marks the record 'COMPLETED', and returns.

Single-Node vs Distributed Reality:

Single-node systems can use in-memory locks or a single database table. Distributed systems must leverage atomic distributed locks or database ACID transactions with unique constraints on (user_id, idempotency_key) to prevent race conditions across parallel worker instances.

RDBMS Unique Constraint Log

Insert key into an 'idempotency_keys' table with a UNIQUE constraint within the business database transaction.

+ Provides 100% ACID consistency; response cache and business mutation commit or rollback atomically.

- Adds write load to primary database; requires schema migrations.

Redis SETNX / Distributed Mutex

Acquire an atomic lease in Redis via SETNX key status='IN_PROGRESS' with an expiration TTL, then execute backend logic.

+ Very fast; relieves write pressure from relational database.

- Requires two-phase coordination if Redis and database become desynchronized during crashes.

Payload Fingerprinting (SHA-256)

Compute SHA-256 hash of (request method + path + body). Store alongside idempotency key and reject if key matches but hash differs.

+ Detects corrupted retries and programmer errors immediately.

- Requires deterministic JSON serialization before hashing.

Engineering Prediction Challenge

Prove It: Predict the System Behavior

Reason before simulating

System State Given

Client sends POST /v1/charges with Idempotency-Key: 'pay_9988' and payload amount: $50.

Prediction Question:

Due to a network drop, the client receives a socket timeout, but the server successfully debited the card. The client retries the exact same request 2 seconds later. What does the server do?

Charges the customer a second time for $50 because it is a new HTTP request.

Detects 'pay_9988' in the idempotency ledger, skips business execution, and returns the cached HTTP 200 response with zero duplicate charges.

Returns HTTP 400 Bad Request because the key has already been consumed.

Interactive Visualizer

Simulate payment mutations using an Idempotency-Key. Test duplicate replays, cached 200 responses, and payload conflict mismatches (HTTP 422).

Idempotent Gateway & Deduplication Simulator

Guarantees f(f(x)) = f(x) over flaky networks.

Outbound Client Request

HTTP Header: Idempotency-Key

Charge Amount ($)

GATEWAY RESPONSE FEEDBACK:

Ready to test idempotency pipeline.

Database Ledger Records (0)UNIQUE(key)

No database records created yet.

Build It Step-by-Step

The engineering blueprint. Expand each sequential milestone to examine requirements, architectural impacts, common pitfalls, and blueprint pseudocode.

Require Client-Generated Idempotency Key

Clients create a UUID v4 per logical intent. If a user clicks 'Submit' twice on the UI, the client sends the identical key. If the user initiates a separate transaction, a new key is minted.

key = request.headers.get("Idempotency-Key")
if not key:
  return BadRequest("Missing Idempotency-Key header")

Compute Request Payload Hash

Compute SHA-256(method + path + body). If a subsequent request reuses an existing key with different parameters (e.g., trying to charge $1000 instead of $100), reject with HTTP 422 Unprocessable Entity.

Atomic Lock Reservation (IN_PROGRESS)

Attempt to insert into idempotency table with status 'STARTED' or acquire a distributed lock. If duplicate key error occurs, query existing record.

INSERT INTO idempotency_records (key, client_id, payload_hash, status, created_at)
VALUES (?, ?, ?, 'IN_PROGRESS', NOW())
ON CONFLICT (client_id, key) DO NOTHING;

Handle In-Flight Concurrent Duplicates

If the key is found and status is 'IN_PROGRESS', a parallel thread is actively processing the first request. Return HTTP 409 Conflict with 'Concurrent request in flight' or wait on an advisory lock.

Execute Business Transaction

Execute charges, inventory deductions, or account transfers within the core database transaction.

Persist Serialized Response & Mark COMPLETED

Once business logic succeeds, update the record: status = 'COMPLETED', response_code = 200, response_body = JSON. Commit atomically.

Replay Cached Response on Retry

When a retry arrives with a COMPLETED key, fetch cached record and write exact headers, status code, and body back to client.

Code Implementation

Production-ready, idiomatic reference implementations across Go, TypeScript, Python, and Java with sticky language switching, copy-to-clipboard, and source download.

Core Architectural Design Decisions

Invariants & Mechanics

Decision 01

sort_keys=True ensures key-order independence during payload serialization.

Decision 02

Thread-safe lock acquisition during record checks and state transitions.

Algorithmic Complexity:O(1) dictionary key lookup.

idempotency_manager.py•61 LOC

import hashlib
import json
import threading
from typing import Callable, Tuple, Any, Dict, Optional

class IdempotencyConflict(Exception): pass
class PayloadMismatch(Exception): pass

class IdempotencyManager:
    def __init__(self):
        self._lock = threading.Lock()
        self._records: Dict[str, Dict[str, Any]] = {}

    def _hash(self, payload: Any) -> str:
        serialized = json.dumps(payload, sort_keys=True)
        return hashlib.sha256(serialized.encode("utf-8")).hexdigest()

    def execute(
        self,
        key: str,
        payload: Any,
        handler: Callable[[], Tuple[int, Any]]
    ) -> Tuple[int, Any, bool]:
        """
        Executes handler with idempotency semantics.
        Returns: (status_code, response_body, is_replayed)
        """
        payload_hash = self._hash(payload)

        with self._lock:
            rec = self._records.get(key)
            if rec:
                if rec["payload_hash"] != payload_hash:
                    raise PayloadMismatch("Idempotency key reused with mismatched payload")
                if rec["status"] == "IN_PROGRESS":
                    raise IdempotencyConflict("Duplicate request already in progress")
                if rec["status"] == "COMPLETED":
                    return rec["status_code"], rec["body"], True

            # Reserve key
            self._records[key] = {
                "payload_hash": payload_hash,
                "status": "IN_PROGRESS"
            }

        try:
            status_code, body = handler()
        except Exception as e:
            with self._lock:
                self._records.pop(key, None)
            raise e

        with self._lock:
            self._records[key] = {
                "payload_hash": payload_hash,
                "status": "COMPLETED",
                "status_code": status_code,
                "body": body
            }

        return status_code, body, False

Implementation Strategy: Thread-safe Python idempotency manager sorting JSON dictionary keys before SHA-256 calculation for deterministic hashing across differing client JSON serializers.

O(1) dictionary key lookup.

Edge Cases & Failure Modes

Investigate real-world production incidents. Expand each failure mode to analyze symptoms, root causes, and defensive engineering mitigations.

Scenario: Worker Crash during IN_PROGRESS

Consequence: The server crashes while processing. The record remains stuck in 'IN_PROGRESS' indefinitely, blocking all future retries from the customer.

Engineering Solution: Attach an expiration lease (e.g., 2 minutes) to the IN_PROGRESS state. If a retry arrives and now() > created_at + lease_time, assume prior worker died and allow re-execution.

Scenario: Client Re-using Key with Different Currency

Consequence: A buggy client reuses key 'tx_123' for a $1,000 transaction after using it for a $10 transaction.

Engineering Solution: Always calculate and verify the SHA-256 fingerprint of the request payload. Return HTTP 422 immediately if the key matches but payload differs.

Scenario: Primary Database Failover during Write

Consequence: Idempotency record is written to Redis, but primary DB fails to commit. Client retries, sees Redis says 'IN_PROGRESS', but DB has no record.

Engineering Solution: Store idempotency records in the same relational database and transaction as the business mutation whenever strict ACID guarantees are required.

Production Considerations

Production checklist covering observability metrics, horizontal sharding, reliability fail-open patterns, and client identity security.

Scaling to 10x / 100x Traffic:

Offload completed idempotency response storage from PostgreSQL to Redis or DynamoDB with native TTL auto-deletion.
Employ lightweight SHA-256 Bloom filters to quickly determine if an idempotency key is definitely new before querying persistent storage.

Telemetry & Observability:

PROMETHEUS METRICS:

➔ idempotency_requests_total{status='fresh|replayed|conflict|mismatch'}

➔ idempotency_cache_hit_ratio

DISTRIBUTED TRACES & LOGS:

➔ Tag span with 'idempotency.replayed = true' when short-circuiting handler logic.

Trade-offs

Select an algorithm below to dynamically compare memory footprints, burst tolerance, boundary precision, and distributed suitability.

Approach	Advantages	Disadvantages	Use When
RDBMS Same-Transaction Table	Guarantees 100% ACID consistency; zero desynchronization between business state and idempotency state.	Increases primary database write IOPS; requires relational storage schema.	Financial ledgers, payment transactions, and mission-critical state mutations.
Redis Distributed Key Store	Sub-millisecond latency; offloads write traffic from database; built-in TTL eviction.	Non-transactional coupling: Redis write and DB write can desynchronize on crash.	High-volume notifications, webhooks, or secondary service integrations.