Low Level Design: Idempotency in Distributed Systems

By /

Idempotency means that performing an operation multiple times produces the same result as performing it once. In distributed systems, network failures require retries, and retries without idempotency cause duplicate operations — double charges, duplicate orders, duplicate emails. Building idempotent APIs and services is a fundamental requirement for any system that needs to be reliable under failure. Stripe, Airbnb, and virtually every payment system has built idempotency into their core architecture.

Idempotency Keys

The standard pattern: clients generate a unique idempotency key (UUID v4) per logical operation and include it in every request (HTTP header or request body field). The server stores the result of the first successful execution keyed by the idempotency key. On subsequent requests with the same key, the server returns the cached result without re-executing the operation. The key is client-generated so the client can retry immediately after a network failure without needing to contact the server first to get a key.

// Server-side idempotency key handling
func CreatePayment(ctx context.Context, req *PaymentRequest) (*Payment, error) {
    key := req.IdempotencyKey
    if key == "" { return nil, ErrMissingIdempotencyKey }

    // Check for existing result
    cached, err := store.Get(ctx, "idem:"+key)
    if err == nil {
        return cached.(*Payment), nil  // return cached result
    }

    // Execute the operation
    payment, err := processPayment(ctx, req)

    // Store result (with TTL, e.g., 24 hours)
    store.Set(ctx, "idem:"+key, payment, 24*time.Hour)

    return payment, err
}

// Client retries safely
key := uuid.New().String()
for attempt := 0; attempt < maxRetries; attempt++ {
    result, err := client.CreatePayment(&PaymentRequest{
        Amount:           100,
        IdempotencyKey:   key,  // same key on every retry
    })
    if err == nil { break }
}

The Exactly-Once Illusion: At-Least-Once + Idempotency

Exactly-once execution requires distributed transactions, which are expensive and often impractical. The pragmatic alternative: at-least-once delivery (retry on failure) plus idempotent handlers (safe to execute multiple times). The combination produces the observable behavior of exactly-once without the coordination overhead. The key requirement: the idempotent check and the operation must be atomic — use a database transaction or a Redis SETNX (set-if-not-exists) to prevent a race condition where two concurrent retries both pass the idempotency check before either stores the result.

Database-Level Idempotency

Use unique constraints to enforce idempotency at the database level — the most reliable approach. For order creation: add a unique constraint on (user_id, order_reference). Duplicate inserts get a constraint violation, which the application handles by returning the existing record. For ledger entries: add a unique constraint on (transaction_id, ledger_type). This eliminates the need for an explicit idempotency key store — the database itself prevents duplicates. Works for insert operations; for updates use conditional updates: UPDATE payments SET status = captured WHERE id = ? AND status = authorized.

Idempotent Message Consumers

Kafka and SQS deliver messages at-least-once. Consumer handlers must be idempotent. Pattern: include a deterministic event ID in each message (e.g., hash of source_id + event_type + timestamp). Before processing, check if this event_id was already processed (Redis SET NX or database unique constraint on processed_events). If already processed, skip and acknowledge. If not, process and record. Use the same database transaction for the business logic and the processed_events insert — so they commit atomically and a crash mid-processing does not cause silent data loss.

Key Interview Discussion Points

  • HTTP method idempotency: GET, PUT, DELETE are idempotent by spec; POST is not — use idempotency keys for POST payment and order endpoints
  • Key expiry: idempotency keys should expire after 24 hours (Stripe model) — clients retrying a week later get a new operation, not the cached old result
  • Partial failure: the operation succeeded but storing the result failed — on retry, the operation runs again. Design operations to be naturally idempotent (charge_if_not_already_charged) rather than relying solely on the key store
  • Sagas and compensation: for multi-step workflows, each step must be idempotent; compensating transactions (for rollback) must also be idempotent
  • Deduplication window: SQS standard queues guarantee at-least-once delivery; SQS FIFO queues have a 5-minute deduplication window using message deduplication IDs
Scroll to Top