Low Level Design: Transactional Outbox Pattern

The transactional outbox pattern solves one of the most common reliability problems in microservices: how to write to a database and publish an event to a message broker atomically. Without this pattern, any crash between the two writes causes permanent inconsistency — data saved but event lost, or event published but data rolled back.

The Dual-Write Problem

Consider an order service. When a user places an order, the service must:

1. Insert a row into the orders table.
2. Publish an OrderPlaced event to Kafka or RabbitMQ.

These are two separate systems with no shared transaction manager. If the process crashes after step 1 but before step 2, the order exists but downstream consumers (inventory, shipping, payments) never learn about it. If you reverse the order, you can publish an event for an order that then fails to persist. There is no safe ordering of two independent writes.

The Outbox Solution

The outbox pattern eliminates the problem by making the event write part of the same database transaction as the business write. Instead of publishing to the broker directly, the service inserts an event record into an outbox table within the same transaction as the business entity update. A separate relay process reads the outbox and publishes to the broker.

Outbox table schema:

• id — UUID, primary key
• aggregate_type — entity type (Order, Payment, Shipment)
• aggregate_id — entity identifier
• event_type — event name (OrderPlaced, PaymentFailed)
• payload — serialized event body (JSON)
• created_at — insertion timestamp
• published_at — set by relay after successful broker publish; NULL until then
• status — ENUM('PENDING','PUBLISHED','FAILED')

The application code:

BEGIN TRANSACTION;
INSERT INTO orders (id, user_id, total, status) VALUES (...);
INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload) VALUES (Order, order_id, OrderPlaced, {...});
COMMIT;

Either both rows land, or neither does. The relay handles publishing asynchronously.

Polling-Based Relay

The simplest relay implementation: a background thread or service polls the outbox table every N seconds for rows with status = PENDING, publishes each to the broker, then updates status to PUBLISHED and sets published_at.

Polling interval is a latency vs load tradeoff. A 1-second poll adds at most 1 second of delivery latency but generates constant DB queries. For high-throughput systems, a short-circuit wake-up (notify after insert via LISTEN/NOTIFY in PostgreSQL) can reduce latency without burning query budget.

The relay must handle partial failures: if publishing succeeds but the status update fails, the row stays PENDING and gets republished on the next poll. This means the relay delivers at-least-once — consumers must handle duplicates.

CDC-Based Relay with Debezium

Change Data Capture (CDC) avoids polling entirely. Debezium connects to the database replication stream (PostgreSQL WAL, MySQL binlog, SQL Server CDC tables) and emits a change event for every INSERT or UPDATE. A Debezium connector configured to watch the outbox table publishes change events directly to Kafka topics without any application-level polling loop.

Benefits: near-zero latency, no additional DB load from polling queries, guaranteed ordering within a partition. The Debezium outbox event router SMT (Single Message Transform) can route each outbox row to a topic derived from aggregate_type, applying envelope transformation to produce clean domain events.

Tradeoff: Debezium adds operational complexity — it needs a Kafka Connect cluster, connector configuration management, and monitoring of connector lag and replication slot health.

At-Least-Once Delivery and Idempotency

Both the polling and CDC approaches guarantee at-least-once delivery: in failure scenarios, the same message may be published more than once. Consumers must be idempotent.

Consumer idempotency strategies:

• Natural idempotency — the operation is safe to repeat by design (e.g. setting a field to a specific value rather than incrementing).
• Deduplication table — consumer records processed event ids in a processed_events table and checks before applying. Insert and event processing wrapped in a transaction.
• Conditional update — consumer applies a WHERE clause requiring the current state to match the expected state before the event (optimistic concurrency).

Partition Ordering

For ordered delivery, events for the same aggregate must land in the same Kafka partition. Use aggregate_id as the partition key. This ensures that all events for a given order are consumed in the order they were produced, which matters for state machine transitions (OrderPlaced must arrive before OrderShipped).

Outbox Cleanup and Compaction

Published rows accumulate indefinitely unless pruned. A cleanup job deletes rows with status = PUBLISHED and published_at older than a retention window (e.g. 7 days). FAILED rows should be retained longer for investigation and may require a dead-letter process for manual or automated retry.

Partition the outbox table by created_at if volume is high — range partitions allow dropping old partitions instantly instead of running expensive DELETE batches.

Monitoring Relay Lag

Relay lag is the age of the oldest PENDING row. Alert if lag exceeds an SLA threshold (e.g. 30 seconds). Track: PENDING row count, oldest PENDING row age, publish rate (rows/sec), FAILED row count. For CDC, also monitor Debezium connector status and replication slot WAL size — an unhealthy connector stops consuming the slot, causing unbounded WAL growth that can fill the disk.

See also: Stripe Interview Guide 2026: Process, Bug Bash Round, and Payment Systems

See also: Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering

See also: Databricks Interview Guide 2026: Spark Internals, Delta Lake, and Lakehouse Architecture