Message Deduplication System Low-Level Design: Idempotency Keys, Exactly-Once Delivery, and Duplicate Detection

Message Deduplication System Low-Level Design

Distributed messaging systems face a fundamental challenge: networks are unreliable, and any message may be delivered more than once. Exactly-once semantics require a deduplication layer at the producer, broker, and consumer levels. This guide walks through the full low-level design of a message deduplication system, covering idempotency key generation, broker-side dedup via producer epochs, consumer-side Redis dedup, persistent PostgreSQL records for long-lived guarantees, and a cleanup job for expired records.

Producer-Side: Idempotency Key Generation

The producer generates a UUID idempotency key for each logical message before sending. This key travels as a message header and identifies the logical operation across all retries. If the producer crashes and retries, it must reuse the same idempotency key for the same logical event — typically by persisting the key alongside the outbox record before any send attempt.

Key design rules for idempotency keys:

Generate once per logical event, not per send attempt.
Store in the outbox table so retries reuse the same key.
Use UUID v4 for global uniqueness without coordination.
Include producer service name as a namespace prefix to avoid cross-service collisions.

Broker-Side Deduplication: Kafka Exactly-Once

Kafka's idempotent producer assigns each producer a producer epoch and maintains a per-partition sequence number. The broker rejects any message whose sequence number has already been committed for that producer epoch. This eliminates duplicates caused by producer retries within the in-flight window (default 5 sessions).

For full exactly-once across produce and consume, Kafka transactions wrap both the consume offset commit and the downstream produce in a single atomic transaction. This is broker-side dedup at the infrastructure level and requires no application code beyond enabling enable.idempotence=true and transactional.id.

Limitations: Kafka's broker-side dedup covers only the in-flight window. It does not protect against application-level redelivery after a consumer crash or against duplicate processing across consumer group rebalances beyond the transaction boundary.

Consumer-Side Deduplication with Redis

The most practical dedup layer for most systems is consumer-side: before processing a message, check a fast key-value store. Redis SET with TTL is the standard approach.

Algorithm:

On message receipt, extract message_id and consumer_group.
Attempt SET dedup:{consumer_group}:{message_id} 1 NX EX {ttl}.
If SET returns nil (key already exists), the message is a duplicate — skip and acknowledge.
If SET succeeds, process the message, then commit the result.

The TTL on the Redis key defines the dedup window. Messages redelivered within that window are caught. Messages redelivered after TTL expiry are processed again — which is acceptable if the window is sized correctly for the system's SLA.

Dedup Window Sizing

The dedup window TTL should be sized based on message type and redelivery risk:

Payment events: 24 hours — financial idempotency must survive overnight batch retries.
Order state transitions: 1–4 hours — covers typical retry windows.
Notification events: 5–15 minutes — duplicate notifications within seconds are the main risk.
Analytics events: often no dedup, or a short 5-minute window for near-real-time pipelines.

Persistent Dedup Store: PostgreSQL

Redis TTL-based dedup has a gap: if a message is redelivered after the TTL but before the downstream system has fully settled, a duplicate slips through. For high-value events (payments, inventory adjustments), a persistent dedup store in PostgreSQL provides a durable guarantee.

SQL Schema

CREATE TABLE deduplication_record (
    id              BIGSERIAL PRIMARY KEY,
    message_id      UUID        NOT NULL,
    consumer_group  VARCHAR(128) NOT NULL,
    processed_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at      TIMESTAMPTZ NOT NULL,
    CONSTRAINT uq_dedup UNIQUE (message_id, consumer_group)
);

CREATE INDEX idx_dedup_expires ON deduplication_record (expires_at);

CREATE TABLE deduplication_window (
    consumer_group  VARCHAR(128) PRIMARY KEY,
    ttl_seconds     INT         NOT NULL DEFAULT 3600,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The UNIQUE(message_id, consumer_group) constraint is the true dedup guarantee. Any concurrent insert for the same pair will fail with a unique violation, which the application catches and treats as a duplicate.

Python: Deduplication Functions

import uuid
import redis
import psycopg2
from datetime import datetime, timedelta, timezone

redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

def is_duplicate(message_id: str, consumer_group: str, ttl_seconds: int = 3600) -> bool:
    """Check Redis first (fast path), then fall back to Postgres for long-lived dedup."""
    redis_key = f"dedup:{consumer_group}:{message_id}"
    result = redis_client.set(redis_key, "1", nx=True, ex=ttl_seconds)
    if result is None:
        # Redis says duplicate — fast path
        return True
    # Redis says new — but also check Postgres for long-lived records
    with psycopg2.connect(dsn="postgresql://app:pass@db/appdb") as conn:
        with conn.cursor() as cur:
            cur.execute(
                """
                SELECT 1 FROM deduplication_record
                WHERE message_id = %s AND consumer_group = %s
                  AND expires_at > NOW()
                """,
                (message_id, consumer_group),
            )
            row = cur.fetchone()
    if row:
        return True
    return False


def mark_processed(message_id: str, consumer_group: str, ttl_seconds: int = 3600) -> bool:
    """
    Insert a deduplication record into Postgres.
    Returns True if this is the first time (not a duplicate).
    Returns False if a duplicate already exists (unique constraint violation).
    """
    expires_at = datetime.now(timezone.utc) + timedelta(seconds=ttl_seconds)
    try:
        with psycopg2.connect(dsn="postgresql://app:pass@db/appdb") as conn:
            with conn.cursor() as cur:
                cur.execute(
                    """
                    INSERT INTO deduplication_record (message_id, consumer_group, expires_at)
                    VALUES (%s, %s, %s)
                    """,
                    (message_id, consumer_group, expires_at),
                )
            conn.commit()
        return True
    except psycopg2.errors.UniqueViolation:
        return False


def cleanup_expired(batch_size: int = 10000) -> int:
    """
    Delete expired deduplication records in batches to avoid long table locks.
    Returns total rows deleted.
    """
    total_deleted = 0
    with psycopg2.connect(dsn="postgresql://app:pass@db/appdb") as conn:
        while True:
            with conn.cursor() as cur:
                cur.execute(
                    """
                    DELETE FROM deduplication_record
                    WHERE id IN (
                        SELECT id FROM deduplication_record
                        WHERE expires_at < NOW()
                        LIMIT %s
                    )
                    """,
                    (batch_size,),
                )
                deleted = cur.rowcount
            conn.commit()
            total_deleted += deleted
            if deleted < batch_size:
                break
    return total_deleted

Cleanup Job

Expired deduplication records must be purged periodically to prevent unbounded table growth. Run cleanup_expired() as a scheduled job (cron or Celery beat) every 15–60 minutes. Use batched deletes to avoid holding locks on the table during heavy write periods. Monitor the table size and the lag between expires_at and actual deletion.

End-to-End Flow

Producer generates message_id = uuid4(), writes to outbox with same ID.
Outbox relay sends message to Kafka with message_id in header and Kafka idempotent producer enabled.
Consumer reads message, calls is_duplicate(message_id, consumer_group).
If duplicate: ack and skip.
If not duplicate: call mark_processed() inside the same DB transaction as the business operation, then commit.
Cleanup job runs periodically to trim the dedup table.

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “Should I use Redis or a database for the deduplication store?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Use Redis for the fast path — it handles the vast majority of duplicates at sub-millisecond latency with TTL-based expiry. Use a database (PostgreSQL) for high-value events where you need a durable guarantee beyond the Redis TTL. The two-tier approach gives you speed for the common case and correctness for the edge case.”
}
},
{
“@type”: “Question”,
“name”: “How do I size the deduplication window TTL?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Size the TTL to cover the maximum plausible redelivery window for your message type. For payment events, use 24 hours. For order state transitions, 1–4 hours is usually sufficient. For notification events, 5–15 minutes covers retry bursts. Setting the TTL too short creates a gap where late redeliveries slip through; setting it too long blooms the dedup store size.”
}
},
{
“@type”: “Question”,
“name”: “Is producer-side or consumer-side deduplication more reliable?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Consumer-side deduplication is more reliable end-to-end. Producer-side idempotency keys prevent duplicate sends during retries, but cannot prevent duplicates caused by broker redelivery or consumer group rebalances. Consumer-side dedup with a persistent store is the last line of defense and catches duplicates regardless of their origin.”
}
},
{
“@type”: “Question”,
“name”: “How do I handle redelivery after a consumer crashes mid-processing?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Mark the message as processed (insert the dedup record) inside the same database transaction as the business operation. If the consumer crashes before commit, both the dedup record and the business operation are rolled back, so the redelivered message is processed correctly. If the consumer crashes after commit, the dedup record is durable and the redelivered message is correctly identified as a duplicate.”
}
}
]
}

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “Why use both Redis and PostgreSQL for deduplication?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Redis provides sub-millisecond dedup checks for the hot window (minutes to hours); PostgreSQL stores long-lived dedup records beyond Redis TTL for compliance or slow retry scenarios.”
}
},
{
“@type”: “Question”,
“name”: “How is the deduplication window sized?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Window TTL is set to the maximum expected message redelivery interval plus a safety margin; short-lived messages use 5-minute windows while payment events may use 24-hour windows.”
}
},
{
“@type”: “Question”,
“name”: “How does consumer-side dedup handle redelivery after a crash?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “On restart the consumer re-processes messages from its last committed offset; the Redis SET NX check returns false for already-processed messages, skipping them without side effects.”
}
},
{
“@type”: “Question”,
“name”: “How are expired deduplication records cleaned up without locking?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “A background job deletes rows WHERE expires_at < NOW() in small batches using LIMIT to avoid long-running locks on the deduplication_record table."
}
}
]
}