Event Streaming Platform Low-Level Design: Log Compaction, Schema Registry, and Backpressure

Problem Statement and Requirements

An event streaming platform extends a basic message broker with features required for long-lived, schema-evolving data pipelines: log compaction to retain the latest state per key indefinitely, a schema registry to enforce and evolve message contracts, and backpressure mechanisms to protect consumers from being overwhelmed. These features make the platform suitable for event sourcing, change data capture, and stream processing.

Functional Requirements

Log compaction: for keyed topics, retain only the most recent message per key; delete markers (tombstones) remove keys entirely
Schema registry: producers register Avro/Protobuf/JSON Schema; consumers fetch schemas by ID embedded in message headers
Schema evolution with compatibility enforcement: backward, forward, or full compatibility modes configurable per subject
Consumer backpressure: consumers signal fetch capacity; broker respects it to prevent consumer OOM
Stream processing primitives: time-windowed aggregations, join streams on shared key

Non-Functional Requirements

Compacted topics must have bounded storage regardless of key cardinality growth rate
Schema registry reads: p99 < 5 ms (hot path for every message deserialization)
Compatibility checks on schema registration must be synchronous and < 200 ms
Consumer fetch rate must be controllable with < 100 ms backpressure response time

Core Data Model

Log Compaction Data Structures

The compaction process maintains an in-memory offset map: a hash map from message key to the highest offset seen for that key across the dirty (uncompacted) portion of the log. The compactor reads the dirty log sequentially, building this map, then rewrites the log retaining only messages whose offset matches the map entry for their key. Tombstone messages (null value) cause the key to be deleted from the offset map and omitted from the compacted output after a configurable delete.retention.ms grace period.

-- Compactor offset map (in-memory)
offset_map: HashMap<Key, HighestOffset>

-- Compaction segments
clean_segment:   [key1:v_latest, key2:v_latest, ...]  -- already compacted
dirty_segment:   [key1:v_old, key3:v1, key1:v_latest, key3:v2, ...]  -- to be compacted

Schema Registry Data Model

schemas (
  id            SERIAL PRIMARY KEY,
  subject       TEXT NOT NULL,          -- e.g. "payments-value"
  version       INTEGER NOT NULL,
  schema_json   TEXT NOT NULL,          -- full Avro/Protobuf schema
  schema_hash   TEXT NOT NULL UNIQUE,   -- SHA-256 for dedup
  compatibility TEXT NOT NULL,          -- BACKWARD, FORWARD, FULL, NONE
  created_at    TIMESTAMPTZ NOT NULL,
  UNIQUE (subject, version)
)

Messages embed only the schema ID (4 bytes) in a magic-byte header, not the full schema. Consumers look up the schema by ID from the registry, with aggressive local caching since schema IDs are immutable once registered.

Key Algorithms and Logic

Log Compaction Scheduling

Each broker runs a compaction thread pool. Partitions are eligible for compaction when the dirty ratio (uncompacted bytes / total bytes) exceeds a threshold (default 50%). The compactor selects the partition with the highest dirty ratio, builds the offset map by reading the dirty segments, and performs a multi-pass merge: in each pass, it reads a dirty segment and writes a compacted output segment, skipping messages for keys with higher offsets elsewhere. The clean/dirty boundary advances as passes complete. Compaction runs at configurable I/O throttle to avoid impacting producer/consumer throughput.

Schema Compatibility Checking

On schema registration, the registry fetches all previous versions for the subject and applies the configured compatibility mode:

BACKWARD: new schema can read data written by the previous schema (consumers can upgrade before producers)
FORWARD: previous schema can read data written by the new schema (producers can upgrade before consumers)
FULL: both backward and forward compatible (safest; most restrictive)

Compatibility is checked by attempting to resolve the new schema against each prior version using the Avro/Protobuf schema resolution rules. Adding optional fields with defaults is backward compatible. Removing fields without defaults is not. The check fails fast and returns a 409 Conflict with a detailed explanation of the incompatibility.

Consumer Backpressure via Fetch Quota

Consumers specify max_bytes and max_wait_ms in each fetch request. The broker responds with at most max_bytes regardless of how many messages are available. If a consumer's processing loop is slow, it simply issues the next fetch only after finishing the previous batch — a pull model naturally implements backpressure. For push-based consumers (WebSocket or server-sent events delivery), the broker tracks per-consumer in-flight byte counts and pauses pushing when the consumer's receive buffer is full, relying on TCP flow control as the backpressure signal.

Time-Window Aggregation in Stream Processing

A stream processing layer (e.g., a stateful consumer application) performs windowed aggregations by maintaining in-memory state stores keyed by (window_start, group_key). Tumbling windows close at fixed intervals; sliding windows overlap. Watermarks (derived from event timestamps or processing time) trigger window closure and downstream emission. State stores are checkpointed to a compacted topic for fault tolerance — on restart, the processor rebuilds state by replaying from the compacted topic.

API Design

POST /subjects/{subject}/versions: register a new schema version; returns schema ID or 409 on incompatibility
GET /schemas/ids/{id}: fetch schema by ID; cached aggressively by consumers
GET /subjects/{subject}/versions: list all versions for a subject
GET /subjects/{subject}/versions/latest: fetch the latest registered version
POST /compatibility/subjects/{subject}/versions/{version}: test a schema for compatibility without registering
DELETE /subjects/{subject}/versions/{version}: soft-delete a version (marks deprecated, schema ID still resolvable)

Scalability Considerations

Schema Registry High Availability

The schema registry runs as a multi-node cluster with a leader for writes and followers for reads, backed by a compacted internal Kafka topic as its storage layer. This eliminates the need for an external database. Followers serve reads from their local in-memory cache of the topic, which is eventually consistent with the leader by at most one replication lag interval.

Compaction Resource Isolation

Compaction is I/O intensive. Brokers dedicate separate disk I/O quotas for compaction vs. producer/consumer traffic using cgroups or I/O throttle APIs. Compaction threads are lower priority than network I/O threads. This prevents compaction from causing read latency spikes on active partitions.

Interview Discussion Points

What happens to a consumer that reads a compacted topic and skips offsets? How does it detect the gap?
How does schema evolution interact with long-running stream processing jobs that cannot be redeployed immediately?
How would you implement a global sequence number across partitions for total ordering in a compacted topic?
What are the failure modes of event-time watermarks in a stream processor with late-arriving events?
How does log compaction interact with consumer group offset commits — can a committed offset be compacted away?

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “How does log compaction work in an event streaming system?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Log compaction retains only the latest record per key by scanning log segments and writing a compacted output that drops superseded entries. Tombstone records (null-value messages) signal deletion so consumers can purge a key's state entirely. The broker runs compaction in the background without blocking producers or consumers, ensuring the log converges to a minimal yet correct snapshot of current state.”
}
},
{
“@type”: “Question”,
“name”: “How does a schema registry enforce compatibility in an event streaming platform?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “A schema registry stores versioned Avro, Protobuf, or JSON Schema definitions and enforces a compatibility mode—BACKWARD, FORWARD, or FULL—on every registration request. When a producer publishes a new schema version, the registry validates it against the existing version before assigning an ID. Consumers embed only the schema ID in the message envelope and fetch the full definition on demand, decoupling schema evolution from deployment cycles.”
}
},
{
“@type”: “Question”,
“name”: “How do you handle consumer backpressure in an event streaming system?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Consumers signal backpressure by calling pause() on specific partitions when their processing queue exceeds a high-water mark, causing the poll loop to stop fetching records for those partitions without revoking the assignment. Once the queue drains below a low-water mark, the consumer calls resume() to re-enable fetching. This cooperative mechanism prevents out-of-memory crashes while keeping the broker's offset tracking intact.”
}
},
{
“@type”: “Question”,
“name”: “What happens during a consumer group rebalance in an event streaming system?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “When a consumer joins or leaves a group, the group coordinator triggers a rebalance: all members rejoin, submit their topic subscriptions, and the elected leader runs the partition assignment strategy (range, round-robin, or sticky). Each member then fetches its assigned partitions and resumes from committed offsets. Incremental cooperative rebalancing minimizes disruption by only revoking partitions that must move, letting unaffected partitions continue processing throughout.”
}
}
]
}