Write-heavy systems must sustain high write throughput without overwhelming the storage layer. Techniques include write batching, asynchronous writes, write coalescing, log-structured storage, and separating the write path from the read path. Each technique trades some latency or consistency for throughput and durability.
Write Batching
Batch multiple individual writes into a single database transaction or API call. Instead of issuing one INSERT per event, buffer events in memory and flush a batch of 100-1000 events every 100-500ms. A single INSERT of 1000 rows completes in 10-50ms; 1000 individual INSERTs take 1000-5000ms. Batching reduces: number of round trips (TCP overhead), number of transactions (commit overhead), and write amplification in the storage engine. Trade-off: events are not persisted until the batch flushes — buffers crash risk losing buffered data.
Async Writes and Event Queuing
Accept the write immediately (return 200 OK) and process it asynchronously. The write goes to a durable queue (Kafka, SQS) and is acknowledged by the client before being persisted to the database. A background consumer processes the queue and writes to the database at a controlled rate. This decouples write throughput (queue ingestion rate) from database write rate. Queues absorb traffic spikes; the database processes at a sustainable rate. Failure: the queue is durable, so no writes are lost even if the database is temporarily unavailable.
Write Coalescing
For entities updated frequently by the same data (counters, metric aggregates), write coalescing combines multiple updates to the same key into a single write. Example: 1000 view events for article_id=42 arriving per second can be coalesced into a single UPDATE articles SET view_count = view_count + N operation per second, rather than 1000 individual increments. Implement with in-memory aggregation (accumulate for N seconds, then flush the aggregate), or using Redis INCRBY as a staging layer with periodic sync to the primary database.
LSM Tree for Write-Heavy Storage
LSM (Log-Structured Merge Tree) databases (Cassandra, RocksDB, LevelDB) are designed for write-heavy workloads. Writes go to a WAL (durable) and an in-memory MemTable (fast). When the MemTable fills, it is flushed to disk as an immutable SSTable. Writes are sequential (fast), not random (slow). Compaction merges SSTables in the background. Write amplification is low; read amplification is higher than B-tree databases. LSM is the right choice when write throughput exceeds what a B-tree database can sustain with acceptable read latency.
Sharding the Write Path
Vertical scaling of a single write primary has limits. Horizontal sharding distributes writes across multiple database shards by partition key (user_id mod N, tenant_id hash). Each shard handles 1/N of the total write load. Tradeoffs: queries that span shards (aggregations, cross-user joins) require scatter-gather. Hot shards (one user with 10x the write rate) may require further subdivision. Consistent hashing minimizes data movement when adding shards. Shard management complexity increases operational overhead — ensure sharding provides a meaningful improvement before introducing it.
Idempotent Write Handlers
Write-heavy systems that use async queues and retries must handle duplicate writes gracefully. Idempotent handlers process the same write twice without producing incorrect results: use a deduplication table keyed on (event_id, source), check before processing, and record after processing. For counter updates, use conditional writes: UPDATE counters SET count = count + N WHERE last_event_id != X AND count + N > count. Idempotency keys ensure that retried writes from network failures don't corrupt data.
Backpressure
When the write rate exceeds what the storage layer can sustain, queues grow without bound — eventually causing OOM or extreme latency. Backpressure signals overload to producers: return 429 Too Many Requests when the queue depth or write latency exceeds a threshold. Producers slow their write rate in response. This prevents the system from entering a death spiral where queued work keeps arriving faster than it can be processed. Combine backpressure with circuit breakers: if the database is completely unavailable, stop accepting writes immediately rather than queuing indefinitely.