Low Level Design: Caching Strategies

Why Caching Matters

Caching is one of the highest-leverage tools in system design. A database read can take 1–10 ms; a cache hit from Redis or Memcached takes under 1 ms. At scale, that gap compounds: a service handling 100k requests/second that saves even 2 ms per request is eliminating 200 seconds of aggregate DB wait per second. Beyond latency, caching reduces DB load, cuts infrastructure costs, and protects downstream systems during traffic spikes.

Cache-Aside (Lazy Loading)

Cache-aside is the most common pattern. The application owns the cache interaction: on a read, check the cache first. On a miss, fetch from the database, populate the cache, and return the result. On a write, invalidate or update the cache entry.

def get_user(user_id):
    user = cache.get(f"user:{user_id}")
    if user is None:
        user = db.query("SELECT * FROM users WHERE id = %s", user_id)
        cache.set(f"user:{user_id}", user, ttl=300)
    return user

Trade-offs: Cold start problem — a freshly deployed or flushed cache means every request is a cache miss until data warms up. Stale data risk exists between DB writes and cache invalidation. Cache-aside is resilient to cache failures since the app falls back to the DB automatically.

Read-Through Cache

In a read-through pattern, the cache sits in front of the database. The application only ever talks to the cache. On a miss, the cache itself fetches the data from the DB, populates the entry, and returns it. This simplifies application code — the app doesn’t need cache-miss handling logic. Libraries like AWS DAX (for DynamoDB) implement read-through transparently.

Trade-offs: The first read for any key is always slow (cache miss + DB fetch). The cache layer must understand your data model, which can add coupling. Works best when your cache provider supports it natively.

Write-Through Cache

Every write goes to both the cache and the database synchronously, in the same operation. The write is not acknowledged until both succeed. This guarantees strong consistency — reads always see the latest data.

Trade-offs: Writes are slower because you pay the cost of two writes on every mutation. Cache is populated with data that may never be read (write-heavy workloads with infrequent reads waste cache space). Most appropriate when read latency and consistency are both critical.

Write-Behind (Write-Back) Cache

Writes go to the cache immediately, and the cache flushes dirty entries to the database asynchronously — typically on a timer or when an entry is evicted. This maximizes write throughput since the application only waits for an in-memory write.

Trade-offs: Risk of data loss if the cache node crashes before flushing. DB and cache are temporarily inconsistent. Appropriate for high write throughput workloads where occasional data loss is acceptable (e.g., analytics counters, activity feeds). Not appropriate for financial transactions.

Write-Around Cache

Writes go directly to the database, bypassing the cache entirely. Only reads populate the cache. This avoids polluting the cache with write-once or rarely-read data — for example, bulk data imports, audit logs, or archival writes that will never be read through the hot path.

Trade-offs: The first read after a write is always a cache miss. Useful when write patterns differ significantly from read patterns and you want to preserve cache space for frequently-read data.

Eviction Policies

When the cache is full, an eviction policy determines which entries to remove:

• LRU (Least Recently Used): Evict the entry that was accessed longest ago. Works well for most workloads. Redis default. Implemented with a doubly linked list + hash map (O(1) get and put).
• LFU (Least Frequently Used): Evict the entry accessed fewest times. Better for workloads with a stable hot set. More complex to implement. Redis supports it via allkeys-lfu policy.
• TTL (Time-To-Live): Entries expire after a fixed duration regardless of access pattern. Simple and predictable. Essential for data freshness guarantees.
• FIFO (First In, First Out): Evict the oldest-inserted entry. Simple but ignores access frequency — rarely optimal.
• Random: Evict a random entry. Surprisingly competitive with LRU in some workloads, with much lower overhead.

TTL Selection

TTL is a critical knob. Too short: frequent cache misses, high DB load, good data freshness. Too long: stale data served to users, but high cache hit rates and low DB load. The right value depends on how often the underlying data changes and how stale is acceptable.

Practical strategies: use shorter TTLs for mutable user-facing data (profile info: 60–300s), longer TTLs for slow-changing reference data (product catalog: hours), and no TTL for immutable data (content addressed by hash). Jitter TTLs (add ±10–20% randomness) to prevent synchronized mass expiry across a cluster.

Cache Stampede (Thundering Herd)

When a popular cache key expires, many concurrent requests simultaneously miss the cache and all rush to the database to recompute the same value. This thundering herd can overwhelm the DB and cause cascading failures.

Solutions:

• Mutex/lock-based cache fill: Only one request acquires a lock to recompute the value; others wait or serve stale data. Simple but adds latency for waiting requests.
• Probabilistic early expiry (PER): Before the TTL expires, each request has a small increasing probability of proactively refreshing the cache. Spreads the refresh work over time with no coordination needed.
• Request coalescing: Deduplicate in-flight requests for the same key. Return the same future/promise to all concurrent waiters. First requester fetches; others get the result when it arrives.
• Background refresh: A background job refreshes popular keys before they expire, ensuring the cache is always warm.

Negative Caching

If your application queries the DB for a key that doesn’t exist (e.g., a user lookup for a non-existent ID), and you don’t cache the miss, every repeat request goes to the DB. Negative caching means storing a sentinel value (e.g., NULL or an empty object) with a short TTL for keys that produce no result. This prevents DB hammering from repeated lookups of non-existent entities — common in abuse scenarios (bots probing IDs) or after deletions.

Distributed Cache Topology and Consistency

Client-side cache: Each application instance maintains its own local cache (e.g., in-process HashMap or Caffeine). No network hop, lowest latency, but each instance has its own copy — consistency is hard and memory usage multiplies with instance count.

Dedicated cache cluster (Redis, Memcached): All application instances share a single cache. Consistent view, but adds a network hop (~0.5–1 ms intra-datacenter). Sharding across cache nodes via consistent hashing minimizes key remapping when nodes are added or removed.

Read-your-writes consistency: After a user writes data, they expect to read their own write immediately. With cache replication, their next read might hit a replica that hasn’t received the invalidation yet. Solutions: route that user’s reads to the primary for a short window after their write, or use sticky sessions to a specific cache node.