Designing a social media news feed like Twitter/X is one of the most popular system design interview questions. It tests your understanding of data modeling, fanout strategies, caching, ranking algorithms, and handling extreme scale differences between regular users and celebrities. This guide covers the end-to-end architecture for generating and serving personalized timelines at scale.
Core Data Model
Three main entities: (1) Users — user_id, username, profile, follower_count, following_count. (2) Tweets — tweet_id (Snowflake time-sorted), user_id, content, media_urls, created_at, like_count, retweet_count, reply_count. (3) Follow relationships — follower_id, followee_id, created_at. Stored in a graph database or a wide-column store partitioned by follower_id. The timeline query: “show me the most recent tweets from everyone I follow, ranked by relevance.” This is deceptively complex — a user following 500 accounts needs tweets from all 500, sorted, ranked, and paginated. Naive approach: SELECT * FROM tweets WHERE user_id IN (SELECT followee_id FROM follows WHERE follower_id = me) ORDER BY created_at DESC LIMIT 20. This is a scatter-gather across potentially hundreds of user tweet streams. At Twitter scale (500M+ users), this query is impossibly slow for every timeline load.
Fanout-on-Write (Push Model)
When a user posts a tweet, immediately push it to the timeline cache of every follower. Each user has a pre-computed timeline stored in Redis (a sorted set: ZADD timeline:{user_id} {timestamp} {tweet_id}). When User A (with 1000 followers) posts a tweet: the fanout service reads A follower list, and for each follower, adds the tweet_id to their timeline cache. When a follower loads their timeline: read from Redis (ZREVRANGE timeline:{user_id} 0 19) — a simple sorted set query returning the 20 most recent tweet IDs. Fetch the full tweet objects from a tweet cache or database. Pros: timeline reads are extremely fast (O(1) Redis lookup, no join). This is critical because reads outnumber writes 100:1 on social media. Cons: write amplification — a user with 1M followers causes 1M Redis writes per tweet. Storage: every user timeline is stored separately, even though many users share the same followed accounts. Twitter used this approach for years.
Fanout-on-Read (Pull Model)
When a user loads their timeline, fetch tweets from each followed account in real-time and merge. No pre-computation. Implementation: get the user following list, for each followed account fetch their N most recent tweets (from a per-user tweet cache), merge all tweets and sort by timestamp or relevance score, return the top 20. Pros: no write amplification (posting a tweet is a single write), no storage for pre-computed timelines. Cons: timeline reads are slow — fetching from 500 accounts and merging takes time. The merge operation is essentially a K-way merge of sorted lists (O(N * K) where N is tweets per account and K is accounts followed). For users following thousands of accounts, this is too slow for real-time. Use fanout-on-read for: users with very few followers (low write amplification benefit from push), celebrity timelines (a celebrity following 100 accounts can pull efficiently), and non-real-time contexts (email digests, weekly summaries).
The Celebrity Problem and Hybrid Approach
The celebrity problem: a user with 50 million followers posts a tweet. With fanout-on-write, this generates 50 million Redis writes — for a single tweet. If the celebrity tweets 10 times a day, that is 500 million writes. This overwhelms the fanout system. Hybrid solution (what Twitter actually uses): (1) For regular users (fewer than ~10,000 followers): use fanout-on-write. Their tweets are pushed to follower timelines. The write amplification is manageable. (2) For celebrities (more than ~10,000 followers): do NOT fanout. Their tweets are stored separately. When a follower loads their timeline, the system merges the pre-computed timeline (from regular users, fanout-on-write) with a real-time fetch of celebrity tweets (fanout-on-read). The merge adds a small amount of latency but avoids the massive write amplification. Threshold: the exact follower count threshold is tunable. Twitter experimented with various thresholds. The key insight is that a small number of accounts (celebrities) cause the majority of fanout volume. Excluding them from push and handling them with pull dramatically reduces write load while keeping read latency acceptable.
Timeline Ranking
Chronological timeline (reverse chronological, most recent first) was Twitter original approach. Simple but problematic: users who check the app every few hours miss important tweets buried under a flood of less relevant content. Ranked timeline: an ML model scores each candidate tweet for a specific user and ranks by predicted engagement (probability the user will like, retweet, or reply). Ranking features: (1) Tweet features — recency, media type (images get more engagement), length, hashtags. (2) Author features — relationship strength (how often the user interacts with this author), author engagement rate, follower count. (3) User features — user interests (inferred from past engagement), active times, device type. (4) Interaction features — has the user interacted with similar content recently? Is this topic trending? The ranking model (typically a neural network or gradient boosted tree) runs on a candidate set (the most recent ~500 tweets from the chronological timeline) and reorders them by predicted score. The top-ranked tweets are shown first. Twitter/X provides a toggle between “For You” (ranked) and “Following” (chronological) to satisfy both preferences.
Caching and Storage Architecture
Multi-layer caching: (1) Timeline cache (Redis) — pre-computed timelines for each user. Store the most recent 800 tweet IDs per user. Redis sorted set with timestamp as the score. Memory: 800 tweet IDs * 8 bytes * 500M users = ~3.2 TB. Spread across a Redis cluster. (2) Tweet cache (Redis/Memcached) — full tweet objects cached by tweet_id. Hot tweets (from popular accounts, trending topics) are cached with high TTL. Cold tweets are fetched from the database on cache miss. (3) User cache — user profiles cached for display in the timeline. Database: tweets are stored in a distributed database partitioned by user_id (for fetching a user tweets) with a secondary index on tweet_id (for fetching individual tweets). Cassandra, DynamoDB, or sharded MySQL. Follow graph: stored in a graph-optimized store or a wide-column database. Partitioned by follower_id for “who does this user follow?” and by followee_id for “who follows this user?” (dual partitioning for bi-directional lookups). Media storage: images and videos in S3/object storage, served via CDN.
{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[{“@type”:”Question”,”name”:”What is the difference between fanout-on-write and fanout-on-read for news feed generation?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Fanout-on-write (push): when a user posts, immediately push the tweet to every follower pre-computed timeline cache (Redis sorted set). Timeline reads are fast — just read from cache. Downside: massive write amplification for popular users. A user with 1M followers causes 1M cache writes per tweet. Fanout-on-read (pull): when a user loads their timeline, fetch tweets from each followed account in real-time and merge. No write amplification. Downside: slow reads — must fetch from hundreds of sources and merge for every timeline load. The hybrid approach (what Twitter uses): regular users (under ~10K followers) use fanout-on-write. Celebrities (over ~10K followers) are excluded from fanout. When a follower loads their timeline, the system merges the pre-computed timeline (regular users, fast) with a real-time fetch of celebrity tweets (pull). This eliminates 99% of write amplification while keeping reads fast.”}},{“@type”:”Question”,”name”:”How does Twitter rank tweets in the For You timeline?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”The ranked timeline uses an ML model to score each candidate tweet for the specific user. The model predicts engagement probability (will the user like, retweet, or reply?). Features include: tweet features (recency, media type, length, hashtags), author features (relationship strength with the user, engagement rate, follower count), user features (interests inferred from past behavior, active times), and interaction features (trending topics, similar recent content). The model (neural network or gradient boosted tree) scores a candidate set of approximately 500 recent tweets from the chronological timeline and reorders by predicted score. Top-ranked tweets are shown first. Twitter/X provides a toggle between For You (ranked) and Following (chronological). The ranking system also handles: out-of-network recommendations (tweets from accounts the user does not follow but might enjoy, based on engagement patterns of similar users) and ads insertion (promoted tweets ranked alongside organic content).”}},{“@type”:”Question”,”name”:”How much Redis memory does a Twitter-scale timeline cache require?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Each user timeline stores the most recent ~800 tweet IDs. Each tweet ID is 8 bytes (64-bit Snowflake ID). Redis sorted set overhead per entry is approximately 40 bytes. Total per user: 800 * 48 bytes = ~38 KB. For 500 million users: 500M * 38 KB = ~19 TB. Add Redis overhead (data structures, hash table, memory fragmentation ~1.5x): approximately 28-30 TB. This is spread across a Redis cluster of 50-100 nodes (each with 256-512 GB RAM). Tweet cache (separate from timelines): store hot tweet objects. A tweet object is ~1 KB. Cache the most recent and popular 1 billion tweets: 1 TB. User profile cache: 500M users * 500 bytes = 250 GB. Total Redis footprint: approximately 30-35 TB across the cluster. This is large but well within the capacity of a production Redis cluster.”}},{“@type”:”Question”,”name”:”How do you handle the celebrity problem in news feed design?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”The celebrity problem: a user with 50 million followers creates 50 million cache writes per tweet with fanout-on-write. If they tweet 10 times daily, that is 500 million writes — overwhelming the fanout system and consuming massive Redis capacity. Solution: hybrid fanout. Set a follower threshold (e.g., 10,000). Users below the threshold use fanout-on-write (their tweets are pushed to follower caches). Users above use fanout-on-read (their tweets are not pushed; followers pull them at read time). At timeline load time: read the pre-computed cache (fast, contains regular user tweets), then fetch recent tweets from followed celebrities (a small number of pull queries — most users follow only 10-50 celebrities), merge and rank. The key insight: a tiny percentage of accounts (celebrities) cause the vast majority of fanout volume. Excluding them from push and handling with pull eliminates the bottleneck. The slight increase in read latency (fetching celebrity tweets) is negligible compared to the massive write reduction.”}}]}