Designing a social media news feed like Twitter/X is one of the most popular system design interview questions. It tests your understanding of data modeling, fanout strategies, caching, ranking algorithms, and handling extreme scale differences between regular users and celebrities. This guide covers the end-to-end architecture for generating and serving personalized timelines at scale.
Core Data Model
Three main entities: (1) Users — user_id, username, profile, follower_count, following_count. (2) Tweets — tweet_id (Snowflake time-sorted), user_id, content, media_urls, created_at, like_count, retweet_count, reply_count. (3) Follow relationships — follower_id, followee_id, created_at. Stored in a graph database or a wide-column store partitioned by follower_id. The timeline query: “show me the most recent tweets from everyone I follow, ranked by relevance.” This is deceptively complex — a user following 500 accounts needs tweets from all 500, sorted, ranked, and paginated. Naive approach: SELECT * FROM tweets WHERE user_id IN (SELECT followee_id FROM follows WHERE follower_id = me) ORDER BY created_at DESC LIMIT 20. This is a scatter-gather across potentially hundreds of user tweet streams. At Twitter scale (500M+ users), this query is impossibly slow for every timeline load.
Fanout-on-Write (Push Model)
When a user posts a tweet, immediately push it to the timeline cache of every follower. Each user has a pre-computed timeline stored in Redis (a sorted set: ZADD timeline:{user_id} {timestamp} {tweet_id}). When User A (with 1000 followers) posts a tweet: the fanout service reads A follower list, and for each follower, adds the tweet_id to their timeline cache. When a follower loads their timeline: read from Redis (ZREVRANGE timeline:{user_id} 0 19) — a simple sorted set query returning the 20 most recent tweet IDs. Fetch the full tweet objects from a tweet cache or database. Pros: timeline reads are extremely fast (O(1) Redis lookup, no join). This is critical because reads outnumber writes 100:1 on social media. Cons: write amplification — a user with 1M followers causes 1M Redis writes per tweet. Storage: every user timeline is stored separately, even though many users share the same followed accounts. Twitter used this approach for years.
Fanout-on-Read (Pull Model)
When a user loads their timeline, fetch tweets from each followed account in real-time and merge. No pre-computation. Implementation: get the user following list, for each followed account fetch their N most recent tweets (from a per-user tweet cache), merge all tweets and sort by timestamp or relevance score, return the top 20. Pros: no write amplification (posting a tweet is a single write), no storage for pre-computed timelines. Cons: timeline reads are slow — fetching from 500 accounts and merging takes time. The merge operation is essentially a K-way merge of sorted lists (O(N * K) where N is tweets per account and K is accounts followed). For users following thousands of accounts, this is too slow for real-time. Use fanout-on-read for: users with very few followers (low write amplification benefit from push), celebrity timelines (a celebrity following 100 accounts can pull efficiently), and non-real-time contexts (email digests, weekly summaries).
The Celebrity Problem and Hybrid Approach
The celebrity problem: a user with 50 million followers posts a tweet. With fanout-on-write, this generates 50 million Redis writes — for a single tweet. If the celebrity tweets 10 times a day, that is 500 million writes. This overwhelms the fanout system. Hybrid solution (what Twitter actually uses): (1) For regular users (fewer than ~10,000 followers): use fanout-on-write. Their tweets are pushed to follower timelines. The write amplification is manageable. (2) For celebrities (more than ~10,000 followers): do NOT fanout. Their tweets are stored separately. When a follower loads their timeline, the system merges the pre-computed timeline (from regular users, fanout-on-write) with a real-time fetch of celebrity tweets (fanout-on-read). The merge adds a small amount of latency but avoids the massive write amplification. Threshold: the exact follower count threshold is tunable. Twitter experimented with various thresholds. The key insight is that a small number of accounts (celebrities) cause the majority of fanout volume. Excluding them from push and handling them with pull dramatically reduces write load while keeping read latency acceptable.
Timeline Ranking
Chronological timeline (reverse chronological, most recent first) was Twitter original approach. Simple but problematic: users who check the app every few hours miss important tweets buried under a flood of less relevant content. Ranked timeline: an ML model scores each candidate tweet for a specific user and ranks by predicted engagement (probability the user will like, retweet, or reply). Ranking features: (1) Tweet features — recency, media type (images get more engagement), length, hashtags. (2) Author features — relationship strength (how often the user interacts with this author), author engagement rate, follower count. (3) User features — user interests (inferred from past engagement), active times, device type. (4) Interaction features — has the user interacted with similar content recently? Is this topic trending? The ranking model (typically a neural network or gradient boosted tree) runs on a candidate set (the most recent ~500 tweets from the chronological timeline) and reorders them by predicted score. The top-ranked tweets are shown first. Twitter/X provides a toggle between “For You” (ranked) and “Following” (chronological) to satisfy both preferences.
Caching and Storage Architecture
Multi-layer caching: (1) Timeline cache (Redis) — pre-computed timelines for each user. Store the most recent 800 tweet IDs per user. Redis sorted set with timestamp as the score. Memory: 800 tweet IDs * 8 bytes * 500M users = ~3.2 TB. Spread across a Redis cluster. (2) Tweet cache (Redis/Memcached) — full tweet objects cached by tweet_id. Hot tweets (from popular accounts, trending topics) are cached with high TTL. Cold tweets are fetched from the database on cache miss. (3) User cache — user profiles cached for display in the timeline. Database: tweets are stored in a distributed database partitioned by user_id (for fetching a user tweets) with a secondary index on tweet_id (for fetching individual tweets). Cassandra, DynamoDB, or sharded MySQL. Follow graph: stored in a graph-optimized store or a wide-column database. Partitioned by follower_id for “who does this user follow?” and by followee_id for “who follows this user?” (dual partitioning for bi-directional lookups). Media storage: images and videos in S3/object storage, served via CDN.