The Feed Problem
A social network feed shows a user the recent posts from people they follow, ranked by relevance or recency. The challenge: at scale (Twitter/X: 300M users, Instagram: 2B users), computing each user’s feed from scratch on every request is too slow. The solution involves pre-computing and caching feeds. Two fundamental approaches: fan-out on write (push) and fan-out on read (pull).
Fan-out on Write (Push Model)
When a user creates a post: immediately write it to every follower’s feed cache. Each follower has a pre-materialized feed (a list of post IDs) stored in Redis. On feed read: just fetch from the user’s pre-computed feed. Reads are O(1). Pros: fast reads, works well for most users. Cons: when a celebrity with 100M followers posts: 100M write operations (fan-out). This write spike can overwhelm the system. Used by: early Twitter architecture.
Mitigation: use a queue (Kafka). The post create event goes to Kafka. Worker consumers read from Kafka and fan-out in parallel. Still 100M writes but spread across workers over seconds. Apply a cap: only fan-out to followers who were active in the last 7 days — inactive users can pull on demand.
Fan-out on Read (Pull Model)
No pre-computation on write. On feed read: query all the accounts the user follows, fetch their recent posts, merge and sort. Reads are slow (query N accounts, merge N result sets). Pros: no write amplification — posting by celebrities is cheap. Cons: reads are slow and expensive at scale. Works for: users who follow many people (reading requires merging many result sets); better for celebrity accounts.
Hybrid Approach (Industry Standard)
Combine both: push for regular users ( 10K followers). When reading the feed: retrieve the pre-computed feed from cache (push model). For celebrity posts: fetch their recent posts separately and merge into the feed. This is how Twitter and Instagram actually work. The threshold (10K followers) is configurable and trades write cost vs read complexity.
Feed Ranking
Chronological: simple, but users miss important posts from less-active accounts they follow. Ranked: ML model scores each candidate post by predicted engagement (likes, comments, shares, view time). Features: post age, poster relationship strength (how often you interact), content type (video, photo, text), past engagement with similar content. Two-stage ranking: Stage 1 — retrieve candidate posts (recent posts from followed accounts + sponsored content, ~1000 candidates). Stage 2 — ML ranking model scores all candidates and returns top-50. Heavy ML ranking on 1000 candidates per request at scale requires GPU inference servers or optimized CPU inference.
Pagination and Cursor-Based Feeds
Never use OFFSET for feed pagination — OFFSET scans and discards N rows before returning results. At large offsets: very slow. Use cursor-based pagination: the client sends the ID of the last seen post. Server returns posts older than that ID. Cursor = post_id or (timestamp, post_id) tuple for stability. New posts inserted at the top never shift cursor positions, preventing duplicates or gaps. Feed freshness: when the user pulls to refresh: show new posts above the cursor (new content), not a reset of the entire feed. Show “You have 12 new posts” as a banner before revealing them.
Interview Tips
- Push vs pull is the core trade-off. The interviewer expects you to discuss both, explain the celebrity problem, and propose the hybrid solution.
- Feed storage: Redis Sorted Set (ZADD with timestamp as score, post_id as member) is the standard implementation. ZREVRANGE for most recent posts. ZREMRANGEBYSCORE to trim old entries (keep last 1000 posts per feed).
- The feed is eventually consistent — a new post may take seconds to appear in all followers’ feeds. This is acceptable for social media.
{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “What is the difference between fan-out on write and fan-out on read for news feeds?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Fan-out on write (push model): when a user posts, immediately write the post ID to every follower’s feed cache. Feed reads are instant (O(1) Redis read). Cost: write amplification — one post causes N writes (N = follower count). For a user with 10M followers, one post = 10M Redis writes. Fan-out on read (pull model): when a user opens their feed, fetch recent posts from each account they follow, merge and sort. No write overhead. Cost: read is slow (query N accounts, merge N result sets). For users following 500 accounts, each feed load queries 500 sources. Industry solution: hybrid. Use push for users with fewer than ~10K followers (fast writes, fast reads). Use pull for celebrities (>10K followers). When reading, merge the pre-computed pushed feed with freshly pulled celebrity posts.”
}
},
{
“@type”: “Question”,
“name”: “How does Instagram store and retrieve its news feed at scale?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Instagram uses a pre-computed feed cache in Redis. Each user has a feed stored as a Redis Sorted Set with post IDs scored by timestamp. On post creation: fan out to followers via a distributed task queue (Celery workers consuming from a Kafka topic). Each worker writes the post ID into a batch of follower feed caches. For users with millions of followers (Kylie Jenner): fan-out is skipped; followers pull celebrity posts at read time. Feed read: ZREVRANGE user:{id}:feed 0 49 (top 50 posts by score). On cache miss (cold start or evicted): pull from the posts database, reconstruct the feed, repopulate cache. Feed cap: keep only the most recent 1000 post IDs per user in Redis — older content is paginated from the database.”
}
},
{
“@type”: “Question”,
“name”: “How do you implement feed ranking with a machine learning model?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Two-stage retrieval-ranking architecture: Stage 1 (retrieval): gather ~1000 candidate posts. Sources: recent posts from followed accounts, sponsored/promoted posts, suggested content from accounts you might like. Use simple recency filter: posts from the last 7 days from accounts you follow. Stage 2 (ranking): score all 1000 candidates with a ranking model. Features: post age (recency decay), relationship strength (how often you interact with the poster), content type affinity (do you engage more with videos or photos?), predicted engagement probability (likes, comments, shares). Model: gradient boosting (XGBoost) or a two-tower neural network. Serve via a low-latency inference server (TensorFlow Serving, Triton). Return the top 50 posts by score. The ranking model is retrained daily on new engagement data.”
}
},
{
“@type”: “Question”,
“name”: “How do you handle new posts appearing at the top of a paginated feed?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Offset-based pagination breaks with new content: page 2 (offset 20) shifts when 5 new posts are added to page 1, causing posts 16-20 to appear on both page 1 and page 2. Solution: cursor-based pagination. The cursor is the ID (or timestamp) of the last seen post. ‘Load more’ request: return posts older than cursor_post_id. New posts added above the cursor never shift the position of posts below it. For ‘pull to refresh’ (load newer content): return posts newer than top_cursor_post_id (the ID of the newest post the user has seen). Show a ‘You have 8 new posts’ banner. This separates the refresh (top of feed) from the infinite scroll (bottom of feed) and prevents the jarring experience of content jumping around.”
}
},
{
“@type”: “Question”,
“name”: “How do you prevent a single celebrity post from overloading your fan-out workers?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “When a user with 100M followers posts, the fan-out job (write to 100M feed caches) is massive. Mitigation: (1) Async processing: the post creation API returns immediately; fan-out happens asynchronously via Kafka workers. Users may see a delay of seconds to minutes before followers see the post — acceptable for social media. (2) Rate limiting fan-out workers: spread the 100M writes over 60-120 seconds to avoid a Redis write spike. (3) Skip inactive followers: only fan-out to followers who were active in the last 7 days. A dormant user can pull their feed on login. Reduces fan-out by 50-80% for most celebrity accounts. (4) Hybrid model: for accounts above a follower threshold (1M), skip fan-out entirely and serve their posts via pull at read time. This is the fundamental solution to the celebrity problem.”
}
}
]
}
Asked at: Meta Interview Guide
Asked at: Twitter/X Interview Guide
Asked at: Snap Interview Guide
Asked at: LinkedIn Interview Guide
Asked at: Netflix Interview Guide