User Activity Feed Low-Level Design: Event Ingestion, Fan-Out, and Personalized Ranking

User Activity Feed: Overview

A user activity feed aggregates events from actors a user follows and surfaces them in reverse-chronological or ranked order. Designing this system at low level requires decisions about event ingestion, fan-out strategy, timeline storage, ranking, and pagination. Each decision has concrete latency and throughput tradeoffs.

Activity Event Types

Events flowing into the feed system represent discrete user actions:

follow — actor_id followed target_user_id
like — actor_id liked object (post, comment, photo)
comment — actor_id commented on object_id
post — actor_id created new content
share — actor_id shared object_id
mention — actor_id mentioned target_user_id in content

Each event is represented as a triple: (actor, verb, object) — an Activity Streams model. The event is written to an ActivityEvent table and then dispatched for fan-out.

Fan-Out Strategies

Fan-Out-on-Write (Push Model)

When an actor publishes an event, the system immediately writes a feed entry for each follower. At read time, the follower's feed is a simple pre-built list requiring no aggregation.

Pro: O(1) read latency — just fetch pre-built feed entries from Redis sorted set.
Con: Write amplification — a user with 10 million followers triggers 10 million writes. Storage scales with follower count × events.

Fan-Out-on-Read (Pull Model)

At read time, the system fetches recent events from each actor the user follows and merges them. No pre-computation on write.

Pro: Write is cheap — one row in ActivityEvent.
Con: Read is expensive — must query N actors, merge N sorted lists, apply ranking. Latency grows with follow count.

Hybrid Model (Recommended)

Push for normal users (follower count below threshold, e.g., 10,000). Pull for celebrities (follower count above threshold). At read time, merge the pre-built push feed with a pull fetch from high-follower actors the user follows. This caps write amplification while keeping read latency acceptable.

A FeedConfig table stores per-user strategy. A background job re-classifies users as follower counts change.

Timeline Storage: Redis Sorted Set

Each user's feed is stored as a Redis sorted set:

Key: feed:{user_id}
Member: activity_id
Score: Unix timestamp (milliseconds) or composite score for ranked feeds

Operations:

Fan-out write: ZADD feed:{follower_id} {timestamp} {activity_id}
Read feed page: ZREVRANGEBYSCORE feed:{user_id} {cursor_score} -inf LIMIT 0 {page_size}
Trim old entries: ZREMRANGEBYSCORE feed:{user_id} -inf {cutoff_score}

TTL-based pruning removes entries older than 30 days. If the sorted set exceeds a max size (e.g., 1,000 entries), trim the lowest-scored members.

Pagination via Cursor

Use score-based cursor pagination instead of OFFSET. The client receives the score of the last item on the current page and passes it as the cursor on the next request. This avoids the drift problem where new entries shift OFFSET-based pages.

For ranked feeds (non-chronological), the cursor encodes both score and activity_id to handle ties deterministically.

Ranking Signals

A ranked feed replaces raw timestamp scores with a composite score computed at fan-out time or at read time:

Recency: Decay function — score *= e^(-lambda * age_hours)
Actor affinity: Interaction history between viewer and actor (likes, replies, profile views)
Content type weight: Videos score higher than plain text events by configurable multiplier
Engagement velocity: Items accumulating likes/comments quickly get a boost

For simplicity, recency-only scoring is computed at write time. Full ML-based ranking is computed at read time on a candidate set fetched from Redis.

SQL Schema

-- Core event log
CREATE TABLE ActivityEvent (
    id          BIGSERIAL PRIMARY KEY,
    actor_id    BIGINT NOT NULL,
    verb        VARCHAR(32) NOT NULL,       -- follow, like, comment, post
    object_type VARCHAR(32) NOT NULL,       -- post, comment, user
    object_id   BIGINT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_activityevent_actor ON ActivityEvent(actor_id, created_at DESC);

-- Pre-built feed entries per user (push model)
CREATE TABLE FeedEntry (
    user_id       BIGINT NOT NULL,
    activity_id   BIGINT NOT NULL REFERENCES ActivityEvent(id),
    score         DOUBLE PRECISION NOT NULL,  -- composite ranking score
    inserted_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (user_id, activity_id)
);
CREATE INDEX idx_feedentry_user_score ON FeedEntry(user_id, score DESC);

-- Per-user fan-out strategy
CREATE TABLE FeedConfig (
    user_id         BIGINT PRIMARY KEY,
    strategy        VARCHAR(16) NOT NULL DEFAULT 'push',  -- push / pull / hybrid
    follower_count  INT NOT NULL DEFAULT 0,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Python Implementation

import redis
import time
import json
from dataclasses import dataclass
from typing import List, Optional

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

FEED_MAX_SIZE = 1000
FEED_TTL_DAYS = 30
CELEBRITY_THRESHOLD = 10_000

@dataclass
class ActivityEvent:
    id: int
    actor_id: int
    verb: str
    object_type: str
    object_id: int
    created_at: float

def publish_activity(actor_id: int, verb: str, object_type: str, object_id: int) -> int:
    """Insert event into DB and trigger fan-out. Returns activity_id."""
    # Pseudocode DB insert
    activity_id = db.execute(
        "INSERT INTO ActivityEvent(actor_id,verb,object_type,object_id,created_at)"
        " VALUES(%s,%s,%s,%s,NOW()) RETURNING id",
        (actor_id, verb, object_type, object_id)
    ).fetchone()[0]

    config = db.execute(
        "SELECT strategy FROM FeedConfig WHERE user_id=%s", (actor_id,)
    ).fetchone()
    strategy = config[0] if config else 'push'

    if strategy in ('push', 'hybrid'):
        fan_out.delay(activity_id, actor_id)  # async via Celery/queue

    return activity_id

def fan_out(activity_id: int, actor_id: int) -> None:
    """Write feed entry to each follower's Redis sorted set."""
    score = time.time()
    cutoff = score - (FEED_TTL_DAYS * 86400)

    follower_ids = db.execute(
        "SELECT follower_id FROM Follow WHERE followee_id=%s", (actor_id,)
    ).fetchall()

    pipe = r.pipeline(transaction=False)
    for (follower_id,) in follower_ids:
        feed_key = f"feed:{follower_id}"
        pipe.zadd(feed_key, {str(activity_id): score})
        pipe.zremrangebyscore(feed_key, '-inf', cutoff)
        pipe.zremrangebyrank(feed_key, 0, -(FEED_MAX_SIZE + 1))
    pipe.execute()

def get_feed(user_id: int, cursor: Optional[float], limit: int = 20) -> dict:
    """Return paginated feed entries. cursor is the score of the last seen item."""
    feed_key = f"feed:{user_id}"
    max_score = cursor if cursor else '+inf'
    # Exclude the cursor item itself by using exclusive range
    exclude = f"({cursor}" if cursor else '+inf'

    raw = r.zrevrangebyscore(
        feed_key, exclude, '-inf',
        start=0, num=limit, withscores=True
    )

    activity_ids = [int(member) for member, score in raw]
    next_cursor = raw[-1][1] if raw else None

    # Fetch activity details from DB or cache
    events = fetch_activity_events(activity_ids)

    return {
        "events": events,
        "next_cursor": next_cursor,
        "has_more": len(raw) == limit
    }

def fetch_activity_events(activity_ids: List[int]) -> List[dict]:
    if not activity_ids:
        return []
    placeholders = ','.join(['%s'] * len(activity_ids))
    rows = db.execute(
        f"SELECT id,actor_id,verb,object_type,object_id,created_at"
        f" FROM ActivityEvent WHERE id IN ({placeholders})"
        f" ORDER BY created_at DESC",
        activity_ids
    ).fetchall()
    return [dict(zip(['id','actor_id','verb','object_type','object_id','created_at'], r)) for r in rows]

Key Design Decisions Summary

Fan-out-on-write for normal users keeps read O(1) at cost of write amplification.
Celebrity threshold (10K followers) triggers hybrid: pull their events at read time.
Redis sorted set provides O(log N) writes and range reads with automatic ordering.
Score-based cursor pagination avoids OFFSET drift on high-velocity feeds.
TTL pruning (30 days) and max-size trim prevent unbounded feed growth.

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “How do you choose between fan-out-on-write and fan-out-on-read for an activity feed?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Fan-out-on-write (push) pre-builds feed entries at event time, making reads O(1) but causing write amplification proportional to follower count. Fan-out-on-read (pull) keeps writes cheap but makes reads expensive as follow count grows. The hybrid approach uses push for users below a follower threshold (e.g., 10,000) and pull for celebrities, merging both at read time.”
}
},
{
“@type”: “Question”,
“name”: “What is the celebrity problem in feed systems?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “The celebrity problem occurs when a high-follower user (e.g., 10 million followers) publishes an event, triggering 10 million fan-out writes simultaneously. This spikes write throughput and strains storage. The solution is to skip push fan-out for celebrities and instead pull their recent events at read time for all followers who subscribe to them.”
}
},
{
“@type”: “Question”,
“name”: “How does cursor-based pagination work for activity feeds?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Cursor-based pagination uses the score (timestamp or ranking score) of the last item returned as the starting point for the next page. In Redis, ZREVRANGEBYSCORE with an exclusive lower bound (the cursor score) fetches the next page without offset drift. Unlike OFFSET pagination, this remains correct even when new items are inserted.”
}
},
{
“@type”: “Question”,
“name”: “What ranking signals are used in a personalized activity feed?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Common ranking signals include recency (time-decayed score), actor affinity (how often the viewer interacts with the actor via likes or replies), content type weight (videos rank higher than text), and engagement velocity (items accumulating interactions quickly get a boost). These signals are combined into a composite score, either at fan-out time for simplicity or at read time for ML-based personalization.”
}
}
]
}

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “How does fan-out-on-write work for activity feeds?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “When a user performs an action, the system immediately writes the activity to the feed timelines of all followers; reads are instant but writes are expensive for users with many followers.”
}
},
{
“@type”: “Question”,
“name”: “What is the celebrity problem in fan-out-on-write?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Users with millions of followers (celebrities) would require millions of write operations per activity; the hybrid strategy pushes to regular followers but pulls celebrity activities at read time to merge with the pre-computed feed.”
}
},
{
“@type”: “Question”,
“name”: “How is feed pagination implemented with Redis sorted sets?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Each user's feed is a Redis sorted set keyed by user_id; ZREVRANGEBYSCORE with a timestamp cursor returns the next page of activities without offset-based scanning.”
}
},
{
“@type”: “Question”,
“name”: “How are ranking signals applied to the activity feed?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “A scoring function combines recency (exponential decay), actor affinity (interaction history), and content type weights; feed entries are re-scored periodically and their sorted set scores updated.”
}
}
]
}