# Real-Time Analytics Platform Low-Level Design

Source: https://www.techinterview.org/post/3233467211/lld-realtime-analytics/
Updated: 2026-04-17 · techinterview.org

## Requirements

  - Ingest billions of events per day (page views, clicks, transactions, errors)

  - Support real-time dashboards with <10 second lag

  - Support ad-hoc historical queries (e.g., weekly active users, funnel analysis)

  - 1M events/second at peak

## Lambda Architecture


```
Events → Kafka → Stream Layer  → Real-time serving layer (Druid/Redis)
                             → Batch Layer    → Historical DWH (BigQuery/Snowflake)
                                              → Serving layer (materialized views)
```

Lambda architecture: stream layer for real-time (last 24h), batch layer for historical (all-time). Kappa architecture alternative: process everything through the stream layer with long retention Kafka. Simpler to operate but requires streaming SQL for complex aggregations.

## Event Ingestion


```
Event(event_id UUID, event_type, user_id, session_id, properties JSON,
      client_timestamp, server_timestamp, app_version, platform)
```

Client SDKs batch events locally (10 events or 5 seconds, whichever first) and send to the Collector API. Collector validates, enriches (add server_timestamp, geolocation from IP, device type from user-agent), and publishes to Kafka. Return 200 immediately — no processing in the hot path. Client SDK uses exponential backoff retry on failure. Kafka topic partitioned by user_id for ordered per-user processing.

## Stream Processing (Flink / Kafka Streams)

Real-time aggregations computed on the stream:

  - **Windowed counts**: COUNT events per (event_type, 1-minute window). Write to Druid or ClickHouse.

  - **Unique users**: HyperLogLog approximation (Redis PFADD per (event_type, hour)) for distinct user counts without storing all user IDs.

  - **Funnel steps**: sessionization — group events by session_id within a 30-minute window. For each session, track which funnel steps were completed.

  - **Anomaly detection**: compare current metric against rolling average; alert on >3 standard deviation deviation.

## Real-Time Serving (Druid / ClickHouse)

Druid: append-only columnar database optimized for time-series aggregation queries. Ingests from Kafka in real time, serves sub-second GROUP BY queries on billions of rows. Schema: timestamp, dimensions (event_type, platform, country), metrics (count, sum, cardinality). Query: SELECT COUNT(*) FROM events WHERE __time BETWEEN NOW()-1h AND NOW() AND event_type='page_view' GROUP BY platform.

ClickHouse: similar but simpler to operate. MergeTree engine, columnar storage, vectorized query execution. Excellent for ad-hoc analytics queries.

## Batch Processing (Data Warehouse)

Kafka → S3 (raw event archive, Parquet format, partitioned by date) → Spark/dbt → BigQuery/Snowflake. Batch jobs run hourly. Produce: daily active users (DAU), weekly active users (WAU), retention cohorts, conversion funnels, revenue attribution. These require complex joins and multi-day windows not suited for streaming. Results stored in DWH tables, queryable by data analysts via SQL and BI tools (Looker, Superset).

## HyperLogLog for Unique User Counts

Exact distinct counts require storing all user IDs — O(n) memory. HyperLogLog (HLL) approximates distinct counts with ~1% error using O(log log n) memory. Redis PFADD adds elements to an HLL sketch; PFCOUNT returns the approximate cardinality. PFMERGE combines sketches (for combining hourly → daily). Use HLL for: daily active users, weekly active users, unique visitors per page. When exact counts are required (billing), use a bitmap (Redis BITSET) for known user ID ranges.

## Dashboard Real-Time Updates

Dashboards poll every 10 seconds via REST or subscribe via WebSocket to a metrics stream. The real-time serving layer (Druid/ClickHouse) handles concurrent dashboard queries efficiently due to columnar storage and pre-aggregated segments. Pre-compute common dashboard queries on a schedule (every minute) and cache results in Redis — dashboards serve from cache, reducing query load. Alert thresholds monitored by a separate alerting service that queries the stream layer every minute.

## Key Design Decisions

  - Kafka as the central event bus — decouples ingestion from all downstream processing

  - Lambda architecture: stream for real-time (Druid), batch for historical (DWH)

  - HyperLogLog for approximate unique counts at scale — exact counting requires too much memory

  - Columnar storage (Druid, ClickHouse, BigQuery) is essential for aggregation query performance on billions of rows

  - Client-side batching reduces ingestion API load by 10-100x vs per-event HTTP calls

Databricks system design covers real-time analytics pipelines and streaming. See common questions for [Databricks interview: real-time analytics and data pipeline design](https://www.techinterview.org/post/3233460280/databricks-interview-guide-2026-spark-internals-delta-lake-and-lakehouse-architecture/).

Twitter system design covers real-time analytics at massive scale. Review design patterns for [Twitter/X interview: real-time analytics and event pipeline design](https://www.techinterview.org/post/3233460305/twitter-x-interview-guide-2026-timeline-algorithms-real-time-search-and-content-at-scale/).

LinkedIn system design covers analytics platforms and data pipelines. See design patterns for [LinkedIn interview: analytics platform and data pipeline system design](https://www.techinterview.org/post/3233460278/linkedin-interview-guide-2026-social-graph-engineering-feed-ranking-and-professional-network-scale/).