Question 1

Why does time series data need a specialized database?

Accepted Answer

Time series data has unique access patterns that general-purpose databases handle inefficiently: (1) Append-only writes — every insert is the latest timestamp; there are no updates or random writes. (2) Time-range reads — queries always specify a time window (last hour, last day); no random access by primary key. (3) Massive volume — millions of data points per second at production scale. (4) High compressibility — consecutive values are similar. Purpose-built TSDBs (Prometheus, InfluxDB, TimescaleDB) achieve 10-100x better compression than PostgreSQL for this data (1.37 bytes/sample with Gorilla compression vs. 16 bytes raw), faster time-range queries (data sorted by time, not by primary key), and built-in downsampling (automatic aggregation as data ages).

Question 2

How does Gorilla compression work for time series data?

Accepted Answer

Gorilla compression (from Facebook's 2015 paper) compresses (timestamp, value) pairs using two techniques: Timestamp compression: store the delta-of-delta — the difference between consecutive deltas. For regular 15-second scrape intervals, deltas are all 15s, so delta-of-deltas are all 0 — compresses to ~1.5 bits per timestamp. Irregular intervals have small delta-of-deltas that compress with variable-length encoding. Value compression: XOR consecutive float values instead of storing raw floats. Consecutive metrics values are often similar (CPU usage fluctuates around 40-45%); their XOR has many leading zeros. Leading zeros count + meaningful bits compress with variable-length encoding. Combined result: ~1.37 bytes per (timestamp, value) pair on average versus 16 bytes raw — an 11.7x compression ratio. This is why Prometheus can store months of metrics on modest hardware.

Question 3

What is cardinality in a time series database and why does it matter?

Accepted Answer

Cardinality is the number of unique time series — each distinct combination of metric name and label values is one series. High cardinality is the primary scalability challenge in TSDBs: each series requires its own in-memory chunk and index entry. 1 million active series * 4KB per chunk = 4GB RAM just for in-memory chunks. Label value explosion: using user_id (1M values) * endpoint (10) * status (5) = 50M series — overwhelming any TSDB. Rules: never use unbounded label values (user_id, request_id, order_id) as TSDB labels. Use aggregate metrics per service/endpoint; use logs/traces for per-request data. Prometheus warns when cardinality exceeds thresholds; VictoriaMetrics includes cardinality explorer tools. Designing label schemas upfront prevents cardinality explosions in production.

Question 4

How does downsampling enable long-term metrics retention?

Accepted Answer

Raw metrics at 15-second resolution generate 5,760 samples/day per series. At 1 million series, that's 5.76 billion samples/day — storing this for 2 years requires enormous capacity. Downsampling reduces resolution for older data while preserving statistical properties: keep raw (15s) for 15 days; compute 1-minute aggregates (min, max, avg, sum, count) for 90 days; compute 1-hour aggregates for 2 years. 1-minute aggregates reduce samples by 4x; 1-hour aggregates by 240x. The aggregations (min, max) preserve outlier detection; avg and count enable rate calculations; sum enables totals. Thanos (Prometheus extension) implements downsampling by reading raw blocks from S3, computing aggregates, and writing downsampled blocks back — reducing 2-year storage by 100-200x vs. keeping raw data.

Time Series Database: Low-Level Design

Data Model

Compression

Storage Architecture

Downsampling and Long-Term Retention

Query Language Design