Time-series databases are optimized for storing and querying timestamped data: application metrics, IoT sensor readings, financial market data, and infrastructure monitoring. They handle millions of writes per second and provide efficient time-range queries that relational databases struggle with at scale. This guide covers time-series database architecture, storage engines, compression, and when to use each — essential for system design interviews involving monitoring, IoT, or analytics.
What Makes Time-Series Data Special
Time-series data has unique characteristics that general-purpose databases handle poorly: (1) Write-heavy — new data points arrive continuously. A monitoring system with 10,000 servers reporting 100 metrics every 10 seconds generates 100,000 writes per second. (2) Append-only — data is almost never updated. A CPU measurement at 14:30:05 is immutable. This enables optimizations impossible for mutable data. (3) Time-ordered access — queries almost always include a time range (show CPU for the last hour, aggregate sales per day for the past year). Sequential access patterns enable efficient compression and storage. (4) High cardinality — the combination of metric name + labels (host, region, service) creates millions of unique time series. Each needs its own storage and indexing. (5) Decreasing value over time — second-resolution data from last hour is valuable. Second-resolution data from last year is not — aggregate to hourly or daily. This enables aggressive downsampling and tiered retention. General databases: PostgreSQL can store time-series data, but at scale the write throughput, storage efficiency, and query performance for time-range aggregations are 10-100x worse than a purpose-built TSDB.
Storage Engine Architecture
Time-series databases use specialized storage engines: (1) LSM-tree based (InfluxDB, Cassandra) — data is written to an in-memory buffer (memtable), then flushed to disk as immutable sorted files (SSTables). Periodic compaction merges files. Excellent write throughput (all writes are sequential). Read performance depends on the number of SSTables to merge. (2) Columnar storage (InfluxDB IOx, ClickHouse, Apache Parquet) — store each column (timestamp, value, tags) separately. Column compression is extremely efficient: a timestamp column with regular 10-second intervals compresses to nearly zero using delta-of-delta encoding. Value columns compress well with gorilla encoding (XOR of consecutive values — similar values produce small XOR results with many leading zeros). (3) Chunk-based (Prometheus TSDB, TimescaleDB) — divide time into chunks (2 hours in Prometheus, configurable in TimescaleDB). Each chunk is a self-contained unit. Recent chunks are in memory for fast writes and queries. Older chunks are on disk or compressed. Old chunks can be deleted by dropping the entire chunk (O(1)) rather than deleting individual rows. Data compression: time-series data compresses 10-20x due to the regularity of timestamps and the similarity of consecutive values. Prometheus achieves approximately 1.3 bytes per data point after compression.
Prometheus vs InfluxDB vs TimescaleDB
Prometheus: pull-based metrics collection for cloud-native monitoring. Scrapes /metrics endpoints every 15-30 seconds. Local TSDB storage with 15-day default retention. PromQL query language for aggregation and alerting. Best for: Kubernetes and microservice monitoring, alerting. Not designed for: long-term storage (use Thanos or Cortex for remote storage to S3), high-cardinality data (performance degrades with millions of unique time series), or non-metrics use cases (logs, events). InfluxDB: push-based time-series platform. Supports high write throughput (millions of points per second). Flux query language (functional, pipeline-based). Built-in retention policies, downsampling, and alerting. Best for: IoT sensor data, application metrics, and analytics. InfluxDB 3.0 (IOx) uses Apache Arrow and Parquet for columnar storage. TimescaleDB: PostgreSQL extension for time-series. Full SQL support, JOINs with relational data, and PostgreSQL ecosystem (pg_stat, extensions, tooling). Automatic time-based partitioning (hypertables). Best for: teams that want SQL compatibility, mixed workloads (time-series + relational in one database), and applications already using PostgreSQL. Performance is 10-100x better than vanilla PostgreSQL for time-series queries.
Downsampling and Retention Policies
Raw data at full resolution is expensive to store long-term. A metric sampled every 10 seconds generates 8,640 data points per day per time series. With 1 million time series: 8.64 billion points per day. Downsampling: aggregate raw data into lower-resolution summaries. Keep 10-second resolution for the last 24 hours, 1-minute averages for the last 7 days, 1-hour averages for the last 90 days, and 1-day averages for the last 2 years. Each level stores the mean, min, max, and count — enough to reconstruct approximate behavior. Retention policies: automatically delete data older than the retention period. InfluxDB: CREATE RETENTION POLICY one_week ON mydb DURATION 7d. Prometheus: –storage.tsdb.retention.time=15d. TimescaleDB: add_retention_policy with a hypertable interval. The combination of downsampling and retention reduces storage by 100-1000x while preserving the ability to query historical trends. Continuous aggregates (TimescaleDB) or continuous queries (InfluxDB) automatically maintain the downsampled materialized views as new data arrives.
Time-Series in System Design Interviews
When to mention time-series databases: (1) Monitoring and observability — “we use Prometheus for metrics, with Thanos for long-term storage.” (2) IoT sensor data — “sensor readings are written to InfluxDB at 100K writes/sec, with 7-day raw retention and 1-year downsampled retention.” (3) Financial data — “stock prices are stored in TimescaleDB with millisecond resolution, enabling SQL window functions for moving averages.” (4) Analytics dashboards — “time-bucketed aggregations (orders per hour, revenue per day) are pre-computed in continuous aggregates.” Key design decisions to discuss: write throughput requirements (determines the TSDB choice), retention policy (how long to keep raw vs aggregated data), query patterns (real-time dashboards need fast recent queries; historical analysis needs efficient range scans), and cardinality (the number of unique time series — high cardinality degrades performance in Prometheus). Do not use a TSDB when: the data is not time-stamped, you need complex relational queries (JOINs, transactions), or the dataset is small enough for PostgreSQL with a timestamp index.
{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[{“@type”:”Question”,”name”:”Why are time-series databases needed instead of regular databases?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Time-series data has characteristics that general-purpose databases handle poorly: (1) Extreme write throughput — 10,000 servers reporting 100 metrics every 10 seconds = 100,000 writes/sec. PostgreSQL struggles at this sustained rate. (2) Append-only data — timestamps are immutable, enabling storage optimizations (compression, sequential writes) impossible for mutable data. (3) Time-range queries — almost all queries include WHERE timestamp BETWEEN x AND y. TSDBs optimize for this access pattern with time-based partitioning. (4) Compression — regular timestamps compress 10-20x using delta-of-delta encoding. Similar consecutive values compress with gorilla/XOR encoding. Prometheus achieves 1.3 bytes per data point. (5) Retention management — dropping an entire time chunk is O(1) vs deleting millions of individual rows. A purpose-built TSDB handles these patterns 10-100x more efficiently than PostgreSQL in both write throughput and query performance.”}},{“@type”:”Question”,”name”:”How do you choose between Prometheus, InfluxDB, and TimescaleDB?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Prometheus: pull-based metrics for Kubernetes and microservices. Scrapes /metrics endpoints. Local storage with 15-day default retention. PromQL for queries and alerting. Best for: cloud-native monitoring. Limitations: no long-term storage (use Thanos/Cortex for S3 remote storage), degrades with high cardinality (millions of unique series). InfluxDB: push-based, high write throughput (millions of points/sec). Built-in retention policies and downsampling. Best for: IoT sensor data, application metrics, high-volume ingestion. InfluxDB 3.0 uses Apache Arrow/Parquet for columnar storage. TimescaleDB: PostgreSQL extension. Full SQL, JOINs with relational data, PostgreSQL ecosystem. Automatic time-partitioning. Best for: teams wanting SQL compatibility, mixed workloads (time-series + relational in one DB), existing PostgreSQL deployments. 10-100x faster than vanilla PostgreSQL for time-series queries. Choose based on: existing stack (PostgreSQL shop -> TimescaleDB, Kubernetes -> Prometheus), query language preference (SQL -> TimescaleDB, PromQL -> Prometheus), and write volume (very high -> InfluxDB).”}},{“@type”:”Question”,”name”:”What is downsampling and why is it essential for time-series data?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Downsampling aggregates high-resolution data into lower-resolution summaries. Raw 10-second data: 8,640 points per day per metric. With 1 million metrics: 8.64 billion points per day. Keeping this for years is impractical. Downsampling strategy: keep 10-second resolution for 24 hours (operational debugging), 1-minute averages for 7 days (recent trend analysis), 1-hour averages for 90 days (capacity planning), 1-day averages for 2 years (long-term trends). Each level stores min, max, mean, and count to preserve the data characteristics. Storage reduction: 100-1000x compared to keeping full resolution indefinitely. Implementation: InfluxDB continuous queries, TimescaleDB continuous aggregates, and Prometheus recording rules automatically maintain downsampled views as new data arrives. Retention policies delete raw data after its retention period. The combination ensures you can answer both is the server healthy right now? (full resolution, last hour) and what was the growth trend last year? (daily aggregates).”}},{“@type”:”Question”,”name”:”How does time-series compression achieve 1-2 bytes per data point?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Two key compression techniques: (1) Timestamp compression with delta-of-delta encoding. Timestamps arrive at regular intervals (e.g., every 10 seconds). The delta between consecutive timestamps is 10. The delta-of-delta (difference of differences) is 0 for regular data. Zero compresses to 1 bit. Irregular intervals produce small deltas that compress to a few bits. (2) Value compression with gorilla encoding (from the Facebook Gorilla paper). XOR consecutive values. If the metric barely changes (CPU = 45.2, 45.3, 45.1), the XOR produces values with many leading and trailing zeros. Encode only the meaningful bits in the middle. For stable metrics, this compresses to 1-2 bits per value. Combined: timestamp (1-2 bits) + value (1-10 bits) + overhead = approximately 1.3 bytes per data point in Prometheus (compared to 16 bytes uncompressed: 8-byte timestamp + 8-byte float). This 10-12x compression makes it feasible to store billions of data points in memory for fast queries.”}}]}