A time-series database stores streams of timestamped measurements. Each stream, called a series, is identified by a metric name plus a set of key-value labels, and each sample is a 64-bit timestamp paired with a 64-bit float. The workload it targets is extreme in one specific way. Writes arrive constantly and almost always for the current moment, old data is never updated, reads scan a time window over one or a few series and aggregate it, and deletes happen only by age. Every internal mechanism in a TSDB exists to exploit that shape.
The scale that forces a specialized engine shows up fast. A monitoring system for 100,000 servers emitting five metrics every ten seconds ingests 50,000 samples per second, about 4.3 billion data points per day. In a plain Postgres table each point costs 50 to 100 bytes once tuple headers and repeated host and metric strings are counted, even though the real information is a 16-byte pair, and insert throughput sags as every write dirties index pages. Facebook hit this wall in early 2013, when the HBase store behind its ODS monitoring system held about 2 petabytes and 90th percentile chart queries stretched to multiple seconds.
The mental model that makes the internals legible is an append-only log wrapped in a purpose-built compression codec, with an inverted index bolted on the side so labels stay searchable. Prometheus, InfluxDB, and Uber's M3 are all variations on that one design, and the compression scheme they share descends directly from Facebook's Gorilla paper. This note walks the design from the data model down to the bit-level encodings, then covers where it breaks and what the production numbers look like.
Data model and how to use it
The unit of storage is the series, one per unique combination of metric name and label values. In InfluxDB's vocabulary the metric is a measurement, the labels are tags, and the measured numbers are fields. The distinction carries real weight because tags are indexed and fields are not. A tag like host=web-42 can be filtered cheaply through the index, while a field like value=45.2 is just payload inside a compressed block. Putting a high-cardinality identifier in a tag creates a new series per value, which is the single most common way people break these systems.
Data gets in either by push or by pull. InfluxDB accepts line protocol over HTTP, and its docs recommend batching 5,000 to 10,000 points per write for throughput. Prometheus inverts the flow and scrapes HTTP endpoints on an interval, which means the database controls its own ingest rate. Queries run in a purpose-built DSL rather than SQL because the common operations, rate over a counter, quantile from a histogram, aggregation across a label dimension, are awkward to express relationally. Retention is a first-class config knob rather than a DELETE statement, with Prometheus defaulting to 15 days.
# InfluxDB line protocol: measurement,tags fields timestamp
cpu_usage,host=web-42,region=us-west,env=prod value=45.2 1700000000000000000
# PromQL: per-service p99 latency from histogram buckets, 5 minute windows
histogram_quantile(0.99,
sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))
# PromQL: five busiest hosts by non-idle CPU over the last 5 minutes
topk(5, avg by (host) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))
# Retention is configuration, not a DELETE query (default is 15d)
prometheus --storage.tsdb.retention.time=30dWhy B-trees fail at append-heavy metric loads
A B-tree keeps every row in globally sorted pages and updates those pages in place. Each insert walks from root to leaf, dirties that leaf, and occasionally splits it, and every secondary index repeats the cost on its own leaves. Metrics need an index on something like (host, metric_name, time) to be queryable, so a stream of samples from thousands of hosts lands on thousands of different leaf pages, which is random I/O. A spinning disk manages only 100 to 200 random operations per second, and while SSDs are far better at random access they still favor sequential patterns and suffer device-level write amplification when small random writes force full page rewrites. Row overhead compounds the problem, since a Postgres heap tuple carries a 23-byte header plus alignment padding before the first byte of your 16 bytes of actual data.
The read side is just as bad. Samples for one series arrive interleaved with samples from every other series, so they end up scattered across heap pages in arrival order. A query for one host's CPU over an hour fetches thousands of pages that each contain a few relevant rows. The data you want is never physically together.
The fix is to stop updating pages in place. Log-structured storage appends every write to the end of a file, buffers recent data in a sorted in-memory table, flushes it as an immutable sorted file, and merges files in the background. Writes become purely sequential, which even spinning disks handle at tens of thousands per second, and the background merge is itself sequential and batched. TSDBs adopt this LSM skeleton and then go one step further than a general LSM store like Cassandra. They lay data out per series inside each file, so the flush that makes writes fast also gives reads the physical locality that the B-tree layout destroyed, and the columnar arrangement of one series' timestamps and values back to back is exactly what the compression in the next sections needs.
The write path: head block, WAL, and immutable blocks
Prometheus is the cleanest reference implementation of the modern write path. An incoming sample is first appended as a record to the write-ahead log, stored in 128 MB segments with a minimum of three segments retained, and only then added to the head block, the in-memory portion of the database. Inside the head each series owns one active chunk, and the sample is appended to it using the Gorilla-style encoding described below. When a chunk reaches 120 samples or spans the 2 hour chunk range, it is cut, and since Prometheus v2.19.0 a full chunk is immediately flushed to disk and memory-mapped, so the head keeps only a reference instead of the bytes. After a crash, the WAL is replayed on top of the memory-mapped chunks to rebuild the head exactly.
When the head spans one and a half times the chunk range, 3 hours with the 2 hour default, its oldest 2 hours are compacted into a persistent block, a directory containing chunk segment files of up to 512 MB, an index file, a meta.json, and a tombstones file for deletion marks. Blocks are immutable. A background compactor then merges neighboring 2 hour blocks into larger ones spanning up to 10 percent of the retention window or 31 days, whichever is smaller, deduplicating overlaps and dropping tombstoned data as it goes. Retention enforcement is the payoff of this whole arrangement, because expiring old data means deleting whole block directories rather than scanning a giant table for rows to remove.
InfluxDB's TSM engine is the same architecture with different constants. Writes are Snappy-compressed and fsynced into 10 MB WAL segments before the write is acknowledged, accumulate in an in-memory cache, and snapshot to immutable TSM files with a header, compressed blocks, an index ordered by series key and time, and a footer. Compaction proceeds through snapshot, leveled, index optimization, and full stages, and data is organized into shard groups covering 7 days by default so retention again reduces to dropping whole shards.
Writes append to the WAL and the in-memory head block. Every 2 hours the head flushes an immutable block, the compactor merges blocks up to 10 percent of retention, rollup tiers hold downsampled aggregates, and queries intersect inverted-index postings before touching any chunk data.
Gorilla compression: delta-of-delta timestamps and XOR floats
Facebook's Gorilla paper (VLDB 2015) is the reason a sample costs bytes instead of tens of bytes. By spring 2015 Facebook's monitoring generated more than 2 billion unique series with about 12 million data points added per second, roughly 700 million per minute and over a trillion per day. At 16 bytes per point, a single day of that stream would have needed 16 TB of RAM. The codec they built compresses the average point to 1.37 bytes, a 12 times reduction, which let the launch dataset fit in 1.3 TB spread across 20 machines.
Timestamps use delta-of-delta encoding. A block header stores a start time aligned to a two hour window, the first timestamp is stored as a 14-bit delta (enough for 16,384 seconds), and every later timestamp stores only how much its interval changed from the previous interval. A perfectly regular 10 second scrape produces a delta-of-delta of zero, encoded as a single 0 bit. Nonzero values fall into variable-length buckets, 2 bits of prefix plus 7 bits for the range -63 to 64, up through 4 bits plus 32 for arbitrary jumps. Against a sample of 440,000 production timestamps, about 96 percent compressed to a single bit, because real scrapers are metronomes with occasional one second jitter.
Values use XOR compression. Adjacent samples of a metric are usually close, so XORing a float with its predecessor zeroes the sign, exponent, and top mantissa bits, leaving a short run of meaningful bits between leading and trailing zeros. An identical value XORs to zero and is stored as a single 0 bit, which covered roughly 51 percent of Facebook's values. If the meaningful bits fit inside the previous value's window, a 2-bit control prefix and just those bits are stored, about 30 percent of values at 26.6 bits on average. Otherwise the encoder spends 5 bits on the leading-zero count and 6 bits on the meaningful length before the payload, the remaining 19 percent at 36.9 bits. The paper also shows why blocks span two hours, since measured compression improves as the window grows and flattens at 1.37 bytes per point near the 120 minute mark.
The scheme became the industry default. Prometheus chunks use it, which is where the official capacity-planning figure of 1 to 2 bytes per sample comes from, InfluxDB applies the same XOR algorithm to float blocks alongside per-type codecs for integers, booleans, and strings, and M3 ships M3TSZ, a tuned variant that squeezes float64 values further.
def encode_timestamp(t, t_prev, t_prev2, bits):
d = (t - t_prev) - (t_prev - t_prev2) # delta of deltas
if d == 0:
bits.write('0') # 96% of real samples
elif -63 <= d <= 64:
bits.write('10'); bits.int(d, 7)
elif -255 <= d <= 256:
bits.write('110'); bits.int(d, 9)
elif -2047 <= d <= 2048:
bits.write('1110'); bits.int(d, 12)
else:
bits.write('1111'); bits.int(d, 32)
def encode_value(v, prev, bits):
x = float64_bits(v) ^ float64_bits(prev)
if x == 0:
bits.write('0') # ~51% of values
elif fits_previous_window(x): # same leading/trailing zeros
bits.write('10'); bits.meaningful(x) # ~30%, avg 26.6 bits
else:
bits.write('11')
bits.int(leading_zeros(x), 5) # new window: 5 + 6 bit header
bits.int(meaningful_len(x), 6)
bits.meaningful(x) # ~19%, avg 36.9 bitsThe inverted label index and cardinality explosions
Compressed chunks are only useful if you can find them, and label matchers like region="us-west", env="prod" cannot be answered by scanning billions of series keys. The answer is the same structure that powers full-text search. An inverted index maps each label pair to a sorted postings list of series IDs, and a query intersects the lists for its matchers, so region=us-west giving series 1 and 2 intersected with env=prod giving series 1, 2, and 3 yields exactly series 1 and 2 before a single chunk is read. Sorted postings intersect in linear time with skip-ahead, the same trick search engines use. In Prometheus the head keeps this index in memory and garbage-collects postings for series that no longer exist, and every persistent block carries its own immutable index file, so on-disk index size is bounded per block.
The index is also where the design fails, because every unique combination of label values is its own series. One thousand hosts times 50 metrics is 50,000 series, entirely manageable. Add a user_id label with 10 million values and the potential series count multiplies into the hundreds of billions. Each series costs head memory for its chunk and symbol table entries, a WAL series record, and postings in the index, so an unbounded label does not degrade the database gracefully, it exhausts memory and crashes it. This is why request IDs, user IDs, and other unbounded identifiers belong in fields, logs, or an OLAP store, never in labels.
A quieter version of the same problem is series churn, which Kubernetes made mainstream. Pods live for hours and carry their identity in labels, so every deploy retires one batch of series and creates another. The set of live series stays flat while the set of series ever seen inside the retention window grows without bound, bloating indexes and slowing any query that touches label metadata. The Prometheus 2.0 storage rewrite by Fabian Reinartz was motivated largely by making index handling and block layout survive this churn.
Downsampling and retention tiers
Raw resolution is for debugging the last few hours, not for a one year dashboard. Downsampling rolls raw samples up into fixed buckets that store min, max, sum, and count, which is enough to reconstruct averages and rates without touching raw data. A month-long query served from a 5 minute tier reads 288 points per series per day instead of the 8,640 that a 10 second scrape produces, 30 times less data for the same chart. Tiers pair a resolution with a retention, and M3's published example policy keeps 10 second data for 2 days, 1 minute data for 30 days, and 1 hour data for 5 years.
M3 goes further and aggregates at ingest time rather than in a background job, which is how it accepts 500 million metrics per second while persisting only 20 million per second to storage, a 25 to 1 reduction before anything hits disk. Prometheus itself takes the minimal position, since a single node with a 15 day default retention rarely needs tiers. Recording rules can precompute expensive expressions into new series, and layered systems like Thanos and Cortex add 5 minute and 1 hour downsampled tiers over object storage for long horizons.
Capacity planning falls out of one multiplication, retention seconds times ingested samples per second times bytes per sample. The 50,000 samples per second fleet from the introduction at the documented 1.5 bytes per sample needs about 97 GB for 15 days of raw data, and Prometheus guidance is to cap size-based retention at 80 to 85 percent of allocated disk so compaction has working room.
Trade-offs and when not to use one
Everything above rests on assumptions, and violating them makes a TSDB worse than the general-purpose database you avoided. The engine assumes writes arrive roughly in time order, so late or out-of-order data historically got rejected outright by Prometheus and remains a configuration headache everywhere. It assumes data is immutable, so correcting a bad measurement means tombstones and compaction rather than an UPDATE. It assumes label sets are small and low-cardinality, so per-user or per-request slicing belongs in ClickHouse or a warehouse instead. It assumes queries aggregate within series selected by labels, so joins, ad hoc relational filtering, and transactional reads are out of scope entirely.
Some failure modes are less obvious. Top-K style questions across an enormous number of series, like the most-liked posts on a social network, look like time-series problems but require sorting and aggregating across the very dimension that must stay low-cardinality, so a TSDB actively hurts there. Compaction itself is a cost, since data gets rewritten multiple times on its way to large blocks. Uber's M3 team called out that Cassandra-style compacting stores spend a large fraction of CPU, memory, and disk I/O just rewriting data they already stored, and designed M3DB to mostly avoid compaction as a result.
The practical advice is to stretch a general-purpose database first. Postgres with a BRIN index or TimescaleDB's chunked tables handles tens of thousands of samples per second comfortably, and a TSDB only earns its operational surface once you have a genuine bottleneck of the kind this note describes, sustained six-figure sample rates, multi-week retention, and dashboard-style aggregate reads.
In production: Gorilla, Prometheus, M3, InfluxDB
Facebook's Gorilla ran as an in-memory write-through cache in front of the HBase-backed ODS store, holding the most recent 26 hours because analysis showed at least 85 percent of all queries targeted that window. It was built for 2 billion series, 700 million points per minute, and more than 40,000 queries per second at peak, with reads answered in under a millisecond. Against the HBase path it cut query latency by 73 times and raised query throughput 14 times, and the cluster grew from 20 machines at launch to 80 per cluster after doubling twice, a scaling story made possible by sharding series across hosts by string key.
Uber's M3 is the largest publicly documented deployment of this architecture. As of its 2018 write-up it stored over 6.6 billion time series, aggregated 500 million metrics per second at ingest, and persisted 20 million aggregated metrics per second globally. Uber built M3DB after outgrowing a Cassandra plus Elasticsearch stack on operational burden and cost, replacing generic compaction with time-windowed immutable storage and standard Gorilla compression with its tuned M3TSZ codec.
Prometheus is the same design at single-node scale and is the default metrics store of the Kubernetes ecosystem. Its storage documentation commits to concrete constants, 2 hour blocks compacted up to 10 percent of retention or 31 days, 128 MB WAL segments, 512 MB chunk segment files, 15 day default retention, and an average of 1 to 2 bytes per sample on disk. InfluxDB's TSM engine shows the same skeleton tuned for push ingest, with fsynced 10 MB WAL segments, four-stage compaction, per-type compression, and 7 day shard groups whose expiry implements retention. Four systems, four scales, one architecture.
Follow-up questions
- Why does a B-tree database struggle with metric ingestion? Every insert dirties a random index leaf per secondary index and occasionally splits pages, which is random I/O that spinning disks serve at only 100 to 200 operations per second, and each 16-byte sample balloons to 50 to 100 bytes of row overhead. An append-only log writes sequentially, defers organization to background compaction, and compresses co-located series data by an order of magnitude.
- How does Gorilla fit a 16 byte sample into 1.37 bytes? Timestamps are stored as delta-of-delta, so a regular scrape interval encodes as a single 0 bit, which covers about 96 percent of real timestamps. Values are XORed with their predecessor, so an unchanged float is one bit (about 51 percent of values) and changed floats store only their meaningful bits with a short header. Two hour block windows are used because compression plateaus at 1.37 bytes per point around 120 minutes.
- Trace a sample through Prometheus from scrape to disk. The sample is appended to the WAL (128 MB segments, minimum three retained), then to its series' active chunk in the in-memory head block. At 120 samples the chunk is cut, flushed, and memory-mapped. When the head spans 3 hours, its oldest 2 hours compact into an immutable block directory with chunks, an index, and tombstones, and background compaction later merges blocks up to 10 percent of retention or 31 days.
- What is a cardinality explosion and why is it fatal? Every unique combination of label values is a separate series, so series count is the product of label value counts. Adding a user_id label with 10 million values to a 50,000 series setup creates potential billions of series, and since each series costs head memory, WAL records, and index postings, the database exhausts RAM rather than degrading gracefully. Unbounded identifiers belong in fields or logs, never labels.
- Why is enforcing retention nearly free in a TSDB? Data is partitioned into immutable time-bounded blocks, so expiring old data is deleting whole directories or shards past the cutoff. There is no row scan, no tombstone accumulation, and no index maintenance, unlike a DELETE over a billion-row table.
- When should you not use a time-series database? When you need per-entity analytics over high-cardinality keys, updates or heavily out-of-order writes, joins and ad hoc queries, or Top-K across huge numbers of series. Below sustained six-figure sample rates a general-purpose store like Postgres or TimescaleDB is usually the better operational bet.
References
- Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015), Source of the delta-of-delta and XOR encodings, 1.37 bytes per sample, 96 percent single-bit timestamps, and the 2 billion series and 700 million points per minute scale numbers.
- Prometheus documentation: Storage, Official constants for 2 hour blocks, 128 MB WAL segments, compaction limits, 15 day default retention, and the 1 to 2 bytes per sample capacity rule.
- Ganesh Vernekar, Prometheus TSDB (Part 1): The Head Block, Prometheus maintainer's walkthrough of the head block, 120-sample chunks, memory-mapping since v2.19.0, WAL replay, and head compaction at 3 hours.
- Uber Engineering, M3: Uber's Open Source, Large-scale Metrics Platform for Prometheus, The 6.6 billion series, 500 million metrics per second aggregated, 20 million per second persisted numbers, retention tier examples, and the case against compaction-heavy stores.
- InfluxDB v1 documentation: In-memory indexing and the TSM storage engine, TSM file layout, 10 MB fsynced WAL segments, four-stage compaction, per-type compression codecs, and 7 day shard groups.
- Fabian Reinartz, Writing a Time Series Database from Scratch (archived), Design rationale for the Prometheus 2.0 storage engine, series churn under Kubernetes, and the inverted index. The original fabxc.org host is offline, so this links the archived copy.