RocksDB

Part I: The mental model

write:  Put(k,v) ─▶ WriteBatch ─▶ WAL append ─▶ memtable (skiplist, sorted)
                                                   │ fills (64 MB default)
                                                   ▼
                                     flush ─▶ L0 SST file (sorted, immutable)
                                                   │ L0 piles up (4 files)
                                                   ▼
                              compaction ─▶ L1 ─▶ L2 ─▶ ... ─▶ L6 (each ~10× larger)

read:   Get(k) ─▶ memtable ─▶ immutable memtables ─▶ L0 files (newest first)
                     ─▶ L1..Ln (one binary-searched file per level)
                          each file: bloom filter ─▶ index block ─▶ data block
                                          (block cache in front of the disk)

The one-sentence identity: RocksDB is a log-structured merge tree in a library: writes are absorbed by a sorted in-memory buffer and a sequential log, disk data lives only in sorted immutable files, and a background merge process perpetually renegotiates the price of reads, writes, and space. There is no server, no port, no SQL; you link the library into your process and it manages a directory of files. Everything in the repository serves one of the three verbs in the diagram: absorb writes (WAL plus memtable), reorganize data (flush plus compaction), or answer reads (filters, indexes, caches, merging iterators).

The deep reason for the shape is the difference between random and sequential I/O. A B-tree updates pages in place, paying a random write per update but keeping one copy of each key. An LSM never updates anything in place: every write is an append or a new sorted file, which turns the write path into pure sequential I/O that both flash and disks love, and pushes the deferred cost into background merges and into reads that must consult several places. RocksDB is what happens when that idea is engineered for a decade against production workloads at Meta: the trade-offs become explicit options, and the bookkeeping (which files exist, at which level, holding which key ranges) becomes its own small database, the MANIFEST.

Part II: Using it

Installing on Linux and macOS

RocksDB is a C++ library. Prebuilt packages exist (librocksdb-dev on Debian and Ubuntu, the rocksdb formula on Homebrew), and building from source is the canonical route and the one you want for reading anyway:

# Debian/Ubuntu build prerequisites
sudo apt install build-essential libgflags-dev libsnappy-dev \
    zlib1g-dev libbz2-dev liblz4-dev libzstd-dev

git clone https://github.com/facebook/rocksdb.git
cd rocksdb
make static_lib -j$(nproc)        # librocksdb.a
make db_bench sst_dump ldb -j$(nproc)   # the tools this chapter uses
cd examples && make simple_example

On macOS, brew install rocksdb gives you the library and tools (the benchmark binary may be installed as rocksdb_bench and similar prefixed names). The debug default of the Makefile is slow by design; when you benchmark, build with DEBUG_LEVEL=0.

A first real program

The natural language is C++, and the minimal program is short enough to read in one breath (this is a lightly trimmed examples/simple_example.cc):

#include "rocksdb/db.h"
#include "rocksdb/options.h"

rocksdb::Options options;
options.create_if_missing = true;
options.IncreaseParallelism();
options.OptimizeLevelStyleCompaction();

rocksdb::DB* db;
rocksdb::Status s = rocksdb::DB::Open(options, "/tmp/testdb", &db);
assert(s.ok());

s = db->Put(rocksdb::WriteOptions(), "key1", "value");
std::string value;
s = db->Get(rocksdb::ReadOptions(), "key1", &value);
assert(s.ok() && value == "value");

Iterators give ordered range scans, which is the capability that separates an LSM store from a plain hash store:

rocksdb::Iterator* it = db->NewIterator(rocksdb::ReadOptions());
for (it->Seek("user:"); it->Valid() && it->key().starts_with("user:");
     it->Next()) {
  // keys arrive in sorted order across memtables and all SST files
}
delete it;

Mistake one: ignoring Status

Nothing throws. Every call returns a rocksdb::Status, and a Get for a missing key is not an error you can skip:

// wrong: value is garbage-or-stale on any failure, and a missing
// key is silently conflated with success
db->Get(rocksdb::ReadOptions(), key, &value);
use(value);

// right
auto s = db->Get(rocksdb::ReadOptions(), key, &value);
if (s.IsNotFound()) { /* distinct, normal case */ }
else if (!s.ok())   { /* corruption, I/O error: handle or die loudly */ }
else                { use(value); }

Mistake two: unbatched multi-key writes

Related updates written as separate Puts are neither atomic nor fast; a WriteBatch is both, applied as one WAL record and one memtable pass:

// wrong: a crash between the two Puts leaves the pair inconsistent
db->Put(wopts, "order:42", order_bytes);
db->Put(wopts, "order_index:2026-07:42", "");

// right: atomic, and cheaper than two round trips through the write path
rocksdb::WriteBatch batch;
batch.Put("order:42", order_bytes);
batch.Put("order_index:2026-07:42", "");
db->Write(wopts, &batch);

Mistake three: opening the same directory twice

RocksDB is an embedded, single-process store. A LOCK file makes the second DB::Open on the same directory fail rather than corrupt, and the fix is architectural, not a flag: one process owns the DB, others talk to that process (or use the read-only and secondary open modes for followers).

Mistake four: reading your own writes with stale iterators

An iterator (and a snapshot) sees the database as of its creation. Beginners create one iterator at startup and wonder why new writes never appear; iterators are cheap views meant to be created, used, and deleted, and long-lived ones also pin old files on disk, inflating space.

Mistake five: benchmarking the debug build with tiny values

The default Makefile target carries assertions and no optimization, and single-record loops measure your syscall overhead rather than the engine. Use DEBUG_LEVEL=0, realistic value sizes, and db_bench (Lab 1) before drawing any conclusion about throughput.

Part III: When it is the right tool

RocksDB is the right tool when you are building a system that needs a fast, persistent, ordered key-value core inside its own process: a database engine, a stream processor's state store, a message broker's index, a blockchain node, an embedded cache that must survive restarts. Its sweet spots are write-heavy workloads (the LSM absorbs bursts sequentially), datasets larger than memory on flash, and any design that needs prefix scans over sorted keys. It scales down badly in one specific sense: it is not a shared database, and it brings real operational surface (compaction tuning, stalls, file descriptors) that a simple app may not want.

The alternatives frame the boundaries. SQLite is the other great embedded engine: B-tree storage, real SQL, superb for read-mostly relational data, but its page-oriented write path and single-writer model make it the wrong core for a write-hammered state store. LMDB is a memory-mapped copy-on-write B-tree, outstanding for read-dominated workloads with modest write rates. LevelDB is RocksDB's ancestor and still fine for small embedded uses, but it lacks column families, real concurrency, and a decade of tuning. And when multiple services need the data, you want a served database; my databases note maps that decision space, and my key-value store design write-up derives this same LSM architecture from requirements instead of from code.

The architecture-shaped warning: RocksDB is a library with exactly one writing process; do not architect as if it were a shared network service, and do not put its directory on NFS or any shared filesystem. The LOCK file, the WAL, and the MANIFEST all assume local-filesystem semantics and a single owner.

safe:      service A ──RPC──▶ storage service ──▶ RocksDB on local SSD
           service B ──RPC──▶      (one process owns the directory)

dangerous: service A ──▶ /shared-nfs/db ◀── service B
           two writers, network filesystem locks: corruption waiting to happen

Part IV: The full life of one operation

This is the core of the chapter. We follow Put("k1", "v1") all the way to a compacted SST file, then follow Get("k1") back up. Every stage names the files that implement it, so this section doubles as a map of db/.

Stage 1: Put becomes a WriteBatch

DB::Put is a convenience: it wraps the single key-value pair in a WriteBatch, the unit the write path actually processes. A batch is a small serialized buffer of operations (Put, Delete, Merge, and friends) with a count and a sequence-number slot in its header, which is why a batch is atomic for free: it is one blob that either lands in the WAL or does not.

Stage 2: the write group forms

The batch enters DBImpl::WriteImpl in db/db_impl/db_impl_write.cc, and the first thing that happens is social. Concurrent writers register with WriteThread (db/write_thread.cc), which links them into a queue; the front writer becomes the leader, absorbs the batches of the writers behind it into one group, and does the WAL work for all of them while the others wait. This is group commit, the same trick every serious storage engine plays: many logical writes, one physical log write, one fsync. Each operation in the merged group receives a monotonically increasing sequence number, the timestamp of the whole MVCC story: snapshots are just sequence numbers, and a newer version of a key is simply the entry with the higher sequence.

Stage 3: the WAL append

The leader appends the group's batches to the current WAL, a .log file written by db/log_writer.cc. The format is records packed into fixed 32 KB blocks, each record framed by a checksum, size, and type, where the type marks whether a record is full or a first/middle/last fragment spanning blocks. Durability is a dial: WriteOptions::sync forces an fsync before acknowledging; the default leaves the data in OS buffers, so a machine crash (not just a process crash) can lose recent writes, and disableWAL removes even process-crash safety for callers who have an external log of their own.

Stage 4: the memtable insert

Only after the WAL append does the group insert into the memtable (db/memtable.cc), by default an InlineSkipList from memtable/: a lock-free-for-readers skiplist keyed by the internal key, which is user key plus inverted sequence number plus type. That composite key is defined in db/dbformat.h, and it is the detail that makes everything else work: entries for the same user key sort newest-first, so a lookup finds the latest version first, and a Delete is just an entry of type tombstone. Nothing is ever modified in place; a write is always a new versioned entry, and reconciliation is deferred to reads and compaction. The Put is now done and acknowledged, typically in microseconds: one queue hop, one log append, one skiplist insert.

Stage 5: flush freezes and writes L0

When the memtable reaches write_buffer_size (64 MB default), it is marked immutable, a fresh memtable takes over, and a background flush job (db/flush_job.cc) walks the frozen skiplist in sorted order and hands the stream to BlockBasedTableBuilder (table/block_based/block_based_table_builder.cc), which emits an SST file into level 0. Once the file is durable, the WAL segments covering that memtable become garbage and are deleted; this is the moment the log has served its purpose. L0 is special: because each L0 file is a frozen memtable, L0 files overlap each other's key ranges, and every one of them must be checked by reads. That is a debt the next stage exists to pay down.

Stage 6: compaction carries it downward

When L0 accumulates level0_file_num_compaction_trigger files (4 by default), the compaction picker (db/compaction/compaction_picker_level.cc for the default leveled style) selects the L0 files plus the overlapping files in L1, and a CompactionJob (db/compaction/compaction_job.cc) merge-sorts them, keeping only the newest version of each key, dropping tombstones when nothing older can exist below, and writes fresh files into L1. Each level except L0 is one sorted run, non-overlapping files of about target_file_size_base (64 MB), and each level's total budget grows by max_bytes_for_level_multiplier (10×) from max_bytes_for_level_base (256 MB), out to num_levels (7). When a level exceeds its budget, files are compacted into the next one down. Our "k1" will migrate down over time until it settles in the bottom populated level or is superseded by a newer write. Every file change is committed as a VersionEdit logged to the MANIFEST, which the third deep dive covers.

Stage 7: Get, from the top

Get("k1") enters DBImpl::GetImpl, grabs the current superversion (a consistent bundle of memtable, immutable memtables, and SST file list), and checks in newest-first order: the active memtable, then immutable memtables awaiting flush. A hit at any point wins immediately, including a tombstone hit, which returns NotFound. Only on a miss does the read go to disk via Version::Get in db/version_set.cc: every L0 file newest to oldest, then for each deeper level a binary search over file metadata locates the single file whose range could contain the key. This ordering is why read amplification in an LSM is "number of sorted runs you might have to consult", and why L0 pileup is the classic read-latency incident.

Stage 8: inside one SST file

For each candidate file, the table cache (db/table_cache.cc) provides an open BlockBasedTableReader. If the file has a bloom filter, the filter block is consulted first and usually answers "definitely not present" for the files that do not hold the key, saving their data-block reads entirely. On a maybe, the index block maps the key to a data block, the block is fetched (through the block cache if resident, from disk if not, decompressing on the way), and a scan inside the block finds the entry. The value returns up the stack; a Get that hits the block cache never touches storage at all. Round trip complete: a write that cost a log append and a skiplist insert, a read that cost a memtable probe plus at most a few filtered block reads.

Part V: Internals deep dives

Deep dive: the LSM tree and the three amplifications

Every LSM design lives inside a triangle of costs. Write amplification: total bytes written to disk per byte of user write, incurred as data is rewritten on its way down the levels. Read amplification: how many places a read might consult. Space amplification: bytes on disk per byte of live data, from not-yet-merged old versions and tombstones. You cannot minimize all three at once, and choosing a compaction style is choosing which one you agree to pay.

              write amp
                 ▲
                / \      leveled:    low read amp, low space amp, HIGH write amp
               /   \     universal:  low write amp, higher space + read amp
              /     \    FIFO:       almost no compaction, drops old data
             ▼       ▼
        read amp ◀─▶ space amp

Leveled compaction (the default) keeps each level a single sorted run with exponentially growing budgets. Reads are cheap: one run per level plus L0, so a handful of candidate files. Space is tight: stale versions survive only until the next merge, and the 10× fanout means roughly 90% of data sits in the bottom level, bounding space overhead. The bill is write amplification: a key rewritten at every level boundary, with a per-level cost on the order of the fanout, can cost tens of physical writes per logical write across a deep tree. Universal (tiered) compaction inverts the deal: it accumulates sorted runs of similar size and merges them together only when there are too many, rewriting data far less often, at the price of more runs to consult on reads and transiently holding duplicate data (a full merge can briefly need on the order of twice the space). Ingest-heavy, scan-light workloads take that trade happily. FIFO compaction barely compacts at all: it drops the oldest files past a size budget, which turns RocksDB into a persistent TTL-ish cache for time-ordered data.

The famous trap in this area is flow control. Compaction is not optional housekeeping; it is the engine's debt service, and RocksDB enforces it: at 20 L0 files (default) writes are slowed, at 36 they stop entirely until compaction catches up. A benchmark that writes flat out for a minute measures the memtable; production throughput is compaction throughput. The second trap: deletes are writes. A Delete adds a tombstone that must itself flow down the tree before the space comes back, and a mass-delete followed by a scan is a worst-case workload, because iterators must step over the tombstones (the wiki's answer is DeleteRange and periodic compaction).

Deep dive: anatomy of an SST file

┌──────────────────────────┐
│ data block │ ~4 KB, sorted keys, prefix-compressed, restart points
│ data block │
│    ...     │
│ filter block   │ bloom bits for every key in the file (if configured)
│ properties     │ counts, sizes, options used, user-defined stats
│ index block    │ last-key-of-each-data-block → block offset
│ footer         │ fixed size: offsets of index+meta, magic number
└──────────────────────────┘  read a file back-to-front: footer first

The block-based table is the default and only format you need to know. Data blocks hold about block_size (4 KB uncompressed by default) of sorted entries; within a block, keys are prefix-compressed, with periodic restart points where a full key is stored so binary search can land and scan forward. The index block is a sorted map from each data block's last key to its offset, so point lookups binary-search the index, then one data block. The properties block records everything about how the file was made, which is why sst_dump (Lab 4) is such a good teacher. A reader starts from the fixed-size footer at the end, which locates the metaindex and index and carries the magic number.

Bloom filters deserve their own paragraph because the default surprises people: filter_policy defaults to null, so a stock RocksDB has no bloom filters until you configure them. The conventional setting, NewBloomFilterPolicy(10), spends 10 bits per key for roughly a 1% false-positive rate, converting most "check this file" operations into a memory test instead of a disk read; point-lookup-heavy workloads should consider it mandatory. The related knobs are about where this metadata lives: by default index and filter blocks are held by open table readers, and cache_index_and_filter_blocks moves them into the block cache (32 MB by default if you do not supply one, far too small for serious use) so their memory is bounded and observable. Compression is per-level in spirit: cheap-or-none at upper levels that get rewritten often, strong (zstd) at the bottom level where 90% of the data rests.

Deep dive: the MANIFEST and version sets

An LSM database is, at any instant, "these exact SST files at these exact levels." That statement is itself state, it changes with every flush and compaction, and it must survive crashes. RocksDB stores it the same way it stores everything: in a log. The MANIFEST file is an append-only log of VersionEdit records (file added at level N, file deleted, WAL number advanced), and a Version (db/version_set.cc, version_edit.cc) is the in-memory result of folding those edits: an immutable list of the live files per level for one column family. The VersionSet owns the chain of versions; the CURRENT file names the active MANIFEST; recovery is "read CURRENT, replay the MANIFEST, then replay WALs newer than the last recorded flush."

CURRENT ─▶ MANIFEST-000123: {edit: +file 7 @L0} {edit: +8 @L0} {edit: -7 -8, +9 @L1} ...
                                  fold ▶ Version N: L0=[]  L1=[9]  L2=[3,4]
reads pin a Version ──▶ files of that Version cannot be deleted while pinned

Versions are immutable and reference-counted, which is how reads, iterators, and compactions all proceed without locking each other: each pins the Version it started with, and obsolete files are physically deleted only when no Version references them. Two practical consequences double as the traps of this subsystem. First, a long-lived iterator or snapshot pins old versions, so disk usage can stay high long after deletes, and the fix is to keep read views short-lived. Second, never "clean up" SST files by hand; a file that looks old may be referenced by the MANIFEST, and deleting it corrupts the database as surely as deleting a B-tree page. The LOG file (plain text, despite the name) narrates all of this machinery live, which is Lab 3.

Deep dive: the tuning surface, at concept level

The options struct has hundreds of fields, but the load-bearing ones map one-to-one onto the pipeline, and the wiki's own tuning guide says most users should touch only a few. On the write side, write_buffer_size (64 MB) and max_write_buffer_number (2) set how much RAM absorbs writes before flush pressure; bigger buffers mean fewer, larger flushes and better dedup of hot keys before disk. The L0 triggers (4 compact, 20 slow down, 36 stop) are the flow control connecting ingest rate to compaction throughput, and max_background_jobs (2) is how much parallelism compaction gets to keep up. On the shape side, target_file_size_base (64 MB) and max_bytes_for_level_base (256 MB) with its 10× multiplier set the granularity and depth of the tree. On the read side, bloom bits per key and the block cache size are the two levers that matter, and per-level compression choices trade CPU for space. The right method is not memorizing numbers: identify which amplification hurts your workload, then move the one or two knobs that trade away something you have spare. Column families exist so that differently-shaped data inside one process can make different choices while sharing one WAL and atomic batches.

Part VI: Reading the repository

Stage 0, the contract. Read include/rocksdb/db.h, options.h, advanced_options.h, and table.h; the option comments are some of the best documentation in the project. Then examples/simple_example.cc and its siblings. Questions: what is a column family? What does a Status-based API imply about error handling? Which options in this chapter can you now find with their documented defaults?

Stage 1, the write path. db/db_impl/db_impl_write.cc (WriteImpl), db/write_thread.cc, db/log_writer.cc, db/memtable.cc, and the internal key in db/dbformat.h. Questions: how does group commit pick a leader? Why does the WAL append happen before the memtable insert? Why do internal keys embed an inverted sequence number?

Stage 2, flush and the file format. db/flush_job.cc, then table/block_based/block_based_table_builder.cc and its reader counterpart. Questions: what makes L0 special? In what order are the blocks of an SST written, and why is the footer last? Where would a bloom filter be consulted?

Stage 3, metadata. db/version_edit.h and db/version_set.cc (start from the class comments), plus db/column_family.cc. Questions: what exactly is in a VersionEdit? How does recovery rebuild the file list? What keeps an SST file alive on disk?

Stage 4, compaction. db/compaction/compaction_picker.cc, the leveled and universal pickers beside it, and compaction_job.cc. Questions: what makes a level "need" compaction under the leveled picker? When can a tombstone finally be dropped? Where do write stalls come from?

Stage 5, the extras. cache/ for the block cache, tools/ for db_bench_tool.cc, sst_dump.cc, and ldb_cmd.cc, and utilities/ for transactions, backup, and TTL, which show how far the core primitives stretch.

Where not to start: db/db_impl/db_impl.cc read top to bottom (it is the hub of everything and reads as noise until you know the spokes), the transaction layer, and BlobDB; and do not begin with the options code, which is machinery, not ideas.

Part VII: Hands-on labs

Lab 1: db_bench basics

./db_bench --benchmarks=fillseq,readrandom --num=1000000 \
           --db=/tmp/rocks_lab --value_size=100

Expect a report per benchmark with micros/op and ops/sec (numbers vary enormously by hardware and build flags; a release build on SSD will show fillseq far faster than readrandom). Concept taught: sequential absorption versus multi-place reads, the LSM's fundamental asymmetry.

Lab 2: watch SST files appear on flush

./db_bench --benchmarks=fillrandom --num=2000000 \
           --db=/tmp/rocks_lab2 --write_buffer_size=1048576 &
watch -n1 'ls -l /tmp/rocks_lab2/*.sst | tail; echo; ls /tmp/rocks_lab2 | wc -l'

With a deliberately tiny 1 MB write buffer, memtables fill fast and .sst files appear every few seconds, then periodically consolidate as compaction merges them. Concept taught: flush creates files, compaction consumes them; the directory is the LSM tree made visible.

Lab 3: read the LOG file

grep -E "flush|compact" /tmp/rocks_lab2/LOG | head -30
grep "Compaction start" /tmp/rocks_lab2/LOG | head
grep -A12 "Compaction Stats" /tmp/rocks_lab2/LOG | tail -20

The LOG narrates every flush and compaction with byte counts, and the periodic Compaction Stats table shows per-level file counts, sizes, and the read/write bytes that compose write amplification. Concept taught: the amplification triangle, measured on your own machine.

Lab 4: dissect a real SST file

SST=$(ls /tmp/rocks_lab2/*.sst | head -1)
./sst_dump --file=$SST --show_properties --command=none
./sst_dump --file=$SST --command=scan --read_num=5

The properties dump shows entry counts, data and index block sizes, compression, and whether a filter is present (with stock options: no filter, confirming the null default); the scan prints real internal entries with sequence numbers and types. Concept taught: SST anatomy from the deep dive, on disk rather than in a diagram.

Lab 5: bloom filters change the read path

./db_bench --benchmarks=fillrandom,readrandom --num=2000000 \
           --db=/tmp/rocks_bloom --bloom_bits=10 --cache_size=8388608
./db_bench --benchmarks=fillrandom,readrandom --num=2000000 \
           --db=/tmp/rocks_nobloom --bloom_bits=-1 --cache_size=8388608

Compare readrandom micros/op with and without 10-bit bloom filters; the gap grows with dataset size relative to cache. The exact ratio varies, so treat the direction, not the number, as the result. Concept taught: filters convert disk probes into memory tests.

Lab 6: dump the MANIFEST

./ldb --db=/tmp/rocks_lab2 manifest_dump | head -40

You will see the folded state: each level's files with their numbers, sizes, and smallest/largest keys, exactly the Version the engine holds in memory. Delete nothing by hand; instead, re-run Lab 2 briefly and dump again to watch the file lists shift. Concept taught: the MANIFEST as the database's database.

Part VIII: Understanding checks

What is RocksDB in one sentence? An embeddable LSM-tree key-value library: writes go to a WAL and a sorted memtable, disk holds only sorted immutable SST files organized into levels, and background compaction continuously trades write work for read and space efficiency.

Why are LSM writes fast? Because the foreground write path is one sequential log append plus one in-memory skiplist insert, independent of how much data exists on disk. All reorganization is deferred to background flush and compaction.

Walk the path of a Put. Put wraps into a WriteBatch; WriteThread forms a group and assigns sequence numbers; the leader appends the group to the WAL; the batch inserts into the memtable keyed by user key plus inverted sequence; on memtable fill, flush writes an L0 SST and frees the WAL segment; compaction later merges it down the levels, with each file change logged to the MANIFEST.

Walk the path of a Get. Check the active memtable, then immutable memtables, then all L0 files newest-first, then one binary-searched file per deeper level; within each candidate file, bloom filter first, then index block, then one data block via the block cache. The first hit wins, and a tombstone hit means NotFound.

What are the three amplifications, and why can't you have all three? Write amp (disk bytes per user byte written), read amp (places consulted per read), space amp (disk bytes per live byte). They trade against each other because keeping data more merged (good for reads and space) requires rewriting it more often (bad for writes); leveled favors reads and space, universal favors writes.

Why is L0 special? Each L0 file is a flushed memtable, so L0 files overlap in key range and every one must be consulted on a read, unlike deeper levels where non-overlapping files allow a single binary-searched candidate. That is why L0 count drives both compaction triggering (4) and write stalls (20 slowdown, 36 stop).

What is in an SST file? Prefix-compressed ~4 KB data blocks with restart points, an index block mapping each block's last key to its offset, an optional bloom filter block, a properties block describing how the file was built, and a fixed footer, read first, that locates everything.

Do you get bloom filters by default? No; filter_policy defaults to null. NewBloomFilterPolicy(10) is the conventional choice, about 10 bits per key for roughly 1% false positives, and point-read-heavy workloads should treat it as mandatory.

What is the MANIFEST and why does it exist? An append-only log of VersionEdits recording every SST file addition and deletion per level. The live file list is itself mutable crash-critical state, so RocksDB stores it the LSM way: as a replayable log named by the CURRENT file, folded into immutable, reference-counted in-memory Versions.

What is a tombstone and why do deletes cost space before they free it? A Delete writes a marker entry that shadows older versions; the actual space returns only when compaction merges the tombstone far enough down that no older version can exist beneath it. Until then, deletes add bytes, and scans must skip over them.

Why did writes suddenly stall? Ingest outran compaction: L0 reached the slowdown (20) or stop (36) trigger, or pending-compaction bytes tripped their limits. Fixes are throttling ingest, giving compaction more parallelism (max_background_jobs), larger write buffers, or accepting universal compaction's trade.

How does a crash recover? Read CURRENT to find the MANIFEST, replay its edits to rebuild the file lists, then replay WAL records newer than the last flushed sequence into a fresh memtable. Both replays are logs; recovery is the same idea applied twice.

Why choose RocksDB over SQLite, or the reverse? RocksDB when you need a write-heavy, ordered key-value core inside a system you are building and are willing to bring your own data model; SQLite when you want relational queries, ad-hoc reads, and a stable single file with a B-tree's read-mostly economics. They are both embedded, and they sit on opposite corners of the write-versus-query-power trade.

A long-running iterator is held open for hours. What goes wrong? It pins a superversion and therefore old SST files and memtables, so disk space and memory stay elevated regardless of deletes and compaction, and the view grows stale. Read views should be short-lived; snapshots are cheap to take and retake.

Why does compaction happen in the background at all, rather than merging on write like a B-tree? Merging on write would put random I/O and rewrite cost on the foreground path, which is exactly what the LSM exists to avoid. Deferring merges keeps write latency flat and lets the engine batch reorganization into large sequential operations, at the price of scheduling debt that the stall mechanism enforces.

Part IX: Design lessons

Turn mutation into accumulation plus merge. The LSM's core move, never update in place, always append and reconcile later, reappears in Git, in CRDTs, in event-sourced services, and in columnar warehouses' delta-plus-base files. It buys sequential I/O and trivial crash stories, and costs a background debt service.

Make the trade-off a first-class option. RocksDB does not pick a winner among the three amplifications; it exposes compaction styles and per-column-family options so each embedder picks. Engines that hardcode one point on the curve (LevelDB) get outgrown by their users.

Metadata deserves the same rigor as data. The MANIFEST is a WAL for the file list, with the same append-replay-checkpoint lifecycle as user data. Any system whose "what exists right now" state must survive crashes ends up reinventing this (Kubernetes' etcd, Iceberg's metadata layers), so it pays to recognize the pattern.

Group commit is free throughput. Letting one leader do the physical log write for a queue of waiting writers multiplies throughput at essentially no latency cost, and the same structure appears in Postgres's WAL, in Kafka's producer batching, and in every syscall-batching layer.

Backpressure must be designed, not discovered. The L0 slowdown and stop triggers are an explicit admission that background debt needs foreground flow control. Systems that lack a designed stall mechanism still stall; they just do it unpredictably, at the OOM killer or the full disk.

Part X: Memorization framework

The one-sentence summary: RocksDB absorbs writes into a log and a sorted memory buffer, spills them as immutable sorted files, merges those files downward forever, and answers reads by checking newest-to-oldest with filters and caches shielding the disk.

write: Put → Batch → Group → WAL → MemTable → Flush → L0 → Compact → L1..L6
read:  Get → MemTable → Imm → L0* → L1..L6 (bloom → index → block cache → disk)

db_impl_write.cc → write_thread.cc → log_writer.cc → memtable.cc
→ flush_job.cc → block_based_table_builder.cc
→ compaction_picker*.cc → compaction_job.cc → version_set.cc (MANIFEST)

Memorize these blocks:

The defaults: write buffer 64 MB, two of them; L0 triggers 4 / 20 / 36 (compact / slow / stop); level base 256 MB growing 10× across 7 levels; target file size 64 MB; block size 4 KB; block cache 32 MB if unset; bloom filters absent until configured, 10 bits per key ≈ 1% false positives.

The invariants: nothing on disk is ever modified, only written and deleted; every entry is (user key, sequence, type); L0 overlaps, deeper levels do not; a Version is immutable and pins its files; recovery is MANIFEST replay plus WAL replay.

The costs: leveled pays write amp for read and space; universal pays space and read for write; deletes are writes until compacted away; stalls are compaction debt made visible.