Lucene

Part I: The mental model

        WRITE PATH                              READ PATH

  Document + Fields                      TermQuery("body", "search")
        |                                          |
     Analyzer                                IndexSearcher
  (tokenizer + filters)                   (one loop per segment)
        |                                          |
   IndexWriter                          term dictionary lookup
  (in-memory buffers,                  (FST index -> .tip -> .tim)
   one per thread)                                 |
        | flush                          postings iteration
        v                              (.doc blocks of 128 ids)
  Segment (immutable files:                        |
  .tim .tip .doc .pos .fdt ...)          BM25 scoring (norms)
        | background merges                        |
        v                                 top-k collection (heap)
  fewer, larger segments                           |
        | commit                                   v
        v                                       TopDocs
  segments_N file names the live set
        |
        v
  Directory (FSDirectory / MMapDirectory) -> OS page cache -> disk

The one-sentence identity: Lucene is a library that turns documents into immutable segment files containing an inverted index, and turns queries into ranked document ids by walking those files. There is no server, no query REPL, no schema store. You link a jar, you hand it documents, and it writes a directory of oddly named files; later you hand it a query and it hands you back scored integers. Everything Elasticsearch and Solr add, HTTP APIs, JSON documents, sharding, replication, is layered on top of exactly this contract.

The two columns of the diagram never touch a shared mutable structure. The write path accumulates postings in memory and periodically freezes them into a segment, a self-contained mini-index that will never change again. The read path opens a point-in-time view over whichever segments existed at open time and searches each independently. The only coordination between the two sides is the segments_N file, a tiny manifest naming the current segment set, swapped atomically at commit. Hold that picture and most of Lucene's behavior, lock-free reads, near real-time refresh, background merges, deletes as bitmasks, follows by necessity rather than by memorization.

A useful frame from the database world: Lucene is structurally an LSM system, buffered writes flushed to immutable sorted runs that are compacted in the background, except the sorted runs are inverted indexes rather than key-value tables, and the read amplification is paid per segment rather than per level.

Part II: Using it

Install

Lucene is a plain Maven artifact; there is nothing to install system-wide beyond a JDK. The 10.x line requires Java 21 or newer. On Linux and macOS the fastest route to a working toolchain is a JDK from your package manager (sudo apt install openjdk-21-jdk, or brew install openjdk@21) plus the dependency:

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-core</artifactId>
  <version>10.4.0</version>
</dependency>

(10.4.0 is the release I verified this chapter against; anything in the 10.x line behaves as described.) Two sibling artifacts show up almost immediately in real code: lucene-queryparser for parsing query strings and lucene-analysis-common for the language-specific analyzers. For poking at indexes without writing Java, download the binary distribution from lucene.apache.org; it ships every module jar under modules/ and the Luke GUI under bin/.

First session

The front-page example from Lucene's own documentation is worth typing out once, because the whole library is visible in it. Indexing is an analyzer, a directory, and a writer:

Analyzer analyzer = new StandardAnalyzer();
Directory directory = FSDirectory.open(Path.of("idx"));
IndexWriter writer = new IndexWriter(directory, new IndexWriterConfig(analyzer));

Document doc = new Document();
doc.add(new Field("body", "This is the text to be indexed.", TextField.TYPE_STORED));
writer.addDocument(doc);
writer.close();

Searching opens a point-in-time reader over the same directory, parses a query with the same analyzer, and asks for the top hits:

DirectoryReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);
Query query = new QueryParser("body", analyzer).parse("text");
ScoreDoc[] hits = searcher.search(query, 10).scoreDocs;
StoredFields stored = searcher.storedFields();
for (ScoreDoc hit : hits) {
  System.out.println(hit.score + "  " + stored.document(hit.doc).get("body"));
}

Run it and you get one hit with a small positive score. If you want a real index to explore before writing any code, the demo module in the binary distribution indexes a directory of files and searches it interactively:

cd lucene-10.4.0
java -cp "modules/*" org.apache.lucene.demo.IndexFiles -index ./idx -docs ./docs
java -cp "modules/*" org.apache.lucene.demo.SearchFiles -index ./idx

Mistakes beginners make

Analyzed field, unanalyzed expectations. The most common first bug is indexing an identifier with TextField and then wondering why exact matches misbehave, or the reverse, indexing with StringField and wondering why nothing matches a lowercased query. TextField runs the analyzer (tokenized, lowercased); StringField indexes the exact bytes as a single term:

// wrong: "ORD-1234" becomes tokens [ord, 1234]; TermQuery("id", "ORD-1234") finds nothing
doc.add(new TextField("id", "ORD-1234", Field.Store.YES));

// right: one untouched term, matched with TermQuery, never with an analyzed parse
doc.add(new StringField("id", "ORD-1234", Field.Store.YES));

Different analyzers at index and query time. The index only contains what the indexing analyzer produced, so the query side must produce the same tokens. Indexing with EnglishAnalyzer (which stems "running" to "run") and querying through StandardAnalyzer (which does not) silently misses documents. Keep one analyzer instance, or one factory, shared by both sides.

Opening a writer per document. IndexWriter is heavyweight and thread-safe; it is meant to live for the life of the process and absorb addDocument calls from many threads. Opening and closing one per document is both slow and a great way to hit LockObtainFailedException, because each index directory admits exactly one open writer, enforced by write.lock.

Updating by adding. There is no update in place. updateDocument(new Term("id", "ORD-1234"), doc) is the supported idiom, an atomic delete-by-term plus add; calling addDocument for a changed record leaves both versions searchable.

Expecting a new reader to see new writes. A DirectoryReader is a snapshot. Writes after it opened are invisible to it forever; you reopen with DirectoryReader.openIfChanged(reader), or use the near-real-time form DirectoryReader.open(writer), which sees flushed but not-yet-committed segments. This is not a bug, it is the visibility model, and it is exactly what Elasticsearch's refresh interval is made of.

Part III: When it is the right tool

Lucene the library is the right tool when search lives inside your application: an embedded product-catalog search, a desktop mail client, a log viewer, a single-node service where you want millisecond queries over tens of millions of documents without operating a search cluster. It is also the right tool when you are building your own search service and want the hard parts, postings encoding, ranking, segment lifecycle, solved while you own the distribution story yourself. And it is the substrate you study to understand the systems above it.

The alternatives are mostly Lucene wearing operational clothing. Elasticsearch and OpenSearch are the answer when you need sharding, replication, JSON APIs, and a team-shared service; Solr is the same trade with different ergonomics. Outside the family, PostgreSQL full-text search is enough when the data already lives in Postgres and the requirements stop at "find rows containing these words"; SQLite's FTS5 fills the same niche in embedded settings; and Tantivy is the Lucene-shaped library if your process is Rust rather than Java.

The architecture-shaped warning: a Lucene index has exactly one writer, and the index directory is not a shared database. The safe shape is one process that owns the writer, with any number of reader processes or replicas consuming copies of the immutable segment files. The dangerous shape is multiple app instances pointing at one directory on shared storage and all expecting to write, which fails fast on write.lock if you are lucky and corrupts subtly on NFS-style filesystems (broken locking, broken mmap semantics) if you are not.

SAFE                                    DANGEROUS

 indexer process                        app-1   app-2   app-3
   IndexWriter                            \       |       /
       |                                   v      v      v
   local disk index                     one index dir on NFS
       |  ship immutable segments        (three would-be writers,
       v                                  remote locks, remote mmap)
 searcher replicas (read-only copies)

Part IV: The full life of one document and one query

This is the core of the chapter. We index one document into the field body and then serve one TermQuery("body", "search"), naming the classes doing the work at every stage. Everything below lives in lucene/core under org.apache.lucene unless said otherwise.

Stage 1: Document and Field

A Document (package document) is nothing but a list of Field objects, and a field is a name, a value, and an IndexableFieldType that answers three independent questions: is it inverted into the index (and with what detail: docs only, plus frequencies, plus positions), is the original value stored for retrieval, and does it carry doc values for sorting and faceting. TextField, StringField, StoredField, and friends are just presets over those flags. Nothing here touches disk; a Document is a description of intent that is consumed once by the writer and then garbage.

Stage 2: IndexWriter accepts the document

IndexWriter.addDocument (package index) does not index anything itself. It delegates to a DocumentsWriter, which hands the document to a DocumentsWriterPerThread (DWPT), one in-memory indexing buffer per active indexing thread. This is the trick that makes indexing scale across threads with almost no contention: each thread builds its own private mini-index, and threads only synchronize at flush time. A DWPT that fills up is flushed as its own segment, which is why a heavily threaded indexing job produces many small segments.

Stage 3: The analysis chain

Inside the DWPT, the IndexingChain runs each analyzed field through the Analyzer. An analyzer builds a TokenStream: a Tokenizer that splits the input, followed by TokenFilters that transform it. StandardAnalyzer (in core, package analysis.standard) tokenizes on Unicode word boundaries and lowercases, so "This is the text to be indexed." becomes the terms [this, is, the, text, to, be, indexed], each carrying a position. The stream API is deliberately allocation-free: filters share attribute objects (CharTermAttribute, PositionIncrementAttribute) that are mutated in place as the stream advances, one of many places where Lucene's age shows up as ruthless mechanical sympathy.

Stage 4: In-memory inverted index

For every term the chain emits, TermsHashPerField and FreqProxTermsWriter update an in-memory hash keyed by term bytes, appending doc ids, frequencies, and positions into shared byte pools. This is a real inverted index, just a write-optimized one: hash table rather than sorted structure, append-only byte slices rather than compressed blocks. The writer's RAM accounting watches these pools, and when the buffer crosses IndexWriterConfig.setRAMBufferSizeMB (default 16 MB) or an explicit document count, the DWPT is scheduled for flush.

Stage 5: Flush, a segment is born

Flush is where the codec takes over (package codecs). The in-memory hash is sorted into term order and streamed through the current codec's writers into a new segment: the blocktree terms writer produces the term dictionary (.tim) and its FST-based index (.tip), the postings writer produces doc ids and frequencies (.doc) and positions (.pos), the stored fields writer produces .fdt/.fdx, norms go to .nvd/.nvm, and a .si file describes the segment. The segment is now durable bytes but not yet part of "the index": searchers cannot see it until a reader reopen (near real-time) and it does not survive crashes until commit(), which fsyncs the files and atomically writes a new segments_N manifest via SegmentInfos. That two-step visibility (flush for searchability, commit for durability) is the exact mechanism under Elasticsearch's refresh versus flush distinction.

Stage 6: A searcher opens the index

On the read side, DirectoryReader.open(directory) reads segments_N, opens a SegmentReader per segment, and freezes that list as a point-in-time snapshot. IndexSearcher (package search) wraps the reader and treats the index as what it really is, a list of independent leaves: every search below is a loop over segments whose per-segment results are merged at the end. With MMapDirectory (the default on 64-bit platforms, package store) all file access is memory-mapped, so a warm index is served almost entirely from the OS page cache.

Stage 7: Query, Weight, Scorer

searcher.search(query, 10) compiles the query in three steps that mirror plan-then-execute in a database. The Query is immutable and reusable. From it the searcher creates a Weight, the per-search compiled form that has seen index-wide statistics (this is where BM25's idf is computed, once). From the weight, one Scorer is created per segment; the scorer is the iterator that actually walks postings and produces scores.

Stage 8: Term dictionary lookup

The scorer's first job is finding the term. The blocktree reader walks a finite state transducer, the .tip index held in memory (built on util.fst), which maps term prefixes to on-disk blocks in .tim. One FST walk plus one block read yields the term's metadata: its document frequency, total occurrences, and file pointers into the postings files. If the term does not exist in this segment, the search of that leaf ends here at almost zero cost.

Stage 9: Postings iteration

The PostingsEnum then iterates the term's documents from .doc: doc ids delta-encoded and bit-packed in blocks of 128, with skip data and per-block score upper bounds (impacts) interleaved. For a single TermQuery this is a straight scan; for conjunctions, the skip data lets one clause leapfrog the other without decoding the blocks in between; and the impacts let top-k search skip whole blocks that provably cannot beat the current worst hit, the block-max WAND family that landed in Lucene 8.

Stage 10: Scoring and top-k collection

For each doc id the postings produce, the BM25 SimScorer combines the precomputed idf, the term frequency from the postings, and the document's length norm (one byte per document, read from .nvd) into a score, and a TopScoreDocCollector keeps the ten best in a small priority queue. The collector feeds its worst-score-so-far back to the scorer via setMinCompetitiveScore, which is what arms the block skipping in stage 9. Per-segment top tens are merged, doc ids are rebased from segment-local to global, and the TopDocs you get back is just scored integers; a second, separate trip through StoredFields turns ids into stored documents.

Part V: Internals deep dives

Deep dive: immutable segments and the index lifecycle

A segment, once written, is never modified. Updates are a delete plus a reinsert, and a delete does not touch the segment at all; it flips a bit in a per-segment live-docs bitmask that scorers consult at search time. This one constraint buys the properties that make Lucene pleasant to operate: readers need no locks because nothing they read can change; snapshots are free because a reader just holds a list of segments; commits are cheap because only the tiny manifest is swapped; caches (both Lucene's and the OS page cache) never need invalidation, only eviction.

segments_5 -> [ seg_0 (500k docs, 3% deleted) ]
              [ seg_1 (120k docs)             ]
              [ seg_2 (8k docs)               ]      <- recent flushes
              [ seg_3 (8k docs)               ]

  merge(seg_2, seg_3, ...) runs in background, writes seg_4,
  then segments_6 -> [ seg_0, seg_1, seg_4 ] and old files are deleted
  once no open reader still references them.

The price is merging. Small segments multiply per-query overhead (every segment pays its own term lookup), and deleted documents occupy space and skew statistics until merged away, so TieredMergePolicy continuously folds segments into larger ones on background threads (ConcurrentMergeScheduler), rewriting live documents and dropping dead ones. This is write amplification, exactly as in LSM storage engines, and it is the knob-rich part of the system: merge policy settings trade indexing throughput against search latency against disk churn. Two traps worth correcting explicitly. First, deleting documents does not free space or even reduce hit counts' denominator immediately; maxDoc still counts them and only merges reclaim them. Second, the old advice to "optimize" an index with forceMerge(1) is a trap for actively written indexes: it creates one giant segment that the merge policy then must rewrite wholesale forever after; it is only sensible for indexes that have become read-only.

Deep dive: the codec layer and the postings format

Every on-disk structure in a segment is written and read through the codec API (codecs package): a codec is a versioned bundle of formats, one per structure, and each segment records which codec wrote it, so one index can contain segments written by different Lucene versions, with the backward-codecs module keeping old generations readable and merges naturally rewriting them into the current one. Format generations are named for the release that introduced them (Lucene90StoredFieldsFormat, Lucene101PostingsFormat, with newer generations on the main branch), which makes the file extensions in an index directory unusually legible:

Two encodings carry most of the weight. The term dictionary uses a finite state transducer as its index: think of a trie whose shared prefixes and shared suffixes are both collapsed, with outputs (file offsets) distributed along the arcs, giving a memory footprint small enough to hold the index of hundreds of millions of terms in RAM; ordered iteration also falls out, which is what prefix, wildcard, and range queries are built on. The postings use delta encoding plus bit-packing in fixed blocks of 128 doc ids, so a block whose deltas all fit in 5 bits costs 5 bits per document, and skip data plus impacts ride alongside. And beside the inverted structures sit doc values, a column store per field serving sorting, faceting, and aggregations: Lucene has quietly been a hybrid row and column store for over a decade. The misconception to kill here: Lucene is not a B-tree database with a text frontend and it does not "store your documents" in the index structures; stored fields are a separate, unrelated row store, and the inverted index knows only terms and integers.

Deep dive: BM25, worked numerically

Ranking is factored behind the Similarity class (package search.similarities); BM25 has been the default since Lucene 6.0, with defaults k1 = 1.2 and b = 0.75. Lucene computes, per matching document:

score = idf * tf_part

idf     = ln( 1 + (N - df + 0.5) / (df + 0.5) )
tf_part = freq / ( freq + k1 * (1 - b + b * dl / avgdl) )

N = docs in index   df = docs containing term
freq = occurrences in this doc   dl = this field's length   avgdl = average length

(Textbook BM25 multiplies by a constant (k1 + 1); Lucene dropped it in 8.0 because a constant factor cannot change ranking, so Lucene's absolute scores run lower than the textbook's.) Work it on a toy corpus: N = 4 documents, the term appears in df = 2 of them, average body length avgdl = 5. Then idf = ln(1 + 2.5/2.5) = ln 2 ≈ 0.693. Document A has freq = 2 in a 4-term body: tf_part = 2 / (2 + 1.2 × (0.25 + 0.75 × 4/5)) = 2 / 3.02 ≈ 0.662, score ≈ 0.459. Document B has freq = 1 in a 10-term body: tf_part = 1 / (1 + 1.2 × (0.25 + 1.5)) = 1 / 3.1 ≈ 0.323, score ≈ 0.224. Now watch the two behaviors that make BM25 better than raw TF-IDF. Saturation: give document B freq = 5 in the same 10-term body and tf_part = 5 / 7.1 ≈ 0.704; five times the occurrences bought barely double the contribution, and as freq grows the tf_part approaches 1, so a term can never contribute more than its idf. Length normalization: A and B both mentioning the term once would not tie; the shorter document wins, because a mention is stronger evidence in four terms than in ten.

Two implementation details matter in practice. Document length is not stored exactly: it is squeezed into a single byte per document (the norm), so lengths are bucketed and two slightly different lengths can score identically; searcher.explain(query, docId) prints the exact numbers Lucene used, and reading an explain tree is the fastest way to debug a surprising ranking. And the whole thing is pluggable: ClassicSimilarity keeps the old TF-IDF behavior, BooleanSimilarity makes scores degenerate to boosts, and a custom Similarity is the sanctioned way to change ranking without touching the index format, though changing similarity may require reindexing when it changes how norms are encoded.

Deep dive: Lucene under Elasticsearch, OpenSearch, and Solr

The fastest way to demystify the search servers is a dictionary. An Elasticsearch or OpenSearch shard is exactly one Lucene index; a Solr core likewise. Refresh is a near-real-time reader reopen, which is why the default 1-second refresh interval is the staleness bound of search results. Flush in Elasticsearch is a Lucene commit, and because commits are too expensive to do per write, the server layers a translog (write-ahead log) on top so acknowledged writes survive crashes between commits, a structure Lucene itself deliberately does not provide. Force merge is forceMerge, with the same read-only-indices-only caveat. Segment replication (copying immutable segment files to replicas, rather than re-executing writes on each) is only possible because segments never change. The mapping layer, analyzers per field, doc values on by default for keywords and numerics, is configuration compiled down to the Field flags from Stage 1. When an Elasticsearch behavior seems mysterious, the productive question is almost always "what is the Lucene mechanism under this", and my Elasticsearch note walks that dictionary in the other direction.

Part VI: Reading the repository

The repository is a Gradle build with one module per artifact under lucene/: core, analysis, queryparser, codecs, backward-codecs, facet, suggest, highlighter, luke, demo, and friends. Nearly everything that matters first lives in lucene/core/src/java/org/apache/lucene/, whose packages map cleanly onto this chapter: document and analysis for what goes in, index for the write side, search for the read side, store for the Directory abstraction, codecs for formats, util for the data structures. Read it as a syllabus:

Stage 0, the demo. Read lucene/demo's IndexFiles and SearchFiles top to bottom and run them. You should be able to answer: what five objects does every indexing program construct, and what three does every search program construct? Which analyzer is used on each side?

Stage 1, what goes in. Read document/Document.java, Field.java, the field-type presets, then analysis/TokenStream.java and one concrete analyzer. Questions: what three independent decisions does a field type encode? Why are token attributes mutated in place rather than allocated per token?

Stage 2, the write path. In index/, read IndexWriter.java selectively (it is the largest class in the project; follow only addDocument, flush, and commit), then DocumentsWriter.java, DocumentsWriterPerThread.java, IndexingChain.java, and skim FreqProxTermsWriter.java. Questions: why does per-thread buffering remove contention, and what does it cost in segment count? What exactly makes a commit atomic? What does a delete physically write?

Stage 3, the read path. In search/, read IndexSearcher.java, then the Query, Weight, Scorer triple for TermQuery, then TopScoreDocCollector.java and similarities/BM25Similarity.java. Questions: which of Query, Weight, Scorer is per-search and which is per-segment, and why must idf be computed at the Weight level? How does the collector's minimum competitive score reach the scorer?

Stage 4, bytes and structures. Read store/MMapDirectory.java for why Lucene leans on the page cache, then in codecs/ the current postings format package and the blocktree reader, then util/fst/ at skim depth and util/bkd/ if you care about numeric and geo points. Questions: what is stored in .tip versus .tim? Why blocks of 128?

Where not to start: not IndexWriter.java cold (thousands of lines of concurrency and lifecycle before any insight), not the FST construction code (the hardest code in the repository), and not the facet or join modules, which presume the core model. Start at the demo and let one document pull you inward.

Part VII: Hands-on labs

Lab 1: name every file in an index. Index a few hundred files with the demo, then list the directory:

java -cp "modules/*" org.apache.lucene.demo.IndexFiles -index ./idx -docs ./some-text-dir
ls -la ./idx
# expect: _0.cfs _0.cfe _0.si segments_1 write.lock  (small flushes use compound files)

Match each extension to the table in the codec deep dive. If you only see .cfs, you have learned that small segments get wrapped in a compound file to save file handles; index more documents until plain .tim/.doc files appear.

Lab 2: inspect the index with Luke. From the binary distribution run bin/luke.sh, open ./idx, and browse: the term list per field with document frequencies, the postings for one term, the stored fields of one document, and the Search tab's explain output. This makes stages 8 through 10 concrete in ten minutes.

Lab 3: watch segments get born and die. In a small Java program, set config.setMaxBufferedDocs(100), index 1,000 tiny documents, and run watch -n 0.5 ls ./idx in another terminal: a new segment appears roughly every hundred documents, then the merge scheduler folds them together and the old files vanish. Finish with writer.forceMerge(1) and watch the directory collapse to one segment. You have now seen the entire lifecycle from the deep dive on disk.

Lab 4: make BM25 move. Index three documents: "duck", repeated twice in a 4-word body, once in a 10-word body, and five times in a 10-word body. Search for it and print searcher.explain(query, hit.doc) for each hit. Check the printed idf, freq, and norm-derived length against the worked example above; then reindex the freq = 5 body as freq = 50 and confirm the score creeps toward the idf ceiling instead of growing linearly.

Lab 5: deletes are bitmasks. Index 1,000 documents with a StringField("id", ...), delete half by term, and print reader.numDocs() versus reader.maxDoc() after reopening: 500 versus 1,000, and a .liv-suffixed file appears next to the segment instead of the segment shrinking. Then forceMerge(1) and check both numbers read 500.

Lab 6: the visibility model. With one writer open, add a document, then open reader A with DirectoryReader.open(writer) (sees it, no commit needed), and reader B with DirectoryReader.open(directory) (does not see it until commit()). Add another document and confirm A still does not see it until openIfChanged. This is Elasticsearch's refresh and flush, reproduced in fifteen lines.

Part VIII: Check your understanding

What is Lucene in one sentence, and what is it not? A Java library that maintains an inverted index as immutable segment files and answers queries with ranked document ids; it is not a server, not a distributed system, and not a document database, which is precisely the surface Elasticsearch, OpenSearch, and Solr sell.

Why are segments immutable, really? Immutability converts the hardest concurrency problem (readers racing writers over shared structures) into a garbage collection problem (when may old files be deleted). Readers get lock-free access and free snapshots, commits reduce to swapping a manifest, and the cost is deferred into background merges, which are schedulable and cancellable in a way that fine-grained locking never is.

Walk the write path of one document in five steps. Document with typed Fields; IndexWriter routes it to a per-thread DocumentsWriterPerThread; the analyzer tokenizes each indexed field; the terms hash accumulates postings in memory byte pools; flush sorts and streams it all through the codec into an immutable segment, made durable by a later commit of segments_N.

What happens between typing a term and getting a score? Per segment: the FST index in memory locates a term block in .tim, which yields postings file pointers; the postings enum walks 128-doc packed blocks from .doc; per doc id, the BM25 SimScorer combines precomputed idf, freq, and the one-byte length norm; a collector heap keeps top-k and feeds its threshold back to enable block skipping.

Why does Lucene search each segment separately? Because segments are independent mini-indexes, per-segment search needs no global structures, parallelizes naturally, lets caches key on immutable segment identity, and makes adding new data as cheap as adding one more leaf to the loop; the only global work is merging k small top-k lists.

What is a codec and why does the design pay for one? A versioned, pluggable specification of every on-disk structure in a segment, recorded per segment. It buys forward evolution: the project can ship better compression or postings layouts as a new generation while backward-codecs keeps old segments readable and merges migrate them incrementally, with no offline index conversion step ever required.

Why is the term dictionary index an FST rather than a hash or B-tree? The FST shares both prefixes and suffixes, so an index over hundreds of millions of terms fits in a few hundred megabytes of RAM or less, and it preserves order, which hash tables do not, so prefix, range, and wildcard iteration come for free; the trade-off is expensive construction, which is fine for write-once segments.

How does BM25 differ from raw TF-IDF, in behavior? Term frequency saturates toward a ceiling (in Lucene's form, the idf itself), so keyword stuffing stops paying quickly, and document length normalizes relative to the corpus average, so short fields are not unfairly dominated by long ones; both are governed by k1 and b rather than hard-coded.

Why do hit counts and disk usage not drop after deletes? A delete only sets a bit in the live-docs mask; the postings, stored fields, and statistics of the deleted document remain physically present and maxDoc still counts it. Space and statistics are reclaimed only when a merge rewrites the segment without its dead documents.

When is forceMerge(1) correct, and when is it a trap? Correct for indexes that will no longer be written, where one segment minimizes per-query overhead forever after. A trap for live indexes, because the resulting max-sized segment must be wholly rewritten to reclaim any future deletes, multiplying merge cost indefinitely.

What consistency does a searcher see during heavy indexing? A perfect point-in-time snapshot: the segment list captured at reader open, plus the deletes visible at that moment, unchanging for the reader's lifetime. Freshness is obtained only by reopening, which is what refresh means in the systems built on Lucene.

Why does Elasticsearch need a translog when Lucene already has commits? Lucene commits are fsync-heavy and too expensive per write, but without them a flushed-but-uncommitted segment does not survive a crash. The translog is a cheap append-only log that makes each write durable immediately, so commits can be batched; on recovery the translog is replayed into Lucene.

A query that should match returns nothing. Debugging order? First check field type: was the field indexed, and as TextField or StringField? Second, analyzer parity: run the query text through the index-time analyzer and compare tokens (Luke does this interactively). Third, reader freshness: was the reader opened before the write, or before a commit it needed? Those three cover nearly all real cases.

Why one IndexWriter per index, and how is it enforced? Concurrent writers would race on segment naming, merges, and the manifest, so the design mandates a single writer (internally multi-threaded) and enforces it with the write.lock file taken at writer construction; a second writer fails with LockObtainFailedException rather than corrupting the index.

Where would you change ranking without touching storage? Implement or configure a Similarity on the IndexSearcher (and IndexWriterConfig if norm encoding changes); Query boosts and query rewriting sit above it, and the postings format below it never needs to know.

Part IX: Design lessons

Immutability converts locking problems into lifecycle problems. Lucene's entire concurrency story is "never mutate what a reader can see," and the residue is reference counting and file deletion policy. The same move powers RocksDB's SSTables, Kafka's log segments, and copy-on-write filesystems: if coordination is hard, stop sharing mutable state and pay with background compaction.

Amortize expensive maintenance in the background. Merges do the deferred work (sorting, compressing, dropping deletes) off the write path at a controllable pace. This is the LSM bargain everywhere: fast ingest bought with background rewriting, tuned by policy objects that are pluggable precisely because the right trade differs per workload.

Version the format, not the API. The codec layer lets a 25-year-old project rewrite its storage encoding repeatedly, because each segment self-describes its format and migration piggybacks on merges. Protocol buffers' field numbering, database page format versions, and Kafka's message format versions are the same discipline: durability outlives any single encoding, so name your encodings.

Compile queries in stages scoped to their statistics. Query (immutable, reusable) to Weight (per-search, sees global stats) to Scorer (per-segment, does the iteration) is a clean separation of when information becomes available. It is the same shape as SQL's plan-then-execute split and as JIT tiering: bind each decision at the latest point it is still cheap and the earliest point it is informed.

Push thresholds down to the iterators. Top-k collection feeding its minimum competitive score down into block-skipping postings is predicate pushdown by another name, the same idea as LIMIT pushdown in databases and early termination in vector search. Senior systems move the stopping condition as close to the data as it can go.

Part X: The memorization framework

One sentence: Lucene analyzes documents into terms, buffers postings per thread, freezes them into immutable segments that background merges consolidate, and serves queries by walking an FST to postings to BM25 scores per segment, collecting top-k.

WRITE: Doc -> Analyzer -> DWPT buffers -> flush -> Segment -> merge -> commit(segments_N)
READ:  Query -> Weight -> per-segment Scorer -> FST(.tip) -> .tim -> .doc blocks -> BM25 -> top-k

Doc/Analyzer     org.apache.lucene.document, analysis
DWPT/flush       index/DocumentsWriterPerThread.java, IndexingChain.java
Segment files    codecs/* (current generation), store/MMapDirectory.java
Merge/commit     index/IndexWriter.java, SegmentInfos (segments_N)
Search           search/IndexSearcher.java, TermQuery -> Weight -> Scorer
Ranking          search/similarities/BM25Similarity.java
Structures       util/fst, util/bkd

Memorize these blocks:

• Defaults: 16 MB RAM buffer per writer before flush; postings packed in blocks of 128 doc ids; BM25 with k1 = 1.2, b = 0.75, default since 6.0; norms are one byte per document.
• Files: .tim/.tip terms, .doc/.pos postings, .fdt/.fdx stored, .nvd norms, .dvd doc values, .si + segments_N metadata, .cfs compound.
• Invariants: segments are immutable; one writer per index (write.lock); deletes are live-doc bits until merge; readers are point-in-time snapshots; flush makes searchable, commit makes durable.
• Ranking ceiling: in Lucene's BM25 the tf part is bounded by 1, so one term's contribution can never exceed its idf.