Elasticsearch

Elasticsearch is a distributed search and analytics engine that stores JSON documents and answers full-text, structured, geospatial, and aggregation queries over them through a REST API. It is best understood as a coordination layer wrapped around Apache Lucene, the low-level Java search library that has been in development since 1999. Lucene does the actual indexing and matching on a single machine. Elasticsearch adds sharding, replication, cluster membership, a JSON query DSL, and near-real-time semantics on top, so that a bag of Lucene indexes spread across dozens of nodes behaves like one searchable collection.

The core design bet is that search data can be a read-optimized copy rather than the system of record. Every structure inside it, from the inverted index to immutable segment files, trades write flexibility and update cost for fast reads over data that rarely changes in place. That bet shapes both how you use it, feeding it denormalized documents from an authoritative database through change data capture, and how it fails, which is mostly when people treat it as a primary store or hit it with update-heavy workloads.

What it is and the mental model

The vocabulary maps loosely onto relational terms. A document is a JSON object, the unit you index and retrieve. An index is a named collection of documents, roughly a table. A mapping is the index schema. It declares each field's type and, for text, how it gets analyzed. The two most important types are text, which is tokenized and normalized for full-text matching, and keyword, which is stored as a single exact value for filtering, sorting, and aggregations. Numeric, date, and geo fields get their own structures, including BKD trees for geo_point and numeric ranges.

The physical picture is a set of nesting dolls. A cluster is made of nodes with roles, one elected master for cluster state, data nodes that hold shards, coordinating nodes that front client requests, and optional ingest nodes for transform pipelines. Each index is split into primary shards, and each primary can have replica copies on other nodes. A shard is exactly one Lucene index, and a Lucene index is a collection of immutable segments, each of which is a self-contained mini search engine with its own inverted index, doc values, stored fields, and a bitmap of live documents.

Almost every behavior that surprises people follows from two facts. Segments are immutable, so updates and deletes are bookkeeping plus background rewriting. And new writes become searchable only when a refresh produces a new segment, so search is near real time rather than read-your-writes.

How to use it

Everything goes through the REST API as JSON. A realistic setup declares the analyzer chain and the mapping up front rather than relying on dynamic mapping, since unmapped or wrongly mapped fields waste memory and cannot be fixed without reindexing. The example below builds a small product catalog, indexes a document, and runs a search that combines full-text matching, a structured filter, and two aggregations in one round trip.

The bool query separates query context from filter context, and the distinction matters for performance. Clauses under must contribute to the relevance score. Clauses under filter are yes-or-no predicates, skip scoring entirely, and their results are cached as bitsets that later queries reuse. Anything that does not need to influence ranking belongs in filter.

Concurrent updates use optimistic concurrency control. Every write returns _seq_no and _primary_term, and a conditional update passes them back as if_seq_no and if_primary_term so the write fails cleanly if someone else changed the document first. The older ?version=N mechanism was removed for this purpose in favor of sequence numbers. For pagination, from and size work for shallow pages but every shard must produce from + size candidates, so the default index.max_result_window caps it at 10,000 results. Deep pagination uses search_after with the sort values of the last hit, optionally combined with a point-in-time (PIT) handle that pins the set of segments being read so results stay consistent while the index changes underneath.

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "english_folded": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "porter_stem"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name":     { "type": "text", "analyzer": "english_folded" },
      "brand":    { "type": "keyword" },
      "price":    { "type": "float" },
      "rating":   { "type": "float" },
      "added_at": { "type": "date" }
    }
  }
}

POST /products/_doc
{
  "name": "Trail running shoes",
  "brand": "Salomon",
  "price": 129.95,
  "rating": 4.6,
  "added_at": "2026-07-01"
}

GET /products/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "name": "running shoe" } } ],
      "filter": [ { "range": { "price": { "lte": 150 } } } ]
    }
  },
  "aggs": {
    "by_brand":   { "terms": { "field": "brand" } },
    "avg_rating": { "avg":   { "field": "rating" } }
  },
  "size": 10
}

Analysis and the inverted index

Full-text search works because both documents and queries pass through the same analyzer, a pipeline of character filters, then a tokenizer, then token filters. The default standard analyzer splits on Unicode word boundaries and lowercases, and custom chains add stemming, ASCII folding, synonyms, or n-grams. In the mapping above, "Trail running shoes" becomes the terms trail, run, shoe, and the query "running shoe" becomes run, shoe, which is why they match. If index-time and query-time analysis disagree, matching silently breaks, which is the most common beginner failure mode (Elastic docs).

The terms feed the inverted index, the structure that makes lookup fast. Instead of storing documents and scanning them for words, it stores every unique term once, in a sorted term dictionary, and attaches to each term a postings list of the document IDs containing it, along with per-document term frequencies and positions for phrase queries. Lucene keeps an in-memory finite state transducer as a compact prefix index into the on-disk term dictionary, so locating a term is cheap even with hundreds of millions of unique terms. Postings lists store doc IDs in ascending order with delta compression and skip pointers, so intersecting the lists for a multi-term query can leapfrog through the longer list. Query execution starts from the rarest term because the intersection can never be larger than the shortest postings list, which is why a query for "bill nye" is driven by the few hundred nye postings rather than the millions of bill ones.

An inverted index answers "which documents contain this term" but is useless for "give me the price of these 200 matches to sort them." That is what doc values are for, a columnar, disk-backed layout that stores one field's values for all documents in a segment contiguously, exactly the trick that column stores use for analytics. Sorting, aggregations, and script field access all read doc values. So every indexed field effectively exists twice, once inverted (term to documents) for matching, and once columnar (document to value) for sorting and aggregating.

The write path inside a shard

When an indexing request reaches the primary shard, the document is analyzed and added to an in-memory indexing buffer, and the operation is appended to the translog, a sequential write-ahead log on disk. With the default index.translog.durability: request, the translog is fsynced on the primary and on every allocated replica before the client gets an acknowledgement, so an acknowledged write survives process crashes and power loss. Setting durability to async moves the fsync to a background timer, sync_interval defaulting to five seconds, which buys indexing throughput at the cost of losing up to that window of acknowledged writes on a crash (Elastic docs).

Documents in the buffer are not yet searchable. A refresh, by default every one second on indices that have been searched in the last thirty seconds, writes the buffer out as a brand new Lucene segment. The trick that makes this cheap is that the segment goes to the operating system's filesystem cache, not through an expensive fsync, and once the file is in the cache it can be opened and searched like any other file (Elastic docs). This is the entire meaning of "near real time": a write becomes visible at the next refresh, about a second later, long before it is durably committed by Lucene. Durability in the gap is the translog's job. A flush performs a real Lucene commit, fsyncing segments to disk and trimming the translog, and is triggered automatically, by default once the translog reaches 10 GB.

Segments are immutable. A delete never touches segment data, it just marks the document dead in a per-segment live-docs bitmap, and an update is a soft delete plus a reindex of the new version into the current buffer. Dead documents still occupy space and still get skipped at query time, so the merge scheduler continuously picks groups of similar-sized segments, using Lucene's tiered merge policy, and rewrites them into one larger segment with the deleted documents dropped. Merges are disk-IO throttled and run in the background with smaller merges prioritized over larger ones so they do not starve indexing (Elastic docs).

Immutability is what pays for everything else. Segment files can be cached by the OS and by Elasticsearch's own filter and query caches without invalidation logic, compressed aggressively because they are written once, and read lock-free because nothing ever changes under a running query. The price is write amplification. Reclaiming a handful of updated documents can eventually mean rewriting a multi-hundred-megabyte segment, which is exactly why update-heavy workloads hurt.

A write is durable via the translog almost immediately, searchable at the next refresh, and committed by a later flush. Merges compact the resulting segments in the background.

Shards, replicas, and query then fetch

Documents are routed to a primary shard by hash(_id) % number_of_primary_shards unless a custom routing key is supplied. Because placement is a function of the shard count, the number of primaries is fixed at index creation, and growing it later means the split API or a full reindex into a new index. This makes shard sizing a real capacity-planning decision, with Elastic's published guidance keeping individual shards in the range of tens of gigabytes. Each primary replicates its operations to its replica shards, which serve two purposes, surviving node loss and multiplying read throughput, since the coordinating node balances searches across all copies of each shard.

Search executes in two phases, called query then fetch. In the query phase, the coordinating node fans the query out to one copy of every shard of the index. Each shard runs the query locally against its segments and returns only a lightweight priority queue of the top from + size entries, doc IDs plus scores or sort values, not documents. The coordinator merge-sorts these per-shard queues into the global top results. In the fetch phase it asks only the shards that own the winning documents for their _source, highlights, and stored fields. This is why deep pagination is expensive, page one thousand forces every shard to score, sort, and ship ten thousand candidates so the coordinator can discard almost all of them.

The same fan-out explains why the coordinating node never needs to know which shards contain a term. It cannot know, so it broadcasts, and the per-shard inverted indexes do the actual pruning. It also explains the consistency model. Replicas refresh independently and a search hits one arbitrary copy per shard, so two identical queries a second apart can see slightly different worlds. Elasticsearch is eventually consistent for search by construction.

Relevance scoring with BM25, and aggregations

When no explicit sort is given, hits are ordered by _score, computed since Elasticsearch 5.0 by Okapi BM25 rather than classic TF-IDF. BM25 keeps the same three intuitions, rarer terms matter more, more occurrences matter more, and long documents get discounted, but it bounds them. The inverse document frequency component penalizes terms that appear in many documents. The k1 parameter, defaulting to 1.2, makes term frequency saturate, so the fifth occurrence of a term adds far less than the first and a keyword-stuffed document cannot run away with the ranking, where TF-IDF's term frequency grew without bound. The b parameter, defaulting to 0.75, controls how strongly the score is normalized by document length relative to the average field length in the index (Elastic, Practical BM25).

One distributed-systems wrinkle is that scores are computed per shard against shard-local term statistics. With few documents or skewed routing, the same document can score differently depending on which shard holds it. The dfs_query_then_fetch search type adds a round trip to gather global term statistics first, and is mostly useful for small or test indexes where the skew is visible.

Aggregations are the analytics half of the engine and the reason Kibana dashboards exist. Metric aggregations compute values like avg, sum, and percentiles, bucket aggregations group documents by term, range, date histogram, or geo grid, and the two nest arbitrarily, average rating per brand per month is one request. They execute during the query phase, each shard aggregates its local documents by scanning doc values, and the coordinator reduces the partial results. Some of that reduction is deliberately approximate. A terms aggregation returns each shard's top buckets, so global counts can be off when a term is popular on one shard and marginal on others, and cardinality uses HyperLogLog++ to estimate distinct counts in fixed memory.

BM25 score for query q against document D:

score(D, q) = sum over terms t in q of

  IDF(t) * ( tf(t, D) * (k1 + 1) )
              / ( tf(t, D) + k1 * (1 - b + b * |D| / avgdl) )

k1 = 1.2  -> term-frequency saturation (higher = slower saturation)
b  = 0.75 -> document length normalization (0 = off, 1 = full)
avgdl     -> average field length across the index

Trade-offs, and when not to use it

Do not make it the system of record. The Jepsen analysis of Elasticsearch 1.5.0 found it lost acknowledged writes in every failure scenario tested, roughly 10 percent of acknowledged documents on crashed nodes, and 22 percent when a primary was isolated by a network partition, and recommended keeping data in a safer database and continuously upserting into Elasticsearch (Jepsen). Modern versions are dramatically better, version 7.0 replaced the old Zen discovery with a formally modeled cluster coordination layer and replication now uses sequence numbers and primary terms, but the industry-standard architecture is unchanged, an authoritative store such as Postgres or DynamoDB in front, with change data capture feeding the search index.

Update-heavy and counter-like workloads fight the storage engine. Every update is a soft delete plus a full reindex of the document, and reclaiming the dead versions means merges rewriting large immutable segments. A field like a like-count that changes constantly generates enormous churn for data that probably should not be searchable anyway. High-volume append-only ingestion, by contrast, is a strength, which is why log and time-series pipelines work so well.

Search results are stale by design. Between the refresh interval, independent replica refreshes, and indexing lag from the CDC pipeline, results trail the source of truth by seconds or more. Most search products tolerate this happily, workflows needing read-your-writes do not.

It is not relational. There are no joins and no multi-document transactions, so data must be denormalized per access pattern, the same discipline as query-first modeling in Cassandra, and a page of results should come from one or two queries. Nested fields and parent-join exist but carry real query-time costs. And below roughly a few hundred thousand documents the operational weight is rarely justified, Postgres full-text search with a GIN index handles a surprising share of "we need search" requirements with zero extra infrastructure to keep in sync, since index drift between the primary store and the search cluster is one of the most common real-world bug sources.

How it shows up in production

The canonical deployment is a projection pipeline. Writes go to the authoritative database, a CDC tool such as Debezium tails the replication log (or DynamoDB Streams on AWS), changes flow through Kafka, and an indexer service applies them to Elasticsearch with the source system's primary key as the document _id so updates are idempotent upserts. Because the index is rebuildable from the source, teams reindex from scratch to change mappings or analyzers rather than migrating in place.

The scale numbers from real operators show both the power and the limits. Uber described storing more than 800 billion documents across its marketplace Elasticsearch clusters, sustaining over 1.5 million document writes per second and scanning billions of documents every second to serve thousands of queries, operated by a three-person team running more than one hundred ingestion jobs (InfoQ, QCon London 2018). The limits show up in Uber's logging platform, which ran on ELK from 2014 at millions of log lines per second from thousands of services with petabytes retained, and was ultimately migrated to ClickHouse because schema-on-write and per-node cost hurt at that scale, a single ClickHouse node ingested about ten times the log throughput of a single Elasticsearch node (Uber Engineering).

Log and metrics analytics through the ELK stack (Elasticsearch, Logstash or Beats, Kibana) is still probably the most widely deployed pattern, and it leans on time-based index management. Data streams roll over to a fresh backing index by age or size, and index lifecycle management migrates aging indices across hot, warm, cold, and frozen data tiers, from fast NVMe nodes down to searchable snapshots on object storage, so retention cost tracks how often anyone actually queries the data. Product search is the other big family, Wikipedia's on-site search runs on Elasticsearch through its CirrusSearch extension, and the same engine increasingly serves hybrid retrieval, BM25 plus dense vector similarity, for RAG-style applications.

Follow-up questions

• Why is Elasticsearch called near-real-time instead of real-time? New documents sit in an in-memory buffer that is not searchable. A refresh, every one second by default on indices that have been searched recently, writes the buffer into a new Lucene segment in the filesystem cache, and only then can searches see it. So visibility lags writes by up to the refresh interval, plus any lag in the pipeline feeding the index.
• If refreshed segments are not fsynced, how does an acknowledged write survive a crash? Through the translog. Every operation is appended to this write-ahead log and, with the default request durability, fsynced on the primary and replicas before the client is acknowledged. After a crash, the shard reloads the last committed segments and replays the translog. A flush performs the real Lucene commit and trims the log, by default once it reaches 10 GB.
• How do updates and deletes work if segments are immutable? A delete just flips a bit in the segment's live-docs bitmap, and an update soft-deletes the old version and reindexes the new one as a fresh document. Dead documents keep consuming space until a background tiered merge rewrites the segments and drops them, which is the write amplification that makes update-heavy workloads a poor fit.
• Walk through what happens when a search request hits the cluster. The coordinating node broadcasts the query to one copy of every shard. In the query phase each shard returns only the top from-plus-size doc IDs with their scores or sort values. The coordinator merge-sorts those into the global top N, then in the fetch phase retrieves the actual documents from just the shards that own the winners. Deep pagination is expensive because every shard must produce all preceding candidates.
• Why is the primary shard count fixed at index creation? Routing places each document with hash of the ID modulo the number of primaries. Changing the count would invalidate the placement of every existing document, so growing an index means the split API or reindexing into a new index, which is why shard sizing is planned up front, typically keeping shards to tens of gigabytes.
• What does BM25 improve over TF-IDF? Two bounded corrections. Term frequency saturates via k1, default 1.2, so repeated occurrences of a term give diminishing returns instead of growing without limit, and document length normalization via b, default 0.75, discounts matches in long documents relative to the average field length. Both defaults are tunable per field through the similarity setting.

References

1. Elastic Docs: Near real-time search, The indexing buffer, refresh, segments in the filesystem cache, and the one second default refresh interval on recently searched indices.
2. Elastic Docs: Translog settings, Durability modes (request vs async), fsync on primary and every allocated replica before acknowledgement, the five second sync interval, and the 10 GB flush threshold.
3. Elastic Docs: Text analysis, Overview of analyzers, tokenization, and normalization, the entry point to analyzer anatomy and index-time versus search-time analysis.
4. Elastic Blog: Practical BM25, Part 2, The BM25 formula with the k1 = 1.2 and b = 0.75 defaults, saturation, and length normalization.
5. Jepsen: Elasticsearch 1.5.0, Measured loss of acknowledged writes under partitions and crashes, about 10 percent on crashed nodes and 22 percent with isolated primaries, and the keep-the-truth-elsewhere recommendation.
6. InfoQ: Scaling Uber's Elasticsearch Clusters, Production scale numbers, more than 800 billion documents and 1.5 million writes per second, run by a three-person team.