How Search Actually Works — Lucene, Inverted Indexes, Segments, and Distributed Search.

A search box looks deceptively simple: a person types “biryani,” presses Enter, and expects nearby restaurants, relevant dishes, spelling tolerance, filters, and sensible ordering almost immediately. A database table can store those facts, but scanning every row and splitting every text field at query time does not scale. Modern full-text search works because it pays a large, deliberate cost before the query arrives: it transforms documents into indexes designed for retrieval.

This article follows the path from a document entering a system to a ranked result returning to a user. The transcript that prompted it has the right central idea—Apache Lucene is the engine beneath many search systems—but the useful engineering story is more precise. Lucene is a search library, not a distributed service. Elasticsearch, OpenSearch, and Solr provide different distributed, operational, and API layers around Lucene. And the speed of a query comes from data structures, analysis choices, ranking models, cache behavior, and capacity planning rather than one magic lookup.

Search is retrieval over a bounded corpus

A web search engine has to crawl, deduplicate, index, and rank much of the public web. Product search, documentation search, a restaurant catalog, and an internal knowledge base solve a narrower version: retrieve useful objects from their corpus. Their product rules differ—distance and delivery time matter for food, permissions and freshness matter for internal documents—but their text-retrieval core is similar.

That is why it is useful to separate a search product from a search engine. The product owns data ingestion, business rules, autocomplete, synonyms, security filtering, UI, experimentation, and operational targets. The engine executes queries against an index. Lucene supplies low-level Java APIs and on-disk index structures. Elasticsearch, OpenSearch, and Solr expose cluster-oriented services that distribute indexes, coordinate queries, handle replicas, and add administration APIs. None automatically knows what “best restaurant” means; the application must model and rank that decision.

Documents, fields, terms, and the analysis contract

In Lucene vocabulary, a document is the unit written to an index. A restaurant could be one document with fields such as name, menu, cuisine, rating, latitude, and is_open. A documentation page might have title, body, product, version, and published_at. A document need not be a file; it is a structured record suitable for retrieval.

Fields matter because search behavior is field-specific. A title is often boosted more heavily than a body. An identifier should usually be indexed as one exact value, while prose must be tokenized. A date should be a sortable/range-queryable value, not merely text. Search quality starts with this schema decision: there is no analyzer configuration that fully compensates for a field whose meaning was never modeled.

Before indexing text, an analyzer turns character input into terms. A typical chain might normalize case, split on word boundaries, remove or retain punctuation according to domain needs, apply language-aware stemming, and expand a synonym set. “Running shoes” might yield running and shoes; an ID such as ERR-404 must usually preserve its punctuation and exact form. The query must be analyzed compatibly with indexed text. If the index turns “C++” into one form and the query parser turns it into another, the apparent relevance bug began before retrieval.

Analysis is not cleanup. It is the retrieval contract that defines what counts as the same term, what can match, and which information has been discarded.

The inverted index: reverse the expensive question

A naïve search asks every document, “do you contain this word?” That is an expensive document-to-term scan. An inverted index reverses the mapping: for each term, it stores the documents that contain it. A tiny corpus might look like this:

Document	Terms after analysis
D1	sun, bright, day
D2	moon, bright, night
D3	sun, moon, light

The inverted form has a term dictionary and postings lists: sun → [D1, D3]; moon → [D2, D3]; bright → [D1, D2]. A search for sun jumps to the posting list rather than reading D1, D2, and D3 in full. Real indexes add compressed document IDs, term frequencies, positions, offsets, norms, and skip data. Positions enable phrase queries such as “machine learning”; offsets support highlighting; frequencies and lengths support scoring.

This design answers more than exact keyword questions. Boolean queries intersect or union postings. A phrase query checks positions. A filter for is_open:true narrows candidates. Numeric and geo structures support range and distance operations. The key principle remains the same: precompute an access path that makes the expected query cheap.

Retrieval is not ranking

Finding candidates is only the first stage. Users care which result comes first. Lucene commonly uses BM25-style lexical scoring, which rewards terms that occur in a document while discounting terms that are common across the corpus and normalizing for length. A match in a concise title is often more informative than the same token buried in a long description.

Production ranking combines this text score with domain signals. A catalog might boost availability, quality, price, distance, and personalization. Documentation search may boost current versions, authoritative sources, and exact title matches. Facets—brand, cuisine, price range, version, location—are usually structured aggregations and filters around the text query; they are not merely visual controls. The ranking contract must be observable: teams need query judgments, click/abandonment metrics, and controlled experiments, not a claim that “the engine is fast.”

Why Lucene uses immutable segments

A Lucene index is composed of segments, each a complete searchable index over a subset of documents. New documents are buffered and then flushed as new segments. Existing segments are immutable: an update or delete does not rewrite a segment in place. An update is logically a delete plus an addition; a delete is tracked so readers exclude the old document; later merges reclaim the obsolete data.

Immutability is an engineering trade. Readers can search a stable point-in-time view without locks around every posting. Writers can create new data beside old data. The cost is temporary duplication and background work. As segments accumulate, Lucene’s merge policy chooses groups to combine into larger segments. The merge writes a new segment, then the system can retire old files when no reader needs them. This improves search efficiency and removes deleted documents, but consumes I/O, CPU, and sometimes substantial temporary disk space.

The transcript’s “merge sort” analogy is helpful only as intuition. A Lucene merge is an index-level rewrite with multiple structures, deletion handling, and policy decisions—not a simple sort of arbitrary infinite input. Operators should avoid casually forcing merges on a busy, mutable index. The Lucene documentation explicitly warns that forced merges can be costly and need extra disk; routine merging should be allowed to follow the configured policy.

Refresh, flush, commit, and the freshness question

These words are often mixed together. A refresh makes recently indexed operations searchable to a new reader view; it does not necessarily mean the data has been durably committed to disk. A flush/commit establishes a durable index state and related transaction-log semantics differ by service. Lucene readers retain a consistent point-in-time view until explicitly refreshed. Therefore “I wrote a document” and “a search sees the document” are distinct events.

This is not a defect. It lets a service trade throughput, latency, and durability. A log-search product can accept near-real-time visibility. A user-facing catalog may require a tight refresh target for inventory. A financial or legal workflow must define exactly which system of record confirms an update; a search index is often a derived view, not the authority.

From one Lucene index to a distributed service

Lucene itself is not a distributed database. Elasticsearch, OpenSearch, and Solr use multiple Lucene indexes to scale and operate search. In Elasticsearch and OpenSearch terms, an index is split into primary shards; each shard is a separate Lucene index. Replica shards provide copies for availability and can help spread read load. A coordinating node fans a query out to relevant shards, each shard returns its best candidates, and the coordinator merges them into the final page.

Sharding is not free scale. Every shard has metadata, caches, segments, and execution overhead. Too many small shards waste resources and increase coordination cost; too few large shards constrain recovery and parallelism. Elasticsearch’s documentation notes that each shard shares Lucene’s maximum-document limits because it is itself a Lucene index. Capacity planning must consider document count, shard size, ingest rate, query concurrency, aggregation cost, replica policy, and recovery time—not just total storage.

Solr and OpenSearch offer their own distribution and operations models, but the same questions remain: what is the routing key, how are replicas placed, how are failures recovered, how fresh is a replica, and what happens when one shard is slow? A fast local inverted-index lookup does not guarantee a fast p99 cluster query.

A practical search architecture

Own the source of truth. Keep transactional inventory, permissions, and primary business records outside the search index unless the search platform is explicitly designed as the authority.
Build an idempotent ingestion path. Transform source changes into document writes, version them, and safely replay from a checkpoint after failure.
Design mappings and analyzers from real queries. List exact, prefix, phrase, fuzzy, filter, sort, and aggregation requirements before indexing production data.
Measure relevance and operations separately. Track zero-result rate, judged relevance, click behavior, indexing lag, refresh cost, query latency, shard imbalance, merge pressure, and disk headroom.
Protect the query surface. Apply authorization filters, limit expensive queries, bound aggregation cardinality, and avoid exposing an unrestricted query-string interface to untrusted users.

What changes in the age of semantic search?

Vector search adds an embedding-based retrieval path for semantic similarity. It does not replace lexical indexing. Exact product names, error codes, IDs, recency filters, permissions, phrases, and explainable matches are often better served by traditional fields and inverted indexes. A hybrid system retrieves candidates from lexical and vector indexes, applies filters, fuses scores, and uses domain-aware ranking. The same hard questions remain: what corpus was indexed, what transformations changed the query, what evidence supports the result, and how is quality measured?

The durable mental model is simple. Search is a data product built around a retrieval index. Lucene makes term-based lookup fast by analyzing fields and inverting document text into postings. Immutable segments make concurrent reads and writes tractable; merge policies pay down the resulting fragmentation. Distributed systems such as Elasticsearch, OpenSearch, and Solr add shards, replicas, APIs, and operational responsibility. The user sees one box. The engineering work is everything that makes its answer fast, relevant, fresh, and safe.

Failure modes worth designing for

Search failures are rarely caused by the absence of an index. They are often caused by a mismatch between a promise made in the interface and a property the system does not actually guarantee. A user may expect a permission change to disappear from results immediately, while the index refreshes on a near-real-time interval. A marketplace may allow a broad wildcard or fuzzy query that expands into expensive work across every shard. A deployment may change an analyzer or synonym list without reindexing the affected fields, leaving old and new documents searchable under different rules. These are product and release problems, not merely tuning opportunities.

Make failure behavior explicit. Show a useful zero-result state and capture the query for analysis. Route indexing failures to a durable retry or dead-letter path rather than silently dropping source changes. Monitor queue age so that an apparently healthy search endpoint cannot hide a stale catalog. Use aliases or equivalent indirection for index migrations: build and validate a new index, switch readers atomically, then retire the old one after rollback safety has expired. Test permission filters with adversarial queries, because a correct text match that exposes an unauthorized document is a security incident.

Finally, distinguish relevance from availability. A search service can return HTTP 200 in ten milliseconds while placing the wrong answer first, omitting a recent document, or leaking a restricted title through a facet. Keep a small judged-query set in version control, replay it before ranking or analyzer changes, and pair it with operational load tests. Search quality is a living contract between language, data, business policy, and infrastructure; the most reliable systems make every part of that contract measurable.