Elasticsearch & Full-Text Search Explained: Inverted Index, Relevance & Scaling

Every e-commerce search box, log explorer and help-center search you use is almost certainly backed by an inverted index, very often Elasticsearch or its fork OpenSearch. This guide explains how it actually works, from tokenizing a product title to the scoring formula that ranks results, to the shard and segment mechanics that make it scale, and the pipeline that keeps it in sync with your primary database. Examples use a food and grocery catalog in the style of Swiggy Instamart or BigBasket.

Why Not Just Use SQL LIKE?

The naive approach is SELECT * FROM products WHERE title LIKE '%butter%'. It fails in four ways:

1. Performance. A leading wildcard cannot use a B-tree index, so the database scans every row. At 50 million products, every keystroke becomes a full table scan. (B-trees and why prefixes work but infixes do not are covered in Database Indexing Explained.)
2. No linguistics. "running shoes" will not match "run shoe". "Biryani" will not match "biriyani".
3. No ranking. LIKE returns matches, not the best matches. A product mentioning butter once in its description ranks the same as "Amul Butter 500g".
4. No multi-field, fuzzy or faceted search without increasingly painful SQL.

PostgreSQL's built-in full-text search (tsvector, GIN indexes) solves the first two well and is a perfectly good choice for modest scale. Dedicated engines win when you need sophisticated relevance tuning, faceting over huge catalogs, horizontal scale, or log analytics.

How an Inverted Index Works

Take three documents:

After lowercasing and splitting on whitespace, the inverted index is:

Query "butter recipe": look up both postings lists and combine. For an OR query, union {D1, D2} ∪ {D1, D2}. For AND, intersect sorted lists with a merge, O(m + n) in list lengths, and Lucene speeds this further with skip lists. Positions enable phrase queries ("butter chicken" requires chicken at position butter + 1, true in D2 only). The term dictionary itself is stored compactly (Lucene uses an FST, a finite-state transducer) so prefix and fuzzy lookups are fast.

from collections import defaultdict
import re

class InvertedIndex:
    def __init__(self):
        self.postings = defaultdict(dict)   # term -> {doc_id: [positions]}
        self.doc_len = {}

    @staticmethod
    def analyze(text):
        return re.findall(r"[a-z0-9]+", text.lower())

    def add(self, doc_id, text):
        tokens = self.analyze(text)
        self.doc_len[doc_id] = len(tokens)
        for pos, tok in enumerate(tokens):
            self.postings[tok].setdefault(doc_id, []).append(pos)

    def search_and(self, query):
        sets = [set(self.postings.get(t, {})) for t in self.analyze(query)]
        return set.intersection(*sets) if sets else set()

    def search_or(self, query):
        out = set()
        for t in self.analyze(query):
            out |= set(self.postings.get(t, {}))
        return out

    def search_phrase(self, phrase):
        terms = self.analyze(phrase)
        hits = set()
        for doc in self.search_and(phrase):
            for p in self.postings[terms[0]][doc]:
                if all(p + i in self.postings[t][doc] for i, t in enumerate(terms)):
                    hits.add(doc)
                    break
        return hits

idx = InvertedIndex()
idx.add("D1", "Paneer Butter Masala Recipe")
idx.add("D2", "Butter Chicken Recipe with Butter Naan")
idx.add("D3", "Masala Dosa")
print(idx.search_and("butter recipe"))      # {'D1', 'D2'}
print(idx.search_phrase("butter chicken"))  # {'D2'}

Analyzers: Tokenization, Normalization and Stemming

An analyzer is a three-stage pipeline applied to every text field at index time and to the query at search time:

1. Character filters clean raw text: strip HTML, map characters (e.g. "&" to "and").
2. Tokenizer splits into tokens. The standard tokenizer follows Unicode word-boundary rules; others split on whitespace, produce n-grams, or handle paths and emails.
3. Token filters transform tokens: lowercase, ASCII folding (café → cafe), stop words, stemming (running → run), synonyms (mobile ⇄ phone ⇄ smartphone), edge n-grams for prefix matching.

PUT /products
{
  "settings": {
    "analysis": {
      "filter": {
        "food_synonyms": {
          "type": "synonym_graph",
          "synonyms": ["biryani, biriyani, briyani", "curd, dahi, yogurt", "mobile, phone, smartphone"]
        },
        "english_stem": { "type": "stemmer", "language": "english" }
      },
      "analyzer": {
        "product_text": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "english_stem"]
        },
        "product_search": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "food_synonyms", "english_stem"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":    { "type": "text", "analyzer": "product_text", "search_analyzer": "product_search",
                    "fields": { "raw": { "type": "keyword" } } },
      "brand":    { "type": "keyword" },
      "category": { "type": "keyword" },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "in_stock": { "type": "boolean" },
      "updated_at": { "type": "date" }
    }
  }
}

Applying synonyms only at search time (via search_analyzer) means you can update the synonym list without reindexing, which is why it is a common pattern. Use POST /products/_analyze to see exactly which tokens a string produces; most "why doesn't this match?" bugs are analysis mismatches.

BM25 Relevance Scoring, Explained with Real Numbers

Since Elasticsearch 5.0, the default similarity is BM25. For a query Q with terms q and a document D:

score(D, Q) = Σ over q in Q of  IDF(q) × ( f(q,D) × (k1 + 1) )
                                       / ( f(q,D) + k1 × (1 − b + b × |D| / avgdl) )

IDF(q) = ln( 1 + (N − n(q) + 0.5) / (n(q) + 0.5) )

f(q,D) = frequency of q in D        |D| = length of D in terms
avgdl  = average document length    N   = number of documents
n(q)   = documents containing q     k1 = 1.2, b = 0.75 (defaults)

Worked example: query "butter recipe" over D1, D2, D3

N = 3. Lengths: D1 = 4, D2 = 6, D3 = 2, so avgdl = 4.

• "butter" appears in 2 documents: IDF = ln(1 + (3 − 2 + 0.5) / (2 + 0.5)) = ln(1.6) ≈ 0.470. Same for "recipe".
• Compare "dosa", in 1 document: IDF = ln(1 + 2.5 / 1.5) ≈ 0.981. Rarer terms carry roughly twice the weight here.

Two lessons hide in this table. First, term-frequency saturation: D2 contains "butter" twice, but its TF component rose only from 1.0 to 1.2, not to 2.0 as raw TF-IDF would give. That blunts keyword stuffing. Second, length normalization: D2 is 1.5× the average length, so its single "recipe" scores 0.83 instead of 1.0. The net effect is D2 barely edges out D1, which matches intuition. Tuning: raise k1 to reward repetition more; lower b toward 0 for fields where length should not matter (short titles).

Use "explain": true on a query to see this breakdown for real documents. In practice, product search layers business signals on top of BM25 with function_score (popularity, margin, in-stock boost, recency), and increasingly blends in vector similarity (hybrid search). For the semantic side, see Vector Databases Compared.

Mappings and Queries: text vs keyword, Query vs Filter

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "multi_match": {
            "query": "amul buter",
            "fields": ["title^3", "brand^2", "description"],
            "fuzziness": "AUTO"
        } }
      ],
      "filter": [
        { "term":  { "in_stock": true } },
        { "range": { "price": { "lte": 500 } } },
        { "terms": { "category": ["dairy", "bakery"] } }
      ]
    }
  },
  "aggs": { "brands": { "terms": { "field": "brand", "size": 10 } } },
  "size": 20
}

Clauses in filter do not affect score and are cacheable, so put every yes/no constraint there. A classic bug is running a term query against a text field: {"term": {"title": "Amul Butter"}} finds nothing because the indexed tokens are "amul" and "butter".

Pagination

from + size is fine for the first few pages but each shard must collect from + size hits, so deep pages get expensive and are capped by index.max_result_window (10,000 by default). For deep scrolling use search_after with a sort tiebreaker, plus a point-in-time (PIT) for a consistent view.

Shards, Replicas and How a Search Executes

• Primary shards partition the data; their count is fixed at index creation because routing uses hash % num_primaries. Changing it means reindexing (or the split/shrink APIs with constraints).
• Replicas provide failover and extra read throughput. A replica is never placed on the same node as its primary.
• Scoring is per shard by default: IDF is computed from each shard's local statistics. With few documents per shard, scores can look inconsistent; at scale it evens out.
• Custom routing (e.g. by tenant_id) sends all of a tenant's documents to one shard, so tenant queries hit one shard instead of all. Watch for large tenants creating hot shards.

Near Real-Time: Segments, Refresh, Translog, Flush and Merge

Each shard is a Lucene index made of immutable segments. Immutability makes segments cache-friendly and lock-free to read, but means updates and deletes work differently:

1. Index request arrives. The document goes into an in-memory indexing buffer and is appended to the translog (a write-ahead log), which by default is fsynced before the request is acknowledged.
2. Refresh (default every 1 s on indexes that received a search recently). The buffer is written as a new small segment to the filesystem cache and opened for search. Now the document is visible. This is why Elasticsearch is "near real-time".
3. Flush. Periodically segments are fsynced to disk via a Lucene commit and the translog is trimmed. On a crash, recovery replays the translog since the last commit.
4. Merge. Background merges combine small segments into larger ones, physically dropping deleted documents.
5. Updates are delete + reindex: the old version is marked deleted in a bitset and the new version is written to a new segment.

If a user saves a product and the next page load must show it in search, either call the index API with refresh=wait_for for that request (blocks until the next refresh) or read that single item from the database. Do not force refresh=true on every write; it creates tiny segments and hurts throughput.

Syncing Elasticsearch from Your Database

Elasticsearch should almost never be your source of truth. It lacks multi-document transactions, and mapping changes often require a full reindex. So you need a pipeline from the database. The options, worst to best:

Rules that make the pipeline correct:

• Use the primary key as _id so replays are idempotent upserts.
• Partition the Kafka topic by primary key so updates to one row stay ordered.
• Use external versioning (version_type=external with the row's version or the log position) so a late, older event cannot overwrite a newer one.
• Denormalize in the indexer. Search documents usually combine several tables (product + brand + stock + rating). Joins happen in the pipeline, not at query time.
• Reindex with aliases. Build products_v8 from a snapshot plus the CDC stream, then atomically swap the products alias. Zero downtime mapping changes.

For choosing the log in the middle, see Kafka vs RabbitMQ vs SQS.

Scaling Pitfalls and Operational Lessons

• Too many small shards. Every shard costs heap and cluster-state overhead. Size shards by data (commonly 10–50 GB guidance from Elastic) instead of defaulting to many.
• Mapping explosion. Dynamic mapping on arbitrary user JSON creates thousands of fields. Set dynamic: strict or use the flattened type.
• Heap sizing. Give the JVM at most about half the RAM and keep it below the compressed-oops threshold (~31 GB); the rest serves the OS filesystem cache, which Lucene depends on.
• Time-series data (logs, metrics): use rollover indexes or data streams with lifecycle policies (hot → warm → cold → delete) rather than one ever-growing index. This is the core of Design a Distributed Logging System.
• Expensive queries: leading wildcards, deep pagination, huge terms aggregations on high-cardinality fields, and scripts. Guard with timeouts and query review.
• Split brain was a classic issue in old versions with misconfigured quorum; since 7.x the cluster coordination layer manages master election quorum automatically, but you still want 3 dedicated master-eligible nodes in production.

How Search Shows Up in System Design Interviews

Search appears as a component in many designs: product search for Flipkart, restaurant search for Swiggy, log search for an observability platform, message search for Slack. Interviewers want to see that you:

1. Separate the search index from the source of truth and explain the sync path (CDC/outbox) and its lag.
2. Know why the inverted index is fast and can sketch it.
3. Talk about relevance beyond "Elasticsearch handles it": BM25 basics, field boosts, business signals, synonyms from query logs, and measuring quality (click-through rate, zero-result rate, NDCG offline).
4. Size and shard: documents × size → total, divided by target shard size → shard count; replicas for read QPS and availability.
5. Handle freshness: price and stock change constantly; index them on a fast path or filter them at serve time from a cache.

Typical follow-up questions

• "How do you implement autocomplete?" Edge n-gram analyzers or the completion suggester, or a dedicated trie service for very high QPS; see Design Search Autocomplete.
• "A product's stock just went to zero. How fast does search reflect it?" One refresh interval plus pipeline lag; for hard guarantees, filter at serve time.
• "How do you change an analyzer on a live index of 100M documents?" New index with new mapping, backfill, catch up via CDC, alias swap.
• "How would you search across 50 languages?" Per-language fields or indexes with language-specific analyzers, plus language detection on query.
• "Elasticsearch or PostgreSQL full-text?" PostgreSQL if data is moderate and you want one system; Elasticsearch when relevance tuning, faceting and scale matter. Choosing a primary database is covered in PostgreSQL vs MySQL vs MongoDB vs Cassandra.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.