Search & Retrieval Essential

Full-Text Search

How inverted indexes, BM25 scoring, and analyzer pipelines power fast, relevant document retrieval at scale.

⏱ 10 min read

What it is

Full-text search is a technique for querying large collections of natural-language documents by matching terms against a pre-built inverted index — a data structure that maps every unique term to the list of documents (and positions) that contain it. Unlike a database LIKE '%term%' scan, which must read every row, an inverted index lets the engine jump directly to the relevant document set in O(log n) time, making sub-second search over millions of documents feasible.

The core relevance model used by modern engines (Elasticsearch, OpenSearch, Solr) is BM25 (Best Match 25), a probabilistic ranking function that improves on classic TF-IDF by saturating term frequency and normalising for document length. BM25 scores a document by combining: how often the query term appears in the document (term frequency, dampened), how rare the term is across the corpus (inverse document frequency), and the document length relative to the average. Tuning the k1 (TF saturation) and b (length normalisation) parameters is the first step in relevance engineering.

Why it exists

Relational databases were designed for structured data and exact-match queries. When users type natural language — with synonyms, typos, stemming variants, and multi-word phrases — SQL's pattern matching becomes slow and semantically blind. As soon as a product catalogue grows past a few thousand items, or a document corpus reaches tens of thousands, users expect Google-like responsiveness and relevance. The inverted index was purpose-built for this problem: index once, retrieve instantly.

Beyond speed, full-text search engines provide an analysis pipeline that normalises text at index and query time, enabling language-aware features impossible in a general-purpose database: phonetic matching, synonym expansion, stemming ("running" → "run"), stop-word removal, and character folding (accents, case). These transformations decouple what a user types from the exact tokens stored, dramatically improving recall without sacrificing precision.

When to use

Product catalogues and e-commerce search where users type natural-language queries against titles, descriptions, and attributes.
Documentation portals and knowledge bases where employees or customers search across thousands of articles.
Log and event search (ELK stack) where engineers need to find specific error messages across billions of log lines.
News, content, and publishing platforms where freshness, phrase matching, and field-level boosting matter.
Enterprise search aggregating content from multiple heterogeneous sources (wikis, tickets, code repos).
Legal, medical, or scientific document retrieval where precision on multi-word phrases is critical.

When not to use

Exact-match lookups: If users are querying by exact ID, SKU, or enum value, a relational index is faster and simpler.
Very small corpora: Fewer than ~1,000 documents rarely justify the operational overhead of running a search cluster.
Highly structured filtering only: If queries are purely numeric range filters or category filters with no text, a database or columnar store is sufficient.
Semantic intent without keywords: When users express intent rather than keywords ("best laptop for video editing"), pure keyword search produces poor results — consider adding semantic search or hybrid search.

Typical architecture

┌─────────────────────────────────────────────────────────┐
│                    INDEXING PIPELINE                    │
│                                                         │
│  Source DB / API                                        │
│       │                                                 │
│       ▼                                                 │
│  ETL / Connector  ──► Character Filter (HTML strip)     │
│                             │                           │
│                             ▼                           │
│                       Tokenizer (standard/whitespace)   │
│                             │                           │
│                             ▼                           │
│                       Token Filters (lowercase,         │
│                         stop, stemmer, synonyms)        │
│                             │                           │
│                             ▼                           │
│                   Inverted Index (shards)               │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│                     QUERY PIPELINE                      │
│                                                         │
│  User Query ──► Same analyzer ──► Query Terms           │
│                                        │                │
│                                        ▼                │
│                              Inverted Index Lookup      │
│                                        │                │
│                                        ▼                │
│                              BM25 Score per doc         │
│                                        │                │
│                                        ▼                │
│                              Top-K results returned     │
└─────────────────────────────────────────────────────────┘

Elasticsearch Cluster:
  ┌──────────┐  ┌──────────┐  ┌──────────┐
  │ Node 1   │  │ Node 2   │  │ Node 3   │
  │ P0  R1   │  │ P1  R0   │  │ P2  R1   │
  │ R2       │  │ R2       │  │ P0       │
  └──────────┘  └──────────┘  └──────────┘
  P=primary shard, R=replica shard

Pros and cons

Pros

Sub-second query latency over millions of documents with proper indexing.
Rich language analysis: stemming, synonyms, stop words, phonetics, and multilingual support.
Field-level boosting allows tuning which fields (title, body, tags) matter most.
Horizontal scaling via sharding; add nodes to increase throughput or storage.
Faceted aggregations, highlighting, and autocomplete built into the same engine.

Cons

Operational overhead of managing a search cluster (Elasticsearch/OpenSearch) in addition to the primary database.
Index synchronisation lag — data written to the primary store takes time to propagate to the search index.
BM25 relevance breaks down for semantic queries; does not understand meaning or intent.
Deep pagination (from/size) is expensive; scroll or search_after APIs required for large result sets.
Mapping explosions and unbounded keyword fields can cause heap pressure and cluster instability.

Implementation notes

The single most impactful design decision is analyzer selection. Use the built-in english analyzer (or equivalent per language) for free-text fields — it applies lowercase, stop-word removal, and the Porter stemmer. For product names, SKUs, and IDs, use the keyword type (no analysis) or the standard analyzer with case folding only. Define custom analyzers explicitly in the index mapping rather than relying on dynamic mapping, which often produces sub-optimal field types. At query time, always use the same analyzer as at index time, or explicitly set search_analyzer in the mapping.

For relevance tuning, start by auditing which fields exist in your documents and assigning boost weights: title fields typically get 3–5×, category tags 2×, and body text 1×. Use a multi-match query with type: best_fields or cross_fields depending on whether each field stands independently or all fields together describe a single concept. Instrument click-through rates and zero-result rates from the beginning — you cannot tune relevance without feedback signals. Tools like Quepid let you manage judgment lists (rated query-document pairs) and compute NDCG scores to measure relevance objectively across releases.

Common failure modes

Analyzer mismatch: Indexing with one analyzer and querying with another produces zero results for perfectly matching terms (e.g., stemmed index vs. keyword query).
Mapping explosion: Dynamic mapping on JSON fields with high-cardinality keys creates thousands of field mappings, causing heap exhaustion and cluster instability.
Split-brain / cluster instability: Misconfigured discovery.zen.minimum_master_nodes (pre-7.x) or quorum settings allow two master nodes, corrupting the cluster state.
Deep pagination performance: Using from: 50000 forces the engine to collect and discard 50,000 results on every shard before returning the page, causing high GC pressure.
Index synchronisation drift: Without a transactional outbox or dual-write strategy, the search index silently falls behind the primary store after failures.

Decision checklist

Have you chosen per-field analyzers (language-specific for text, keyword for IDs)?
Have you defined explicit mappings rather than relying on dynamic mapping?
Is your index synchronisation strategy fault-tolerant (outbox pattern, CDC)?
Have you benchmarked query latency under expected peak QPS?
Do you have a relevance baseline (judgment list + NDCG score) before tuning?
Have you defined an index lifecycle policy (ILM) for retention and rollover?
Is deep pagination handled via search_after instead of from/size?

Example use cases

E-commerce product search: Multi-field BM25 over product title, brand, description, and category with synonym expansion for "TV" → "television", boosted by sales rank.
Developer documentation portal: Phrase-match queries over code snippets, headings, and body text; highlighted matching terms in result snippets.
Security log analysis: Free-text search over structured log events ingested via Logstash; field-specific queries like error.code:403 AND source.ip:192.168.*.

Faceted Search — Adding aggregation-based filters on top of full-text results.
Semantic Search — Vector-based retrieval that understands meaning, complementing keyword matching.
Hybrid Search — Combining BM25 scores with vector similarity for best-of-both-worlds relevance.
Search Infrastructure — Cluster sizing, shard strategy, and ILM for production deployments.