Search & Retrieval AI/ML

Semantic Search

Using dense vector embeddings and approximate nearest-neighbour indexes to retrieve documents by meaning rather than keyword overlap.

⏱ 11 min read

What it is

Semantic search retrieves documents based on the meaning of a query rather than its exact keywords. An embedding model (such as a sentence-transformer or OpenAI's text-embedding API) converts both the query and each document into a dense floating-point vector, typically 384 to 3,072 dimensions. Semantically similar texts produce vectors that are close in high-dimensional space, measured by cosine similarity (or dot product for normalised vectors). Retrieving the k most similar documents is an Approximate Nearest Neighbour (ANN) search — finding the k closest vectors without exhaustively comparing every document.

The primary ANN index algorithm in production today is HNSW (Hierarchical Navigable Small World), a graph-based structure that allows logarithmic-time approximate neighbour lookup. Elasticsearch (from v8.0), pgvector, Pinecone, Weaviate, Qdrant, and Milvus all offer HNSW-based ANN indexes. The critical distinction in architecture is between bi-encoders — models that encode query and document independently and compare vectors — and cross-encoders — models that take a (query, document) pair and score relevance jointly. Bi-encoders scale to millions of documents (pre-encode offline); cross-encoders are slower but more accurate, used only for re-ranking a small candidate set.

Why it exists

BM25 and inverted index search suffer from the vocabulary mismatch problem: a document about "myocardial infarction" will not match a query for "heart attack" unless synonyms are manually configured. Semantic search solves this because both phrases map to nearby points in the embedding space trained on vast text corpora. This makes it invaluable for long-tail queries, paraphrastic retrieval, cross-lingual search (multilingual embedding models), and conversational AI pipelines such as RAG (Retrieval-Augmented Generation).

The rise of large language models provided both the embedding models (trained on internet-scale text) and the downstream use case (RAG). Vector databases and ANN indexes were engineered to serve the retrieval step in a RAG pipeline at production scale — handling millions of document chunks, refreshing embeddings as content changes, and maintaining sub-100ms ANN query latency under high concurrency. This ecosystem matured rapidly between 2022 and 2025, making vector search a standard component of the search architecture toolkit.

When to use

Question-answering over a knowledge base where users phrase questions in natural language rather than keywords.
RAG pipelines for LLM-based chatbots that need to ground answers in a specific document corpus.
Similar item / "more like this" recommendations where semantic similarity defines relatedness.
Cross-lingual search where users query in one language and documents are in another.
Long-tail queries with rare or specialised vocabulary where synonym lists and BM25 tuning provide insufficient coverage.
Code search where a natural-language description should match semantically related code snippets.

When not to use

Exact keyword matching requirements: Semantic search may miss exact product codes, serial numbers, or legal citations that BM25 would find trivially.
Freshness-critical content: Re-embedding new documents takes time; for real-time news search, BM25 indexing is far faster and simpler.
Very small corpora: Under ~1,000 documents, brute-force cosine similarity is fast enough and requires no vector index infrastructure.
Budget-constrained systems: Embedding generation (especially via API) and vector index infrastructure have significant cost at scale; profile ROI against BM25 accuracy first.

Typical architecture

INDEXING PIPELINE:
  Documents
     │
     ▼
  Chunking (512–1024 tokens per chunk)
     │
     ▼
  Embedding Model (bi-encoder)
  e.g. sentence-transformers/all-MiniLM-L6-v2
  or   text-embedding-3-small (OpenAI API)
     │
     ▼
  Dense vectors (384 or 1536 dims)
     │
     ▼
  HNSW Index (Elasticsearch dense_vector,
              pgvector, Pinecone, Qdrant...)

QUERY PIPELINE:
  User Query: "heart attack symptoms"
     │
     ▼
  Same Embedding Model → query vector
     │
     ▼
  ANN Search (knn, top-K=100)
  (HNSW graph traversal, ~1-5ms)
     │
     ▼
  Optional: Cross-encoder reranker
  (top-100 candidates → top-10 reranked)
     │
     ▼
  Results returned to user / LLM

BI-ENCODER vs CROSS-ENCODER:
  Bi-encoder: fast, scales to millions
              encode once, compare vectors
  Cross-encoder: slow (full attention over
              query+doc pair), used for
              reranking top-K candidates only

Pros and cons

Pros

Handles vocabulary mismatch and paraphrastic queries that BM25 misses entirely.
Enables cross-lingual retrieval with multilingual embedding models (e.g., multilingual-e5).
Core enabler for RAG and LLM-grounded question answering over private documents.
HNSW delivers sub-10ms ANN latency even over tens of millions of vectors.
Embeddings are reusable across tasks — one embedding index can serve search, recommendations, and clustering.

Cons

Embedding generation cost: re-encoding millions of documents takes hours of GPU/API time and must be repeated when the model is updated.
Hallucination of relevance: embeddings can rank thematically related but factually wrong documents highly, especially with generic models.
Exact keyword matching is worse than BM25 — a product code "XK-9021" may not match semantically.
HNSW index is memory-resident; at 1536 dimensions × 4 bytes × 10M vectors = ~58GB RAM required.
Model selection and versioning complexity: changing the embedding model invalidates the entire index.

Implementation notes

Choose your embedding model based on the trade-off between quality and cost. Open-source bi-encoders from the sentence-transformers library (e.g., all-MiniLM-L6-v2 at 384 dims, or e5-large-v2 at 1024 dims) run on CPU or GPU in-process and avoid per-call API costs. OpenAI's text-embedding-3-small (1536 dims) and text-embedding-3-large (3072 dims) offer strong quality at ~$0.02/1M tokens but require chunked batching for large corpora. Match the model used at index time exactly at query time — even a minor version change produces incompatible embeddings that silently degrade recall.

For HNSW tuning, the two key parameters are m (number of edges per node, typically 16–64) and ef_construction (search depth during index build, typically 100–400). Higher values improve recall but increase memory and build time. At query time, ef_search controls the recall-vs-latency trade-off. A recall@10 of 0.95 at ef_search=100 is typical for well-tuned HNSW. Implement a two-stage retrieval pipeline: retrieve top-100 candidates with ANN (fast), then re-rank with a cross-encoder (accurate) to produce the final top-10. This hybrid approach achieves near cross-encoder accuracy at bi-encoder speed.

Common failure modes

Model version mismatch: Indexing with v1 of a model and querying with v2 produces garbage results because the embedding spaces are different.
Chunking strategy mismatch: Chunks that are too short (single sentence) lose context; chunks that are too long (full document) dilute the signal of specific passages.
Memory underprovisioning: HNSW indexes must fit entirely in RAM; an index that exceeds available memory causes severe latency spikes due to disk swapping.
Embedding drift: As the document corpus grows, the distribution of new documents may shift away from the original embedding space, degrading recall for new-topic queries.
No pre-filtering: Running ANN across a multi-tenant index without filtering by tenant returns results from all tenants, leaking data; always apply metadata pre-filters before ANN.

Decision checklist

Have you selected and locked an embedding model version for consistent index/query compatibility?
Have you defined a chunking strategy (size, overlap) appropriate for your document length?
Have you sized the vector index RAM for your expected corpus size and dimensions?
Is there a re-embedding pipeline for when the model is updated?
Are multi-tenant or access-controlled corpora filtered before ANN search?
Have you evaluated whether hybrid search (BM25 + vector) is needed for exact-term queries?
Have you measured recall@K against a held-out evaluation set?

Example use cases

RAG chatbot for internal documentation: Employee queries in natural language retrieve relevant policy documents and API docs as context for an LLM, grounding responses in authoritative sources.
Customer support ticket deduplication: New incoming tickets are embedded and ANN-searched against resolved tickets; if a semantically similar resolved ticket exists, it is surfaced to the agent automatically.
Academic paper discovery: Researchers describe a research gap in natural language; the system retrieves the 20 most semantically relevant papers across a corpus of millions.

Hybrid Search — Combining semantic vector scores with BM25 keyword scores for best-of-both relevance.
Full-Text Search — Keyword baseline that semantic search complements for exact-term queries.
Vector Databases — Infrastructure layer purpose-built for dense vector storage and ANN retrieval.