Vector Databases

What it is

A vector database is a data storage and retrieval system purpose-built for high-dimensional dense vectors — the numerical representations (embeddings) produced by machine learning models when they encode text, images, audio, or other unstructured data. Unlike relational or document databases that retrieve records by exact key or structured query, a vector database retrieves records by semantic similarity: given a query vector, it finds the stored vectors that are geometrically nearest to it in the embedding space, corresponding to the most semantically similar items.

The core operation is Approximate Nearest Neighbor (ANN) search. Exact nearest neighbor search in high-dimensional space requires comparing the query vector against every stored vector — an O(n) operation that becomes prohibitively slow at millions or billions of records. ANN algorithms trade a small amount of recall accuracy (missing some true nearest neighbors) for orders-of-magnitude faster retrieval. The dominant ANN index algorithms are HNSW (Hierarchical Navigable Small World graphs) — which builds a multi-layer graph structure for fast logarithmic search — and IVF (Inverted File Index) — which clusters vectors into Voronoi cells and searches only nearby clusters. Most production vector databases support both, and the choice affects the trade-off between memory usage, build time, query latency, and recall accuracy.

Vector databases also provide metadata filtering — allowing searches to be scoped to a subset of vectors based on structured attributes (e.g., "find vectors similar to this query, but only from documents published after 2023 with category = 'technology'"). This hybrid filtering requirement is non-trivial: naive pre-filtering reduces the candidate set before ANN search (potentially missing results), while post-filtering reduces recall by discarding results after ANN search. Leading vector databases implement pre-filtering with payload-aware indexes that maintain recall.

Why it exists

Traditional databases are optimized for exact lookups and structured queries — finding a row by primary key, filtering by equality, sorting by a scalar value. These operations map poorly onto the semantics of ML embeddings. An embedding for "automobile" and an embedding for "car" should be close neighbors, but there is no SQL query that can express this without precomputing and storing their similarity explicitly. As ML-powered applications that work with unstructured data became mainstream, the need for infrastructure that could efficiently serve nearest-neighbor queries over embeddings at scale became clear.

The explosive growth of RAG architectures and LLM-powered applications dramatically accelerated adoption of vector databases. Every RAG system needs to store document embeddings and retrieve similar chunks for a given query; every semantic search application needs to find similar items to a user's query; every recommendation system can be expressed as finding items whose embeddings are nearest to a user's embedding. The volume of embeddings to store (billions in large-scale systems) and the latency requirements (under 10ms for online serving) made existing approaches like database extensions inadequate for many use cases.

Beyond purpose-built vector databases, many existing databases have added vector search extensions: pgvector for PostgreSQL, Elasticsearch's dense vector search, Redis Stack, and MongoDB Atlas Vector Search. These "vector-capable" databases are appropriate for smaller scales and teams that prefer to consolidate infrastructure. Purpose-built vector databases (Pinecone, Qdrant, Weaviate, Chroma, Milvus) offer better performance, richer ANN index tuning, and more sophisticated filtering at very large scale.

When to use

• RAG pipelines where document chunk embeddings must be indexed and retrieved per user query at low latency.
• Semantic search applications where results should be ranked by meaning rather than keyword overlap.
• Recommendation systems using collaborative or content-based embeddings — find items similar to the ones a user interacted with.
• Duplicate detection, near-duplicate detection, or deduplication over large unstructured datasets.
• Image, audio, or video similarity search where deep learning embeddings encode perceptual features.

When NOT to use

• Primary transactional data storage — vector databases are purpose-built for ANN search and should not replace relational or document databases for structured data.
• Small datasets (under 100,000 vectors) where the overhead of a dedicated vector database is not justified — pgvector or in-memory FAISS may be sufficient.
• Applications where exact match is required — vector databases return approximate nearest neighbors, not guaranteed exact results, which is incorrect for use cases that require strict determinism.

Typical architecture

 WRITE PATH (indexing)
 ┌──────────────────────────────────────────────────────────┐
 │  Documents / Items / Events                               │
 └────────────────────────┬─────────────────────────────────┘
                          │
                          ▼
 ┌──────────────────────────────────────────────────────────┐
 │  EMBEDDING MODEL                                          │
 │  text-embedding-ada-002 / CLIP / custom model             │
 │  item → float[768] or float[1536]                        │
 └────────────────────────┬─────────────────────────────────┘
                          │ (id, vector, metadata)
                          ▼
 ┌──────────────────────────────────────────────────────────┐
 │              VECTOR DATABASE                              │
 │                                                           │
 │  ┌─────────────────────────────────────────────────────┐ │
 │  │              ANN INDEX                               │ │
 │  │  HNSW:  Layer 2  o─────o─────o                      │ │
 │  │         Layer 1  o──o──o──o──o──o                   │ │
 │  │         Layer 0  o-o-o-o-o-o-o-o-o-o                │ │
 │  │                                                      │ │
 │  │  IVF:  [centroid 0]→[v1,v2,v3...]                   │ │
 │  │        [centroid 1]→[v4,v5,v6...]                   │ │
 │  │        [centroid 2]→[v7,v8,v9...]                   │ │
 │  └─────────────────────────────────────────────────────┘ │
 │                                                           │
 │  ┌──────────────────┐  ┌──────────────────────────────┐  │
 │  │ Metadata Store   │  │  Payload / Attribute Index   │  │
 │  │ id → JSON attrs  │  │  (for pre-filtering)         │  │
 │  └──────────────────┘  └──────────────────────────────┘  │
 └──────────────────────────────────────────────────────────┘

 READ PATH (query)
 ┌──────────────┐     ┌─────────────────┐     ┌────────────────────┐
 │  Query Text  │────▶│ Embed Query     │────▶│ ANN Search         │
 │  or Item ID  │     │ (same model)    │     │ top-K candidates   │
 └──────────────┘     └─────────────────┘     └─────────┬──────────┘
                                                         │ filter
                                                         ▼
                                              ┌────────────────────┐
                                              │ Metadata Filter    │
                                              │ (pre or post)      │
                                              └─────────┬──────────┘
                                                        │
                                                        ▼
                                              ┌────────────────────┐
                                              │ Results: top-K IDs │
                                              │ + similarity scores│
                                              │ + metadata         │
                                              └────────────────────┘

Pros and cons

Implementation notes

Calibrate HNSW parameters (M and ef_construction) to your recall and memory requirements. M controls the number of bidirectional edges per node in the graph — higher values improve recall but increase memory consumption and build time. ef_construction controls the size of the dynamic candidate list during index construction — higher values produce higher-quality graphs at the cost of longer build time. A common starting point is M=16, ef_construction=100; benchmark on your specific data distribution using recall@K metrics and adjust empirically. The query-time parameter ef_search can be tuned separately to trade per-query latency against recall.

Plan your vector dimensionality carefully. Higher-dimensional embeddings generally encode more semantic information but consume more memory and increase ANN search latency. Modern embedding models offer multiple dimensionality tiers — for example, 512, 1024, or 1536 dimensions. Evaluate whether a lower-dimensional model achieves sufficient recall for your task before defaulting to the highest-dimension option. Matryoshka Representation Learning (MRL) models (like OpenAI's text-embedding-3) allow truncating embeddings to lower dimensions without retraining, providing flexible dimensionality/performance trade-offs.

Design for index lifecycle management. Unlike relational databases where inserts are immediate and consistent, vector databases with HNSW indexes handle inserts incrementally, but rebuilding the entire index from scratch (for example when migrating to a new embedding model) is a major operation. Maintain a "shadow index" — a second index built on the new configuration — and swap traffic to it once validation passes, rather than attempting an in-place migration. Always maintain the raw embeddings or the original source data so that re-embedding is possible if you change models.

Common failure modes

• Embedding model mismatch: Query and document embeddings computed by different model versions are not comparable; results are meaningless. Enforce strict versioning of the embedding model used for both indexing and querying, and include the model version as a namespace or collection qualifier in the vector database.
• Memory exhaustion at scale: HNSW indexes for large datasets are significantly larger than the raw vector data due to graph edges; underestimating index memory requirements causes OOM failures in production. Size the index server with 2–3× the raw vector storage as a memory budget guideline.
• Recall degradation with heavy filtering: Aggressive metadata pre-filtering reduces the candidate pool so severely that ANN search has too few candidates to achieve the desired top-K recall. Use post-filtering or partition the index by filter dimension to maintain recall under heavy filtering workloads.
• Index corruption on unclean shutdown: HNSW indexes built in memory may not be fully persisted if the process is killed mid-operation. Use databases with Write-Ahead Logs or atomic snapshot mechanisms to ensure index durability across restarts.

Decision checklist

• What embedding model will be used, and is it the same model for both indexing time and query time?
• What vector dimensionality and index size is expected, and does the infrastructure have sufficient memory for the HNSW graph?
• What recall@K threshold is required for your application, and have you benchmarked HNSW parameters to meet it?
• Is metadata filtering required, and does the chosen database support payload-aware pre-filtering to maintain recall?
• Is a purpose-built vector database necessary, or does pgvector or a similar database extension meet your scale requirements?
• What is the re-indexing strategy when the embedding model changes — is the original source data retained for re-embedding?

Example use cases

• RAG document retrieval: A Qdrant collection stores 10 million document chunks from an enterprise knowledge base; each query embeds the user's question and retrieves the top-5 semantically similar chunks in under 5ms; chunks are filtered by department metadata to scope retrieval to authorized content.
• E-commerce visual similarity search: A Pinecone index stores CLIP embeddings for 50 million product images; users upload a photo to find visually similar products; results are filtered by price range and in-stock status using Pinecone's metadata filtering.
• Duplicate content detection: A Weaviate collection stores embeddings of news articles; a daily job embeds new articles and queries for near-duplicates with similarity above a threshold, flagging potential duplicate stories for editorial review without manual comparison of thousands of articles.

Related patterns

• RAG Architecture
• Feature Stores
• Search & Retrieval
• LLM Architecture

Further reading

• Qdrant Vector Database Documentation
• What is a Vector Database? – Pinecone
• Efficient and Robust Approximate Nearest Neighbor Search Using HNSW – Malkov & Yashunin
• FAISS: A Library for Efficient Similarity Search – Meta AI