Vector Databases

What it is

A vector database stores embeddings — dense numerical vectors (typically 384–3072 dimensions) that encode the semantic meaning of text, images, audio, or structured data, generated by neural network models (OpenAI, Cohere, Sentence-Transformers, etc.). The core operation is Approximate Nearest Neighbor (ANN) search: given a query embedding, find the K most similar stored embeddings by distance metric (cosine similarity, dot product, or Euclidean distance) in milliseconds, even across millions or billions of vectors.

Dedicated vector databases: Pinecone (managed, serverless, production-grade); Weaviate (open-source, multi-modal, GraphQL API); Qdrant (open-source, Rust-based, fast filtering); Milvus (open-source, Kubernetes-native, ANNS + metadata filtering). Vector extensions for existing databases: pgvector (PostgreSQL extension — HNSW and IVF indexes); Redis Vector (Redis module); OpenSearch kNN plugin; Elasticsearch dense_vector. Managed cloud services: Amazon OpenSearch Service, Azure AI Search, Google Vertex AI Vector Search.

The two dominant ANN index structures are HNSW (Hierarchical Navigable Small World) — a graph-based index with high recall, fast queries, and slow build time — and IVF (Inverted File Index) — a cluster-based quantization index with faster build time, less memory, and slightly lower recall. Both offer a recall vs. speed tradeoff controlled by tunable parameters.

Why it exists

Semantic similarity is not expressible as a relational query. A SQL WHERE text LIKE '%machine learning%' misses documents about "neural networks" or "AI" even though they're semantically related. Embedding models encode meaning in geometric space: semantically similar concepts cluster near each other. The nearest-neighbor search over embeddings retrieves semantically relevant results regardless of exact keyword overlap. Exact nearest-neighbor search over millions of 1536-dimensional vectors requires billions of distance computations — intractable in milliseconds. ANN algorithms trade a small, tunable amount of recall for dramatically faster search: HNSW returns top-10 from 10 million vectors in under 10ms with 95%+ recall.

When to use

• Retrieval-Augmented Generation (RAG): Retrieve semantically relevant document chunks from a knowledge base to include as context in LLM prompts — the standard architecture for grounding AI responses in factual documents.
• Semantic search: Find content matching the intent of a natural language query rather than exact keyword matches — product search, document search, Q&A systems.
• Recommendation systems: Find items similar to a user's interaction history or profile by embedding both users and items in the same vector space.
• Duplicate detection: Find near-duplicate images, code snippets, or documents by embedding and searching for high-similarity neighbors.
• Anomaly detection: Identify data points with no close neighbors in embedding space as potential anomalies or outliers.

When not to use

• Don't use vector search for exact keyword lookup — a standard full-text search engine (Elasticsearch, PostgreSQL FTS) handles exact keyword matching more efficiently and with more precise recall.
• Don't use a dedicated vector database if your vector dataset is small (<1M vectors) — pgvector on PostgreSQL provides ANN search with acceptable performance without additional infrastructure.
• Don't expect vector search alone to solve all semantic search problems — hybrid search (combining vector similarity with BM25 keyword relevance) consistently outperforms either approach alone.
• Don't assume vector search returns "correct" results — ANN is approximate; there's no guarantee that the semantically most relevant result is always in the top-K returned; tune recall thresholds and test with domain-specific evaluation sets.

Typical architecture

  RAG (Retrieval-Augmented Generation) with Vector Database:

  Ingestion pipeline:
  Raw documents ──▶ Chunking (512-1024 token windows with overlap)
                ──▶ Embedding model (OpenAI text-embedding-3-small)
                ──▶ Store in vector DB (chunk text + embedding + metadata)

  Query pipeline:
  User question ──▶ Embedding model ──▶ Query vector
                ──▶ ANN search in vector DB (top-K chunks by cosine similarity)
                ──▶ Optional: metadata filter (date > X, source = Y)
                ──▶ Retrieved chunks + original question
                ──▶ LLM prompt: "Answer based on context: [chunks]\nQuestion: [q]"
                ──▶ Response

  pgvector (PostgreSQL):
  CREATE EXTENSION vector;
  CREATE TABLE documents (
    id      BIGSERIAL PRIMARY KEY,
    content TEXT,
    source  TEXT,
    embedding VECTOR(1536)      -- OpenAI text-embedding-3-small dimension
  );
  CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

  -- ANN search:
  SELECT content, 1 - (embedding <=> $1) AS similarity
  FROM documents
  ORDER BY embedding <=> $1  -- cosine distance operator
  LIMIT 10;
          

Pros and cons

Implementation notes

For pgvector's HNSW index, tune m (number of bi-directional links per node, default 16) and ef_construction (candidate list size during index build, default 64) for recall vs. memory tradeoff. Higher m values (32–64) improve recall at the cost of more memory. At query time, set SET hnsw.ef_search = 100 (default 40) to increase candidate exploration for better recall on important queries. For IVF_FLAT indexes in pgvector, set lists to approximately sqrt(N) where N is the number of rows, and probes to 10–20% of lists at query time. Always benchmark recall on your specific data and query distribution — synthetic benchmarks don't represent domain performance.

For production RAG systems, implement hybrid search (vector + BM25 keyword): retrieve candidate documents using both methods, then combine scores using Reciprocal Rank Fusion (RRF) or a learned re-ranker. This handles cases where exact keywords are more discriminative than semantic similarity (specific error codes, product model numbers, names). Implement embedding caching: identical text always produces identical embeddings from a deterministic model; cache query embeddings in Redis with the query text as key to avoid redundant API calls to the embedding service.

Common failure modes

• Stale embeddings: Source documents are updated but their stored embeddings aren't re-computed; the vector index returns chunks from outdated document versions; AI responses cite superseded information.
• No recall evaluation: A team deploys a vector search system and assumes high recall; unmeasured recall is actually 70% on their domain; 30% of relevant results are never surfaced; users report poor search quality but can't articulate why.
• Chunking too small: Documents are chunked into single sentences to maximize semantic precision; each chunk lacks sufficient context for the LLM to answer; retrieved chunks are semantically relevant but individually uninterpretable without surrounding context.
• HNSW memory exhaustion: A large corpus with 10M vectors and a high-m HNSW index requires 80GB of RAM; the vector database server runs out of memory and crashes; no capacity planning was done before ingestion.
• Wrong distance metric: Embeddings from an OpenAI model are stored with Euclidean distance; the model's embedding space is optimized for cosine similarity; search results are suboptimal; the index must be rebuilt with the correct metric.

Decision checklist

• Is the use case semantic search, RAG, or recommendation — where meaning-based similarity outperforms keyword matching?
• Is the corpus size large enough to justify a dedicated vector database, or can pgvector on existing PostgreSQL serve the need?
• Is there a recall evaluation pipeline using domain-specific labeled queries to measure actual retrieval quality?
• Is there an embedding update pipeline that re-indexes documents when source content changes?
• Is the distance metric (cosine, dot product, Euclidean) matched to what the embedding model was trained for?
• Has memory capacity been estimated: (vector_count × dimensions × 4 bytes) × 3–4x for HNSW overhead?

Example use cases

• Enterprise knowledge base RAG: 50,000 internal documents (policies, runbooks, product specs) embedded with text-embedding-3-small. Support engineers ask natural language questions; the system retrieves the 5 most semantically relevant document chunks and passes them to GPT-4 to generate a grounded answer with source citations.
• E-commerce semantic product search: Product descriptions and images embedded and stored in Weaviate. A search for "comfortable shoes for long walks" retrieves walking shoes, orthopedic insoles, and hiking boots — not just exact matches for "shoes". Boosted with BM25 for brand name exact matches.
• Code similarity and duplicate detection: Code functions embedded with a code-specific model (CodeBERT). New code submissions are checked against the corpus for semantic similarity — detecting plagiarism, copy-pasted code with variable renames, or functionally equivalent implementations in different syntax.

Related patterns

• Graph Databases — Another specialized database for non-relational data; graph for relationships, vectors for semantic similarity.
• Data Cataloging — Vector search is used in modern data catalogs for semantic discovery of datasets and columns by description.
• AI & ML Systems — Vector databases are the retrieval layer in AI pipelines; LLM integration patterns build on top of this foundation.

Further reading

• pgvector — PostgreSQL Extension for Vector Similarity Search
• HNSW Algorithm Explained — Pinecone
• Chroma Vector Database Documentation