Search & Retrieval

Search Infrastructure

Designing production-grade Elasticsearch clusters: shard strategy, Index Lifecycle Management, hot-warm-cold tiers, index aliases, and capacity planning.

⏱ 11 min read

What it is

Search infrastructure encompasses the cluster architecture, data node configuration, index design, and operational policies that make a search engine reliable and cost-efficient at scale. An Elasticsearch or OpenSearch cluster is a distributed system composed of specialised node roles — dedicated master nodes for cluster state management, data nodes for storing and querying shards, coordinator nodes for request routing, and optional ingest/ML nodes. Getting this topology right is critical: an under-resourced master node can make the entire cluster unavailable during high write load, while over-sharded indexes burn heap memory even when idle.

Index Lifecycle Management (ILM) is the Elasticsearch feature that automates the movement of indexes through storage tiers as they age and their query rate decreases. A time-series index (e.g., logs or events) starts on hot nodes (fast NVMe SSDs, high CPU) for real-time indexing and search, transitions to warm nodes (larger, cheaper SSDs) after 7 days for less-frequent queries, moves to cold nodes (bulk HDD or object-storage-backed) after 30 days, and is eventually deleted or snapshotted to S3. This tiered architecture can reduce search infrastructure costs by 60–80% compared to keeping all data on hot hardware.

Why it exists

Running a search engine at production scale is operationally more complex than most distributed systems because it combines the challenges of a database (durable storage, index consistency), a cache (heap-resident data structures), and a compute system (aggregation and ranking pipelines). Misjudging shard count is one of the most common mistakes: too few shards limits parallelism; too many shards wastes heap (each shard has a JVM overhead of ~1.5KB of heap per segment per node). The Elasticsearch guidance of targeting 10–50GB per shard, not exceeding 20 shards per GB of heap, and keeping total shard count below ~1,000 per node gives concrete sizing boundaries.

Index aliases are the operational mechanism that enables zero-downtime reindexing and schema migrations. An alias is a stable logical name that points to one or more physical indexes. When you need to change a field mapping (which requires a full reindex in Elasticsearch), you build the new index in the background, then atomically swap the alias from the old index to the new — applications see no interruption. Without alias-based indirection, any mapping change requires taking the search feature offline during reindex, which can take hours for large indexes.

When to use

  • Any production Elasticsearch or OpenSearch deployment serving user-facing queries where availability and performance matter.
  • Time-series data (logs, events, metrics) where a tiered storage strategy dramatically reduces cost.
  • Indexes that will require periodic schema changes or re-ranking logic updates (alias-based reindex).
  • Multi-tenant deployments where different customers have different data volumes and SLAs.
  • High-write-throughput scenarios (event ingestion) where separating write and query workloads prevents interference.

When not to use

  • Managed services with no direct cluster access: Elastic Cloud and OpenSearch Service abstract most of these concerns; follow their managed service best practices instead.
  • Small-scale deployments: A single-node Elasticsearch instance serving a small internal tool does not need ILM, dedicated master nodes, or hot-warm-cold tiers.
  • Non-Elasticsearch engines: If you are using Algolia, Typesense, or Meilisearch, these are managed services with their own capacity and configuration models.
  • Temporary or POC deployments: Invest in infrastructure design only when you have confirmed the search feature will go to production.

Typical architecture

ELASTICSEARCH CLUSTER TOPOLOGY (production):

  ┌─────────────────────────────────────────────────────┐
  │  Dedicated Master Nodes (3)                         │
  │  node.roles: [master]                               │
  │  Small instance (4 CPU, 8GB RAM)                    │
  │  Purpose: cluster state, shard allocation           │
  └─────────────────────────────────────────────────────┘
               │ cluster management
  ┌────────────┼────────────┐
  ▼            ▼            ▼
  ┌────────┐  ┌────────┐  ┌────────┐
  │ Hot    │  │ Warm   │  │ Cold   │
  │ Node   │  │ Node   │  │ Node   │
  │NVMe SSD│  │SATA SSD│  │ HDD /  │
  │32CPU   │  │16CPU   │  │ Object │
  │128GB   │  │64GB    │  │ Store  │
  └────────┘  └────────┘  └────────┘
  Recent data  7-30 days    30d+

  ┌─────────────────────────────────────┐
  │  Coordinator Nodes (2-3)            │
  │  node.roles: []  (no data, no master│
  │  Receive queries, scatter/gather    │
  └─────────────────────────────────────┘

ILM POLICY (time-series index):
  Hot phase:   rollover at 50GB or 1 day
               (ensure shards stay 10-50GB)
  Warm phase:  after 7 days → move to warm
               force merge to 1 segment
               disable replicas (save space)
  Cold phase:  after 30 days → move to cold
               freeze index (searchable snapshot)
  Delete phase: after 90 days → delete

INDEX ALIAS PATTERN:
  Alias: "products"
  Points to: products-v3 (current)

  Reindex:
  1. Create products-v4 with new mapping
  2. POST _reindex from products-v3 to products-v4
  3. Atomic alias swap:
     POST /_aliases { "actions": [
       { "remove": { "index": "products-v3", "alias": "products" } },
       { "add":    { "index": "products-v4", "alias": "products" } }
     ] }
  4. Delete products-v3

Pros and cons

Pros

  • Hot-warm-cold tiering reduces storage costs by 60–80% for time-series use cases.
  • Dedicated master nodes eliminate cluster instability from master elections during data node overload.
  • Index aliases enable zero-downtime schema migrations and reindexing.
  • ILM automates operational toil (rollover, merge, tier migration, deletion) that would otherwise require manual cron jobs.
  • Coordinator nodes absorb scatter-gather overhead, improving data node search throughput.

Cons

  • Cluster topology planning is complex and mistakes (wrong shard count) are expensive to undo.
  • Hot-warm-cold migration can spike latency for queries that touch warm or cold tiers.
  • ILM policies need careful testing; a misconfigured delete phase can purge data prematurely.
  • Elasticsearch's JVM heap requirements mean vertical scaling is bounded — large clusters require many moderately-sized nodes rather than a few large nodes.
  • Cross-cluster replication (for geo-distribution or DR) adds significant configuration and operational complexity.

Implementation notes

For shard sizing, use the rule of thumb: aim for 10–50GB per shard, and 1 primary shard per 2 CPU cores on hot nodes for write-heavy workloads. Use the _cat/shards API to audit current shard sizes. For a 500GB product index queried but rarely updated, 10 primary shards × 1 replica = 20 total shards is reasonable on a 3-node cluster. Set index.refresh_interval: 30s on hot write-heavy indexes (logs) to reduce indexing pressure; the default 1s refresh is only needed for near-real-time search use cases. Enable _source: false on fields not needed in response (computed fields, large blobs) to reduce storage footprint.

For heap sizing, follow the Elasticsearch recommendation: set JVM heap to no more than 50% of available RAM, and cap at 30–31GB to stay below the JVM compressed ordinary object pointer threshold. Above 32GB, pointer compression is disabled, which wastes memory and can crash the node under segment cache pressure. Monitor heap usage per node in Kibana's Stack Monitoring; sustained heap above 75% indicates the node is under-provisioned. Snapshot to S3/GCS regularly — Elasticsearch snapshots are incremental and cheap, and they are your only defence against accidental index deletion or hardware failure.

Common failure modes

  • Shard over-allocation: Creating an index with 50 shards for 1GB of data creates 50 × nodes segments in memory, increasing heap overhead and slowing queries.
  • Master node instability: Co-locating master-eligible and data roles on the same node means a heavy indexing burst can prevent master elections, causing the entire cluster to go red.
  • ILM stalling: ILM policies fail silently when the cluster has yellow health (unassigned shards); always monitor ILM errors separately from cluster health.
  • No alias indirection: Hardcoding index names in application code makes mapping changes require downtime; establish the alias pattern from day one.
  • Heap at 90%+: Sustained heap above 90% triggers frequent GC pauses, increasing query p99 latency from milliseconds to seconds and potentially crashing the node.

Decision checklist

  • Are dedicated master nodes configured (3 instances, quorum-based election)?
  • Is shard count estimated at 10–50GB per shard with headroom for growth?
  • Are all production indexes accessed via aliases rather than direct index names?
  • Is an ILM policy defined for time-series indexes with rollover, tier migration, and delete phases?
  • Is JVM heap set to ≤50% of RAM and ≤30GB per node?
  • Is there a snapshot policy (daily to S3/GCS) and has restore been tested?
  • Are coordinator nodes deployed to isolate scatter-gather load from data nodes?

Example use cases

  • Application log search (ELK): Daily rollover ILM policy on log indexes; hot (7 days, NVMe) → warm (30 days, SSD) → cold (90 days, searchable snapshot) → delete.
  • E-commerce product index: 3-shard primary index on hot nodes with alias; zero-downtime reindex performed monthly when product schema evolves.
  • Multi-tenant SaaS search: One index per tenant with tenant-specific ILM policies; coordinator nodes protect data nodes from large tenants' scatter-gather overhead.

Further reading