Distributed Systems Infrastructure

Service Discovery

How services find each other dynamically in distributed environments — client-side vs server-side discovery, service registries, health checks, DNS-based discovery, and service mesh integration.

⏱ 11 min read

What it is

Service discovery is the mechanism by which services in a distributed system locate each other's network addresses at runtime, without hardcoding IP addresses or hostnames. In dynamic environments like container orchestration platforms, service instances start and stop frequently, are assigned ephemeral IP addresses, and scale horizontally. A service discovery system maintains a registry of available service instances and their current network addresses, enabling clients to find healthy backends without manual configuration.

Why it exists

In traditional deployments, services had stable hostnames or IP addresses configured in load balancers and application config files. In containerised microservices environments, this model breaks down: a single service may have dozens of instances with ephemeral IPs, instances spin up and down in seconds during scaling events or rolling deployments, and multiple versions may be simultaneously in service. Static configuration cannot track these changes. Service discovery automates the process of tracking service instances and their health, replacing manual load balancer management with a dynamic, self-updating registry.

When to use

Containerised microservices running on Kubernetes, ECS, or Nomad where service IPs are ephemeral.
Auto-scaling groups where the number and identity of instances changes dynamically.
Multi-region deployments where services need to find region-local or global backends.
Blue/green and canary deployments where traffic must be selectively routed to specific versions.

When not to use

Small-scale deployments with stable addresses — a load balancer with static backends is simpler and sufficient.
Serverless architectures where the cloud provider abstracts away instance addresses entirely.

Typical architecture

Client-Side Discovery
══════════════════════
  ServiceA ──► ServiceRegistry.lookup("serviceB")
           ◄── [10.0.1.4:8080, 10.0.1.5:8080]
  ServiceA ──► load balances itself ──► 10.0.1.5:8080
  (client library: Ribbon, Eureka client)

Server-Side Discovery
══════════════════════
  ServiceA ──► Load Balancer / API Gateway
  LB ────────► ServiceRegistry.lookup("serviceB")
           ◄── [10.0.1.4:8080, 10.0.1.5:8080]
  LB ────────► 10.0.1.4:8080
  (client is unaware of individual instances)

DNS-Based Discovery (Kubernetes)
══════════════════════════════════
  serviceB.namespace.svc.cluster.local
    resolves via kube-dns/CoreDNS to ClusterIP
    ClusterIP is load-balanced by iptables/IPVS
  Headless service: resolves to individual Pod IPs directly

Service Registration Flow
══════════════════════════
  Instance starts → register with health endpoint
  Registry polls /health every 10s → marks healthy
  Instance stops → deregister (or TTL expires)
  Clients receive updated instance list via watch or TTL cache

Pros and cons

Pros

Handles dynamic instance lifecycle — no manual load balancer reconfiguration.
Health-check integration ensures traffic is only sent to passing instances.
Enables advanced routing: version-based, zone-aware, latency-based load balancing.
Foundation for service mesh traffic management without code changes in services.

Cons

Client-side discovery requires each language/framework to implement discovery logic and maintain a client library.
The registry itself is a critical dependency — its unavailability can prevent all service-to-service communication.
DNS-based discovery has TTL caching delays, which can route traffic to stopped instances during transitions.
Health check false positives can prematurely deregister a slow-but-functioning instance.

Implementation notes

Consul provides service discovery with health checks, a key-value store, and multi-datacenter support. It uses a gossip protocol (Serf) for cluster membership and Raft for consistent KV state. Consul's DNS interface allows services to discover each other using standard DNS without a client library. Consul Connect adds service mesh mTLS between services using Envoy as a sidecar proxy.

Kubernetes service discovery works via DNS (CoreDNS) and the Kubernetes API. Services are assigned a stable ClusterIP and DNS name; the kube-proxy routes ClusterIP traffic to healthy Pods via iptables or IPVS. For StatefulSets requiring direct Pod addressing (e.g., Cassandra), a headless service omits the ClusterIP and DNS resolves directly to Pod IPs. Kubernetes Endpoints/EndpointSlices track the current set of ready Pods.

Zero-downtime registration: during deployment, a new instance should be fully initialised and pass health checks before being registered. During shutdown, it should be deregistered first, then the process should wait for in-flight requests to complete (graceful drain) before terminating. Kubernetes readinessProbe and preStop lifecycle hooks implement this pattern.

Service mesh integration (Istio, Linkerd) takes service discovery to its logical conclusion — the data plane sidecar proxy (Envoy) receives service endpoints directly from the control plane (xDS API) and handles all routing, retries, mTLS, and observability transparently without any application-level discovery code.

Common failure modes

Registry unavailability: if Consul or etcd goes down, clients with stale cached endpoints may continue working temporarily, but new instances cannot register and failed instances cannot deregister.
DNS TTL caching: aggressive DNS caching by JVM (default: 30s or infinite with certain security settings) causes traffic to be sent to terminated instances long after deregistration.
Health check misconfiguration: a health endpoint that always returns 200 regardless of actual readiness causes the registry to route traffic to unhealthy instances.
Split-brain in registry: network partition can cause different Consul/etcd nodes to have different service lists; clients on different sides may see different healthy instances.

Decision checklist

Is Kubernetes handling service discovery? If so, is CoreDNS/ClusterIP sufficient or do you need a service mesh?
Are services polyglot? Server-side discovery avoids per-language client library maintenance.
Are health checks accurately reflecting instance readiness (not just liveness)?
Is the service registry deployed with HA (e.g., 3-node Consul cluster) and monitored?
Have you configured graceful shutdown and deregistration before SIGTERM?

Example use cases

Netflix Eureka + Ribbon for client-side service discovery across hundreds of microservices.
HashiCorp Consul for service discovery with health checks in a Nomad cluster running mixed workloads.
Kubernetes CoreDNS resolving service names for Pod-to-Pod communication within a cluster.
AWS App Mesh with Envoy sidecars providing service discovery via Cloud Map for ECS services.

Distributed Coordination — Consul and etcd serve dual roles as service registries and distributed coordination primitives.
Gossip Protocols — Consul uses gossip (Serf) for cluster membership, which underpins its service discovery reliability.
Microservices — service discovery is a core infrastructure requirement for microservices architectures.