Service Discovery and Load Balancing Between Microservices
Once a system has dozens of services with instances starting and stopping constantly, “call the payments service” needs an answer to “at which address?” That’s service discovery.
Client-side discovery
Instances register in a registry (Consul, etcd, ZooKeeper, Eureka) with heartbeats. Callers query the registry, cache the list, and pick an instance themselves.
- Smart load balancing in the client: round robin, least outstanding requests, or “power of two choices” (pick two at random, use the less busy one).
- Every language needs a client library.
Server-side discovery
Callers send requests to a stable address (a load balancer or a virtual IP); the load balancer knows the healthy instances.
- Kubernetes Services work this way: a stable DNS name and virtual IP, with kube-proxy or the CNI routing to healthy pods.
- Simple for callers, but adds a hop.
Service mesh
A sidecar proxy (Envoy) next to each instance handles discovery, load balancing, retries, timeouts, mutual TLS and metrics. A control plane (Istio, Linkerd) pushes configuration. Powerful, but operationally heavy.
Health checking
- Liveness: is the process alive? Restart it if not.
- Readiness: can it serve traffic now? Remove it from rotation during startup or overload.
- Passive checks: eject instances that return many errors (outlier detection).
DNS caveats
Plain DNS works for discovery but caches can hold stale addresses longer than the TTL, and DNS alone doesn’t balance load well. Many systems combine DNS for coarse routing with smarter client or proxy balancing.
Interview tip
When you draw arrows between services, mention once how they find each other and how unhealthy instances are removed. It rounds out a design answer.
Disclosure: I help build Linkjob AI (https://www.linkjob.ai/), an AI interview helper.