Load Balancer
The traffic cop that lets you run many servers as if they were one.
Introduction
A load balancer sits between clients and a pool of servers and spreads incoming requests across them. To the outside world there is a single address; behind it, any number of interchangeable servers share the work. It is the piece that turns one machine into a fleet.
Why it exists
A single server has a hard ceiling: CPU, memory, connections. When traffic grows past that ceiling you must run more servers — but clients only know one address. Something has to decide which server handles each request, notice when a server dies, and stop sending traffic to it. That is the load balancer's job: it enables horizontal scaling and removes any single server as a single point of failure.
Analogy
Think of a busy airport check-in. Instead of everyone queuing at one desk, a coordinator at the front waves each traveler to whichever desk is free. If a desk closes, the coordinator simply stops sending people there. Travelers never need to know how many desks exist — they just follow the coordinator.
How it works
The load balancer accepts a connection and applies an algorithm to pick a backend: round-robin (rotate through servers), least-connections (send to the least busy), or a hash of some key (to keep a user on one server). It continuously runs health checks — if a server fails to respond, it is removed from rotation until it recovers.
Load balancers operate at different layers. An L4 balancer routes by IP and port without looking at the payload — fast and protocol-agnostic. An L7 balancer understands HTTP, so it can route by path, header, or cookie, terminate TLS, and do things like sticky sessions. Most modern web systems use an L7 balancer at the edge.
Critically, servers behind the balancer should be stateless so any request can go to any server. Session data lives in a shared store (Redis, a database) rather than in a server's memory.
- 1
Client resolves the public hostname to the load balancer's IP.
- 2
The load balancer accepts the connection and (at L7) terminates TLS.
- 3
It selects a healthy backend using its algorithm (e.g. least-connections).
- 4
The chosen server reads/writes shared state and builds the response.
- 5
The response flows back through the balancer to the client.
- 6
A background health check removes any unhealthy server from rotation.
Interactive
Architecture
Requests fanned out across a stateless fleet
Single entry point
Stateless fleet
Shared state
Any server can serve any request because session state lives in the shared store, not in memory.
In code
upstream app_pool {
least_conn; # send to the server with fewest active connections
server 10.0.0.11:3000 max_fails=3 fail_timeout=15s;
server 10.0.0.12:3000 max_fails=3 fail_timeout=15s;
server 10.0.0.13:3000 max_fails=3 fail_timeout=15s;
}
server {
listen 443 ssl;
location / {
proxy_pass http://app_pool;
proxy_next_upstream error timeout http_502; # retry another server on failure
}
location /healthz { return 200; }
}// app/healthz/route.ts
import { NextResponse } from "next/server"
import { db } from "@/lib/db"
export async function GET() {
try {
// "Are my dependencies reachable?" — a readiness check, not just "am I alive".
await db.query("SELECT 1")
return NextResponse.json({ status: "ok" }, { status: 200 })
} catch {
return NextResponse.json({ status: "degraded" }, { status: 503 })
}
}In the real world
Where you've seen this
Every large web product — from an e-commerce checkout to a streaming API — runs behind one or more load balancers (AWS ALB/NLB, GCP Cloud Load Balancing, or nginx/HAProxy). During a traffic spike the autoscaler adds servers and registers them with the balancer; during a deploy, the balancer drains connections from old instances before removing them.
When to use it
Reach for it when
- You run more than one instance of a service and need a single entry point.
- You want zero-downtime deploys via connection draining and rolling updates.
- You need to remove failed instances from rotation automatically.
- You want to terminate TLS and route by path/host at the edge.
Avoid it when
- A single low-traffic instance where the balancer adds cost and a hop for no benefit.
- Purely internal point-to-point calls better served by service discovery + client-side balancing.
- When you actually need sticky in-memory state and cannot externalize it (fix the design instead).
Trade-offs
Advantages
- Enables horizontal scaling behind one stable address.
- Removes individual servers as single points of failure.
- Centralizes TLS termination, routing, and health checking.
- Supports rolling deploys and graceful draining.
Disadvantages
- Becomes a critical component that itself must be made redundant.
- Adds a network hop and some latency.
- L7 features (TLS, inspection) cost CPU at high throughput.
- Sticky sessions, if used, undermine even balancing.
L4 vs L7 is the core trade-off: L4 is faster and protocol-agnostic but blind to the request; L7 is smarter (path routing, TLS, cookies) at higher CPU cost. Round-robin is simple but ignores real load; least-connections adapts but needs live state. Sticky sessions simplify stateful apps but re-introduce the single-point-of-failure and uneven load you were trying to avoid — prefer externalized state.
| L4 vs L7 | L4 (Transport) | L7 (Application) |
|---|---|---|
| Routes by | IP + port | Path, host, header, cookie |
| TLS termination | No (pass-through) | Yes |
| Throughput | Very high, low CPU | Lower, more CPU |
| Best for | Raw TCP/UDP, extreme throughput | HTTP APIs, smart routing |
Common mistakes
Watch out for
- Storing session state in server memory, breaking statelessness.
- Health checks that only verify the process is up, not that it can serve traffic.
- Making the load balancer itself a single point of failure (no redundancy).
- Relying on sticky sessions instead of a shared session store.
- Ignoring connection draining, so deploys drop in-flight requests.
Failure thinking
What breaks it
What happens if the load balancer itself goes down?
It becomes a total outage — nothing reaches the fleet. This is why managed balancers are horizontally redundant across zones and fronted by DNS (or anycast). Run at least two balancer nodes and health-check them too; never leave a single balancer as your only entry point.
What if a server passes health checks but is silently slow?
A shallow health check keeps sending it traffic while users see timeouts. Use latency-aware checks and outlier detection (eject a backend when its error/latency spikes), plus per-request timeouts and retries to a different backend.
Think like a senior
Senior Engineer Insight
The load balancer is where you get graceful degradation almost for free: timeouts, retries to other backends, outlier ejection, and connection draining all live here. Configure them deliberately rather than accepting defaults.
Senior Engineer Insight
Horizontal scaling is only as good as your statelessness. If a senior candidate says 'add a load balancer' without addressing where session/state lives, they haven't finished the thought.
Remember
A load balancer only buys you scale if the servers behind it are stateless.
Remember
The balancer must never be your only single point of failure — make it redundant.
Interview questions
How does a load balancer decide which server to send a request to?
Why must servers behind a load balancer be stateless, and how do you achieve it?
Active recall
Check yourself
Why should application servers behind a load balancer be stateless?
Practical challenges
Round-robin balancer in memory
Build a tiny in-process load balancer that distributes calls across N backends with health checks.
Requirements
- Implement `next()` returning the next healthy backend in rotation.
- Support marking a backend unhealthy and healthy again.
- Skip unhealthy backends without breaking rotation.
Constraints
- No external libraries.
- O(1) selection where possible.
Acceptance criteria
- With all healthy, calls are evenly distributed.
- Marking one unhealthy removes it from rotation immediately.
- Restoring it puts it back in rotation.
Edge cases
- All backends unhealthy.
- Only one backend healthy.
Bonus
- Add a least-connections strategy behind the same interface.
Reflection
- How would round-robin behave if request costs vary wildly?
Zero-downtime rolling deploy
Design a rolling deploy that never drops an in-flight request behind a load balancer.
Requirements
- Define readiness vs liveness checks.
- Describe connection draining on the outgoing instances.
- Specify the order of operations for replacing each instance.
Acceptance criteria
- At all times at least N-1 instances serve traffic.
- In-flight requests on a retiring instance complete before it stops.
- A failed new version does not take the fleet down (halt + rollback).
Edge cases
- A new instance passes liveness but fails readiness.
- A long-running request exceeds the drain timeout.
Bonus
- Add a canary step that sends 5% of traffic to the new version first.
Reflection
- Where do database migrations fit so they stay backward compatible during the overlap?
Architecture challenge
You run 3 app servers behind one load balancer. Traffic triples for a flash sale and one server's disk fills up mid-event, so it returns 500s intermittently while still accepting connections. Design the health checking, timeout, and autoscaling behavior so users never notice. What signals do you check, how fast do you eject the bad node, and how do you avoid a retry storm hammering the two healthy servers?
Flashcards
Summary
A load balancer presents one address for a fleet of interchangeable servers, spreading traffic by an algorithm and removing unhealthy nodes via health checks. It is the enabler of horizontal scaling and the place where much of your resilience (timeouts, retries, draining) is configured — provided your servers are stateless.
Key takeaways
- One address, many stateless servers — that is the whole idea.
- Health checks + connection draining give you resilience and zero-downtime deploys.
- L4 is fast and blind; L7 is smart and costs CPU.
- Never let the balancer be your only single point of failure.
Your notes
Saved to this device