Horizontal Scaling
Add more machines instead of a bigger machine.
Introduction
Horizontal scaling (scaling out) means handling more load by adding more machines that share the work, rather than making one machine bigger (scaling up / vertical scaling). It is the foundation of nearly every large system, because it removes the ceiling of a single box and improves fault tolerance.
Why it exists
Vertical scaling hits physical and economic limits: there is a largest server you can buy, upgrades require downtime, and price grows faster than capacity at the top end. A single big box is also a single point of failure. Horizontal scaling answers 'what do we do after the biggest server isn't enough?' — spread the load and let the fleet survive individual failures.
Analogy
If a restaurant kitchen can't keep up, you can buy one enormous super-oven (vertical) or add more ordinary ovens and cooks (horizontal). More ovens means no single ceiling, and if one breaks, dinner service continues.
How it works
You run many identical instances of a service behind a load balancer. Because the instances are stateless, any of them can serve any request, so adding an instance adds capacity almost linearly — until you hit a shared bottleneck like the database.
An autoscaler watches a signal (CPU, request latency, queue depth) and adds or removes instances to match demand. State that used to live in a process (sessions, in-memory caches, locks) must move to shared infrastructure (Redis, the database) so instances remain interchangeable. The hard part of scaling out is rarely the app tier — it is the stateful layers (databases, caches) that need replication, partitioning, or sharding to scale with it.
- 1
Load rises; the autoscaler sees CPU/latency cross a threshold.
- 2
New identical instances start and pass readiness checks.
- 3
The load balancer begins routing to them, raising total throughput.
- 4
Shared state (Redis/DB) absorbs the extra reads/writes.
- 5
When load falls, the autoscaler scales in and drains removed instances.
Interactive
Architecture
Scaling the stateless tier; the datastore becomes the bottleneck
Autoscaled app tier (linear)
Shared state (the real limit)
App instances scale almost linearly; the database is where scaling-out gets genuinely hard.
In code
// Bad: session in process memory — breaks the moment you add a second instance.
const sessions = new Map<string, Session>()
// Good: session in a shared store any instance can read.
import { redis } from "@/lib/redis"
export async function getSession(id: string): Promise<Session | null> {
const raw = await redis.get(`session:${id}`)
return raw ? (JSON.parse(raw) as Session) : null
}
export async function saveSession(id: string, s: Session) {
await redis.set(`session:${id}`, JSON.stringify(s), "EX", 3600)
}In the real world
Where you've seen this
A Black Friday sale drives 10x normal traffic. The web/app tier autoscales from 8 to 80 instances within minutes and back down afterward. The database, which cannot be cloned as freely, is protected with read replicas, aggressive caching, and a queue for writes — because that is the layer that doesn't scale by 'just add machines'.
When to use it
Reach for it when
- Traffic is variable or growing beyond a single machine.
- You need fault tolerance — surviving the loss of individual nodes.
- Your workload parallelizes across independent requests.
Avoid it when
- Small, steady workloads a single (redundant pair of) server handles comfortably.
- Workloads dominated by a single non-shardable stateful component.
- When the real bottleneck is the database and adding app servers just moves the queue.
Trade-offs
Advantages
- No single-machine ceiling; near-linear capacity for stateless tiers.
- Better fault tolerance — one node failing is not an outage.
- Elastic: scale out for peaks, in for troughs, paying for what you use.
Disadvantages
- Requires statelessness and shared state infrastructure.
- The database/cache tier becomes the hard scaling problem.
- More moving parts: service discovery, config, observability across nodes.
- Distributed concerns appear (consistency, coordination).
Vertical scaling is simplest — no code changes, no distribution — but has a hard ceiling and a single point of failure. Horizontal scaling has no ceiling and better resilience but forces statelessness, shared state, and distributed-systems concerns. In practice teams scale up first (cheap, easy) and scale out when they must — and the moment they scale out, the database becomes the design's center of gravity.
| Vertical vs Horizontal | Vertical (scale up) | Horizontal (scale out) |
|---|---|---|
| How | Bigger machine | More machines |
| Ceiling | Hard limit of one box | Effectively unbounded for stateless tiers |
| Fault tolerance | Single point of failure | Survives node loss |
| Complexity | Low — no code change | Higher — statelessness, coordination |
| Best for | Quick wins, stateful monoliths | Elastic, large, resilient systems |
Common mistakes
Watch out for
- Adding app servers while the database is the actual bottleneck.
- Keeping state (sessions, locks, in-memory caches) in process memory.
- Autoscaling on CPU when the real signal is queue depth or latency.
- No connection pooling, so more app instances exhaust DB connections.
Failure thinking
What breaks it
Traffic suddenly increases 100x. What breaks first?
Usually not the app tier (it autoscales) but the database — connection limits, lock contention, or replica lag. Protect it with caching, connection pooling/proxies (e.g. PgBouncer), queueing writes, and load shedding. Autoscaling the app tier without protecting the datastore just delivers the flood faster.
You add 40 instances and the database starts refusing connections. Why?
Each instance opens a pool of DB connections; 40 × pool size can exceed the DB's max_connections. Put a connection pooler in front of the database and cap per-instance pools. Connections are a finite shared resource that does not scale with app instances.
Think like a senior
Senior Engineer Insight
Scaling the stateless tier is the easy 20%. The interview signal is whether you immediately talk about the database and cache tier — replication, sharding, pooling — because that is where scaling out actually gets hard.
Senior Engineer Insight
Autoscale on the metric that reflects user pain (p99 latency, queue depth), not just CPU. CPU can look fine while requests pile up waiting on a slow dependency.
Remember
Horizontal scaling is only linear until you hit a shared bottleneck — usually the database.
Remember
You cannot scale out until your services are stateless.
Interview questions
When would you choose vertical scaling over horizontal scaling?
Active recall
Check yourself
A key advantage of horizontal scaling over vertical scaling is…
Practical challenges
Make a stateful service stateless
Refactor an app that stores sessions and a rate-limit counter in memory so it can run on many instances.
Requirements
- Move session storage to a shared store.
- Move the rate-limit counter to a shared store with atomic increments.
- Ensure no correctness depends on which instance handles a request.
Acceptance criteria
- A user logged in via instance A is recognized by instance B.
- Rate limits are enforced across the whole fleet, not per instance.
Edge cases
- Two requests for the same user hit two instances simultaneously.
Bonus
- Add a TTL so sessions and counters expire automatically.
Reflection
- Which remaining pieces of state, if any, are safe to keep local?
Architecture challenge
Your app tier autoscales beautifully, but during peaks p99 latency still spikes and errors climb. Metrics show CPU on app servers is only 40%. Diagnose where the bottleneck likely is and design the layers of defense (caching, pooling, read replicas, queueing, load shedding) that let the system absorb a 20x spike without falling over.
Flashcards
Summary
Horizontal scaling handles growth by adding interchangeable, stateless machines behind a load balancer, with an autoscaler matching capacity to demand. It removes the single-machine ceiling and improves resilience — but shifts the hard problem onto the stateful database and cache tiers, which need replication, pooling, and sometimes sharding to keep up.
Key takeaways
- Scale out = more machines; scale up = bigger machine.
- Stateless app tiers scale nearly linearly; databases do not.
- Externalize sessions, locks, and caches to shared stores.
- Autoscale on user-facing signals and protect the datastore.
Your notes
Saved to this device