Horizontal scaling – more instances behind a load balancer – is the standard answer to growing traffic. It works well, provided the application was built to run as more than one copy. Most of the difficulty lives in that condition, not in the load balancer.
What has to be true first
- No request-scoped state stored on the instance’s local disk or memory.
- Sessions held in a shared store, or the design is genuinely stateless.
- Uploads written to object storage, not the local filesystem.
- Scheduled jobs that must run once guarded by a lock or moved to a single worker.
- A health endpoint that reflects real readiness, not just that the process started.
Work through that list before adding a second instance. Each item causes a distinctive and confusing class of intermittent bug when violated.
Health checks decide reliability
A health check that only confirms the process is listening will happily send traffic to an instance whose database connection is dead. A check that queries every dependency will take the whole fleet out when one dependency has a blip. The useful middle is a readiness check covering what the instance needs to serve requests, with a short timeout and a tolerance for transient failures.
| Check type | Should test |
|---|---|
| Liveness | The process is not deadlocked. Restart if it fails. |
| Readiness | This instance can serve now. Remove from rotation if not. |
| Startup | Slow initialisation completed, before liveness applies. |
Autoscaling needs the right signal
- Choose a metric that correlates with user-visible pain, not just CPU.
- Scale out quickly and scale in slowly, to avoid oscillation.
- Set a floor that survives your normal trough, and a ceiling you can afford.
- Ensure new instances become useful quickly; slow boot defeats autoscaling.
- Test a scale-out event deliberately rather than discovering it in an incident.
Know the ceiling of vertical scale
Before distributing, check whether a larger instance simply solves the problem. One bigger machine is dramatically simpler than a fleet, and modern instances are large. Scale vertically until it stops being cost-effective or until availability requirements demand redundancy – then scale out.
Summary
Make the application safe to run twice, give the balancer an honest readiness signal, and scale on a metric tied to user experience. The load balancer is the easy part.
