Vertical vs horizontal scaling

What to do before changing anything

Measure where each request spends its time and find the bottleneck.

Lesson 1-2 in one sentence is to find the bottleneck first. Effort spent anywhere else changes almost nothing, as the 25 ms to 21 ms arithmetic showed.

Skipping the measurement is tempting because everyone has a hunch about what is slow. Hunches are wrong often enough that the measurement is cheaper than the wasted week, and unit 8 is about making the measurement routine.

In this unit we assume you measured and the app tier is the limit, so we scale it. If the database had been the limit instead, units 3 through 5 would be the right destination, and the same measurement is what tells you which.

Two ways to get more capacity

Vertical scaling (scaling up): replace the box with a bigger box. More CPU cores, more RAM, faster disks. Nothing about your code changes.

Horizontal scaling (scaling out): keep the box, add more boxes just like it, and split traffic between them.

Vertical scaling is the right first move. It is one config change at your cloud provider and zero changes to your architecture. But it has two hard limits:

1. A ceiling. The biggest machine money can buy is still one machine
2. A price curve. Big machines cost more per unit of power, and doubling the hardware rarely doubles real throughput

And one quiet flaw: one box is still a single point of failure, one part whose death takes the whole system down. When it reboots, your product is offline.

The price curve of bigger machines

Typical cloud prices, with the cost per vCPU computed alongside.

for cpus, price in [(2, 70), (4, 150), (8, 340), (16, 780)]:
    per_cpu = round(price / cpus, 1)
    print(f"{cpus} vCPUs costs ${price}/mo (${per_cpu} per vCPU)")

Output

2 vCPUs costs $70/mo ($35.0 per vCPU)
4 vCPUs costs $150/mo ($37.5 per vCPU)
8 vCPUs costs $340/mo ($42.5 per vCPU)
16 vCPUs costs $780/mo ($48.8 per vCPU)

The per-vCPU price climbs from $35 to $48.80 as the machines get bigger, which is a 39% premium for the same unit of compute. That curve is why vertical scaling gets expensive, and it steepens further at the top of a provider's range.

f-strings like f"{cpus} vCPUs" fill values into the text, which keeps the line readable compared to concatenation. The $ before {price} is just a literal character, and only the braces are special.

The curve is the opposite of what intuition suggests, since buying in bulk usually gets cheaper. Large instances cost more per unit because they need specialized hardware and because the customers who need them will pay.

Note that the raw price is only part of the comparison. The 16-vCPU box also does not deliver eight times the throughput of the 2-vCPU box, since memory bandwidth and lock contention get in the way, so the effective premium is larger than this table shows.

Why horizontal wins at scale

Horizontal scaling has no ceiling: need 10x capacity, run 10x the servers. It also removes the single point of failure, because when one server dies the others keep serving.

The cost: your architecture must change. Two new problems appear the moment you have two servers:

1. Who sends each request to which server? That is the load balancer, next lesson
2. What happens to data a server was holding in memory? That is state, lesson 2-3

Every large system ends up horizontal. The skill is knowing that the simple vertical move buys you time, and taking it, before doing the harder thing.

Comparing one big box against eight small ones

Same total vCPUs, two very different bills.

big_server_price = 780
small_server_price = 70
small_servers_needed = 8
horizontal_price = small_server_price * small_servers_needed
print("Vertical: $" + str(big_server_price) + "/mo")
print("Horizontal: $" + str(horizontal_price) + "/mo")
print("Savings: $" + str(big_server_price - horizontal_price) + "/mo")

Output

Vertical: $780/mo
Horizontal: $560/mo
Savings: $220/mo

Eight small machines cost $220 less per month for the same 16 vCPUs, which is a 28% saving that comes straight from the price curve in the previous block. Cheap units bought many times beat expensive units bought once.

The money is not even the main argument. Eight machines also survive one machine dying, while the single big box is a single point of failure whose reboot is an outage.

The bottom two rows are the price you actually pay, and it is engineering time rather than dollars. Those are the next two lessons, and they are why the $220 saving is not the reason anyone scales out.

Note that this comparison assumes the eight small boxes deliver the same real throughput as the one big box. They usually deliver more, for the memory-bandwidth reason in the previous block, so the table understates the horizontal case.

The sensible first move at 90% CPU

Scale vertically now with a bigger box, and plan horizontal scaling next.

Vertical scaling is a config change that buys immediate headroom with zero code risk. It can often be done in an hour, and lesson 1-3 says why it matters so much, since dropping utilization from 90% to 45% collapses the queueing latency users are feeling right now.

Horizontal scaling is the durable fix and needs a load balancer plus stateless services, which is real engineering work. Two weeks of runway is enough to do it properly, and not enough to do it properly under the pressure of a site that is already falling over.

The sequencing is the actual answer. Relieve the pressure with the cheap move, then do the structural work calmly, and arrive with both.

Buying time with the simple fix is good engineering rather than laziness. The failure mode to avoid is the opposite one, where a team starts a two-week architecture change while paying customers watch a slow site.