The two speed numbers
Before we scale anything, two measurements need precise names, because every design decision in this course is justified by one of them, and engineers quote both in every capacity discussion. Latency is how long one request takes from send to response, measured in milliseconds. Throughput is how many requests the system completes per second (the RPS from lesson 1-1 is a throughput). Latency is what a user feels on each tap. Throughput is how many users you can serve at once.
They are different dimensions, not two views of the same thing:
- Adding identical servers multiplies throughput, but each individual request still does the same work, so its latency barely changes
- Making one request cheaper (a better query, a cache) improves latency, and as a side effect frees the server to take more work
A delivery company shows the difference: hiring more drivers raises packages delivered per day (throughput) but does not make any single package arrive sooner (latency).
Servers multiply throughput, not speed
One server takes 25 ms per request and handles one at a time, and adding servers changes only one of the two numbers.
service_ms = 25 print("Latency per request (ms):", service_ms) print("Throughput, 1 server (req/sec):", 1000 // service_ms) servers = 4 print("Throughput, 4 servers (req/sec):", 1000 // service_ms * servers) print("Latency with 4 servers (ms):", service_ms)
Output
Latency per request (ms): 25 Throughput, 1 server (req/sec): 40 Throughput, 4 servers (req/sec): 160 Latency with 4 servers (ms): 25
The first and last lines are identical, and that is the whole lesson. Four servers do four times the work per second, and each individual request still takes exactly as long as it did, because it still runs the same code against the same database.
Throughput scales by multiplication because the servers are independent. Each one contributes its own 40 RPS, so the totals add, which is what makes horizontal scaling attractive in the next unit.
To improve the 25 ms itself you must make the request cheaper, which is what unit 3 does with caching and lesson 4-1 does with indexes. Those are different tools than adding capacity, aimed at a different number.
There is one exception worth flagging, and the next block is about it. Adding servers can improve latency when the old servers were overloaded, because then most of the latency was queueing rather than work.
Why latency explodes near full load
There is one place the two numbers collide, and it explains most mysterious slowness in production. When a server is busy, a new request waits in line before its work even starts, so its latency is wait time plus service time. Queueing theory gives a rough but famous estimate of the average: with a service time of s and a utilization of u (the fraction of time the server is busy, from 0 to 1),
average time in system ≈ s ÷ (1 − u)
Read the shape of that formula. At u = 0.5 the denominator is 0.5, so requests take about 2× the service time. At u = 0.95 the denominator is 0.05, about 20×. Latency does not degrade gradually, it explodes as utilization approaches 100%.
This is why production teams add capacity around 60 to 70% utilization instead of squeezing out 95%, and it explains lesson 1-1's evening crashes: the peak pushed utilization toward 1, and the shrinking denominator did the rest.
The latency cliff, computed
The same formula at four utilization levels, with a 20 ms service time.
service_ms = 20 for utilization in [0.5, 0.8, 0.9, 0.95]: time_ms = service_ms / (1 - utilization) print("utilization " + str(round(utilization * 100)) + "% -> average time " + str(round(time_ms, 1)) + " ms")
Output
utilization 50% -> average time 40.0 ms utilization 80% -> average time 100.0 ms utilization 90% -> average time 200.0 ms utilization 95% -> average time 400.0 ms
Going from 90% to 95% busy doubles the response time, and that cliff is the point of the exercise. The utilization changed by five percentage points and the user-visible latency changed by 200 ms.
Look at the whole curve rather than any single row. Half the increase in latency across this table happens in the last five points of utilization, which is why the region above 90% is worth staying out of entirely.
The denominator is doing all the work, since 1 - utilization shrinks toward zero and dividing by a small number produces a large one. At 99% utilization the formula gives 2,000 ms, and at 100% it gives infinity, which is the mathematical way of saying the queue never stops growing.
The formula is a rough model rather than a prediction, and it assumes random arrivals and one server. Real numbers differ, and the shape does not, which is what makes it worth remembering.
That shape also explains a common operational surprise. A system at 70% utilization absorbing a 30% traffic increase does not get 30% slower, it gets several times slower, because the increase lands on the steep part of the curve.
Which fix attacks the right number
Add servers, so utilization drops well below the explosion zone.
Near saturation, latency is dominated by waiting rather than working, per the s ÷ (1 − u) shape you just computed. Of the 300 ms a user sees, roughly 30 ms is real work and the rest is time spent in line.
More capacity lowers u, which collapses the wait. This is the case where adding servers does improve latency, and it is not a contradiction of the earlier block, because what shrinks is the queue rather than the work.
Shaving 1 ms of service time barely moves the denominator, so it is the wrong lever here. The service time is not the problem when service time is 10% of the response.
A longer timeout is worse than useless, because it hides the queue from your metrics while users still wait. The graph stops showing errors and the experience is unchanged, which is the kind of fix that makes a problem harder to find later.
Capacity problems need capacity answers, which is exactly where unit 2 goes. Traffic doubling with utilization already at 90% is a straightforward capacity problem, and the only question left is whether to grow the box or add boxes.