The two speed numbers
Before we scale anything, two measurements need precise names, because every design decision in this course is justified by one of them, and engineers quote both in every capacity discussion. Latency is how long one request takes from send to response, measured in milliseconds. Throughput is how many requests the system completes per second (the RPS from lesson 1-1 is a throughput). Latency is what a user feels on each tap. Throughput is how many users you can serve at once.
They are different dimensions, not two views of the same thing:
- Adding identical servers multiplies throughput, but each individual request still does the same work, so its latency barely changes
- Making one request cheaper (a better query, a cache) improves latency, and as a side effect frees the server to take more work
A delivery company shows the difference: hiring more drivers raises packages delivered per day (throughput) but does not make any single package arrive sooner (latency).
Code exercise · python
Run this comparison. One server takes 25 ms per request and handles one request at a time. Adding servers multiplies throughput while latency stays exactly where it was.
Why latency explodes near full load
There is one place the two numbers collide, and it explains most mysterious slowness in production. When a server is busy, a new request waits in line before its work even starts, so its latency is wait time plus service time. Queueing theory gives a rough but famous estimate of the average: with a service time of s and a utilization of u (the fraction of time the server is busy, from 0 to 1),
average time in system ≈ s ÷ (1 − u)
Read the shape of that formula. At u = 0.5 the denominator is 0.5, so requests take about 2× the service time. At u = 0.95 the denominator is 0.05, about 20×. Latency does not degrade gradually, it explodes as utilization approaches 100%.
This is why production teams add capacity around 60 to 70% utilization instead of squeezing out 95%, and it explains lesson 1-1's evening crashes: the peak pushed utilization toward 1, and the shrinking denominator did the rest.
Code exercise · python
Your turn: print the average time in the system for a 20 ms service time at each utilization in the list, using time = service ÷ (1 − utilization). Round the time to 1 decimal. The percent shown is the utilization times 100, rounded to a whole number. Match the format exactly.
Quiz
Your API's latency is a steady 30 ms until traffic reaches about 90% of capacity, then it jumps past 300 ms. Traffic is about to double. Which fix attacks the right number?