The traffic director
A load balancer (LB) is a server whose only job is to receive every incoming request and forward it to one of your app servers. The browser talks to a single address, the load balancer's, and never knows how many servers sit behind it.
What a load balancer gives you:
- Distribution. Spreads requests so no single server is overwhelmed
- Health checks. It pings each server every few seconds. A server that stops answering is removed from rotation, and users never see its errors
- Elasticity. Add or remove servers at any time. The LB just updates its list
How does it pick a server? The two algorithms to know:
- Round robin: take turns. Server A, B, C, A, B, C. Simple and fair when requests are similar
- Least connections: send each request to the server currently handling the fewest. Better when some requests are much slower than others
Round robin in three lines
The modulo operator cycles through the server list, so request i goes to server i % 3.
servers = ["A", "B", "C"] for i in range(7): print("request", i + 1, "-> server", servers[i % len(servers)])
Output
request 1 -> server A request 2 -> server B request 3 -> server C request 4 -> server A request 5 -> server B request 6 -> server C request 7 -> server A
i % len(servers) is the whole algorithm, and using len(servers) rather than a hardcoded 3 is what makes the list resizable. Adding a server D means the cycle length changes with no other edit, which is the elasticity property from the intro.
With servers = ["A", "B"] the cycle would shorten, alternating between just those two. Removing an unhealthy server from the list is exactly this operation, which is how health checks take effect.
Round robin is fair in the number of requests and not in the work done. Seven requests split as three, two, two here, and if request 1 happens to be an expensive report while the rest are cheap lookups, server A is doing most of the work.
That is the gap the next algorithm closes. Counting requests is a proxy for load, and counting active connections is a better one.
Least connections
The dict holds how many active requests each server has, and each new request goes to the smallest.
active = {"A": 2, "B": 0, "C": 1}
for i in range(5):
server = min(active, key=lambda s: active[s])
active[server] += 1
print("request", i + 1, "-> server", server)
print("final load:", active)Output
request 1 -> server B request 2 -> server B request 3 -> server C request 4 -> server A request 5 -> server B final load: {'A': 3, 'B': 3, 'C': 2}
min(active, key=lambda s: active[s]) finds the key with the smallest value, since iterating a dict yields keys and the key function tells min what to compare. Without it, min would compare the server names alphabetically and always answer "A".
Watch the assignments correct an imbalance. B starts with zero and takes the first two requests, and by the end the loads are 3, 3, and 2, which is far more even than the starting 2, 0, 1.
This is the behavior round robin cannot produce, because round robin does not look at current state. It would have sent request 1 to A, which was already the busiest server.
Note that the counter has to go down too, and this simulation never decrements. A real load balancer subtracts one when a response completes, and that is what keeps the numbers meaningful over time.
Least connections is the better default when request costs vary, which on most real APIs they do. A search endpoint and a health check do not belong in the same fair-by-count rotation.
What users experience when a server crashes
A brief window of possible errors, then nothing, because the load balancer stops routing to B.
Health checks run every few seconds, so within one check interval the load balancer marks B unhealthy and routes only to A and C. Requests already in flight to B may fail during that window, which is a handful of users seeing one error.
The remaining servers absorb B's share, and that is worth planning for. Three servers becoming two means each survivor takes 50% more traffic, so a fleet running at 70% utilization is suddenly past 100%, which is the reasoning behind N+1 provisioning in the next lesson.
Compare with unit 1, where the same crash on one box meant total downtime. Every user, every request, until someone woke up and restarted it.
This is why horizontal scaling improves availability and not just capacity. The capacity argument is about cost curves, and the availability argument is often the one that actually justifies the work.
Who balances the load balancer?
One worry should nag you: the load balancer is a single machine sitting in front of everything. Did we just move the single point of failure from lesson 2-1 instead of removing it?
Yes, and the fix is standard practice. A load balancer does almost no work per request (no app code, no database queries), so one machine can forward enormous traffic, and you rarely need many. For failure you run a redundant pair: an active load balancer plus a standby that monitors it with the same heartbeat idea as health checks, and takes over its network address within seconds if it dies. Very large systems add DNS-level balancing above that, publishing several load balancer addresses so browsers spread across them.
The habit to take away: for every tier you draw, ask what happens when this one machine dies. Interviewers ask exactly that, and this course will keep answering it tier by tier.
What limits the damage when the load balancer fails
A standby load balancer detects the failure via heartbeats and takes over the shared address within seconds.
Load balancers are deployed as redundant pairs precisely because they sit in front of everything. A single one is the one machine whose death is a total outage, no matter how many app servers are healthy behind it.
The standby watches the active one and claims its address on failure, a pattern called failover. Taking over the address is the essential part, since clients and DNS keep pointing at the same place and never learn that the machine behind it changed.
Remember the word, because in lesson 4-2 the database tier pulls the same trick when a leader dies. Failover is one of a handful of patterns this course keeps reusing at different tiers.
Managed cloud load balancers do this for you, which is worth knowing before you build one. An AWS or GCP load balancer is already redundant across machines, so the design question becomes whether you trust the provider rather than how to configure a standby.