Course outline · 0% complete

0/29 lessons0%

Course overview →

Health checks and staying alive

lesson 9-2 · ~10 min · 26/29

Running is not the same as working

A running container is not proof of a working app, because the process can be alive while the app inside is deadlocked or unable to reach its database. Lesson 5-2 met this as started is not ready.

Every platform from lesson 9-1 therefore asks the app the same thing on repeat, which is whether it is healthy.

The convention is a health endpoint, a URL such as /health that answers 200 OK quickly after checking its own vitals, including database reachability. Declaring it in compose looks like this:

  app:
    build: .
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 10s
      retries: 3
    restart: unless-stopped
KeyEffect
testthe probe command, run inside the container
interval: 10sprobe every ten seconds
retries: 3mark unhealthy only after three consecutive failures
restart: unless-stoppedDocker restarts the container if it dies

curl -f exits non-zero on HTTP errors, so this is exit codes again from lesson 7-1. The restart policy is what keeps a crash at 3am from waiting for a human.

Health checks power everything else

Once the platform can tell healthy from unhealthy, the patterns from earlier lessons click together.

FeatureWhat the health check enables
compose startup order, lesson 5-2condition: service_healthy waits for the db probe to pass
rolling and blue-green deploys, lesson 8-3traffic shifts only to containers that probe healthy
self-healingrestart policies and orchestrators replace unhealthy containers

An unhealthy new version therefore aborts a rollout automatically, with no human watching a dashboard.

One subtlety is worth naming. Platforms retry probes before declaring death, because a single slow response might be a blip rather than a failure. The next block simulates that retry loop.

A prober that retries before giving up

A loop standing in for a platform probing a service that needs a moment to warm up.

for attempt in 1 2 3; do
  if [ "$attempt" -lt 3 ]; then
    echo "attempt $attempt: no response, retrying"
  else
    echo "attempt $attempt: healthy"
  fi
done
echo "container marked healthy"

Output

attempt 1: no response, retrying
attempt 2: no response, retrying
attempt 3: healthy
container marked healthy
AttemptResultVerdict so far
1no responsekeep trying
2no responsekeep trying
3healthymark healthy

The first probes fail and the prober retries rather than declaring the container dead on attempt 1. Without retries, every slow start would look like an outage, and -lt is the numeric less-than test bash uses inside [ ].

A rollout that stops itself

When the new version's containers never pass their health checks, a well-configured platform halts the rollout and keeps the old version serving.

This is the payoff of lesson 8-3 combined with health checks. Traffic only moves to containers that prove themselves healthy, so a bad build fails its rollout without users ever reaching it.

Container stateReceives traffic
old version, healthyyes, throughout
new version, never healthyno

The old version keeps serving the whole time, so no rollback is even needed. The failed deploy shows up as a red pipeline rather than as an incident, which is the entire goal.

The condition that waits for readiness

The condition is service_healthy, and the word describes exactly what the health check proves.

With a healthcheck on the db service, depends_on: db: condition: service_healthy makes compose hold the app back until the database probe passes.

ConditionCompose waits for
service_startedthe container to start
service_healthythe healthcheck to pass

That closes the started-versus-ready trap from lesson 5-2. It also means the db service must actually define a healthcheck, since without one the condition has nothing to wait on.