Work fails, so we retry
A worker pulls a job, calls a payment provider, and the network drops (as How the Internet Works warned it would). Did the charge happen? You often cannot know. The queue's answer is blunt: if a worker does not confirm a job finished, the job is handed out again. This is at-least-once delivery, and it is the standard guarantee, because the alternative risks silently losing jobs.
At-least-once has a sharp edge: the same job can run twice. If the first charge actually succeeded and the retry charges again, a customer just paid double.
The cure is idempotency: designing an operation so that running it twice has the same effect as running it once. Setting balance = 80 is idempotent. balance = balance - 20 is not. Deleting a row is idempotent. Appending a row is not.
Code exercise · python
Run this idempotent payment handler. Every event carries a unique ID. The handler records processed IDs in a set and skips any ID it has seen, so the retried delivery of evt_1 is harmless.
Retry politely: exponential backoff
When a dependency fails, retrying instantly and forever makes things worse: a struggling service gets hammered by every client at once, a self-inflicted flood called a retry storm.
The standard discipline is exponential backoff: wait 1 second before the first retry, then 2, then 4, then 8, doubling each time, and give up after a few attempts. Failures get time to heal, and load on the sick service drops instead of spiking. Production systems also add jitter, a little randomness in each delay, so a thousand clients do not all retry in the same instant.
You saw this schedule from the receiving side in lesson 4-2's failover timing. Now compute it from the sender's side.
Code exercise · python
Your turn: print an exponential backoff schedule. Start with delay = 1 second and double after each retry, for 5 retries. Track the cumulative wait and print one line per retry exactly as shown.
Problem
A queue delivers a send-welcome-email job at least once, and a bug causes it to be delivered 3 times. The worker's handler is idempotent, keyed on the job ID. How many welcome emails does the user receive?