Poison messages and dead letter queues

When retrying is the wrong answer

Lesson 6-2 taught workers to retry because most failures are temporary: a network blip heals, an overloaded service recovers. But some jobs fail for reasons that never heal. A message referencing a video that was deleted, a job whose code path has a bug, an event in a format the worker cannot parse. Such a job is a poison message, and pure retry logic turns it into a small disaster: it fails, at-least-once delivery hands it out again, it fails again, forever. Workers burn time on it endlessly, and in queues that process in order it can block every job behind it. On-call engineers meet this one within their first year.

The standard defense has two parts:

1. Cap the attempts. Track how many times each message has been tried (real queue systems count this for you) and stop after a small number, like 3 or 5
2. Move it aside, never delete it. A message that exhausts its attempts goes to a dead letter queue (DLQ): a separate queue that humans inspect. The data is preserved for debugging, the main queue keeps flowing, and after the bug is fixed the dead letters can be replayed through the main queue

A worker loop with a poison message

Good jobs succeed on their first attempt, and the poison job exhausts its 3 attempts and is moved aside.

from collections import deque

MAX_ATTEMPTS = 3
queue = deque([("job1", "good"), ("job2", "poison"), ("job3", "good")])
dead_letter = []

while queue:
    name, kind = queue.popleft()
    attempts = 0
    done = False
    while attempts < MAX_ATTEMPTS and not done:
        attempts += 1
        if kind == "good":
            done = True
    if done:
        print(name, "succeeded on attempt", attempts)
    else:
        dead_letter.append(name)
        print(name, "failed", attempts, "times -> dead letter queue")

print("dead letter queue:", dead_letter)

Output

job1 succeeded on attempt 1
job2 failed 3 times -> dead letter queue
job3 succeeded on attempt 1
dead letter queue: ['job2']

Notice job3 still ran, since capping attempts is what kept the poison job from starving everything behind it. Without the cap the inner loop never exits and the third line never prints.

dead_letter.append(name) rather than a silent discard is the second half of the defense. The failing job is preserved for a human to inspect, which is what makes the failure debuggable instead of merely survivable.

The while attempts < MAX_ATTEMPTS and not done condition encodes both exit paths, meaning success or exhaustion. Both are normal outcomes, and treating exhaustion as an outcome rather than an error is what keeps the outer loop simple.

In a real queue the attempt counter lives on the message rather than in a local variable. Attempt 2 usually happens on a different worker minutes later, so the count has to travel with the job, and SQS and RabbitMQ both track it for you.

Note that the real DLQ is a queue rather than a list, which is what makes replay possible. Once the bug is fixed, the dead letters can be pushed back through the main queue and processed normally.

The cost of one poison message

Each poison message burns a full backoff schedule before reaching the DLQ.

attempts = 5
poison_per_day = 200
delay = 1
wasted_per_message = 0

for attempt in range(attempts):
    wasted_per_message += delay
    delay *= 2

print("Seconds wasted per poison message:", wasted_per_message)
print("Worker-seconds wasted per day:", wasted_per_message * poison_per_day)

Output

Seconds wasted per poison message: 31
Worker-seconds wasted per day: 6200

1 + 2 + 4 + 8 + 16 = 31, which is the same schedule as lesson 6-2 seen as a cost rather than a courtesy. Without any cap the sum has no end, which is the point of the exercise.

6,200 worker-seconds is about 1.7 hours of worker time per day spent on jobs that can never succeed. That is roughly the output of one full-time worker process, paid for and producing nothing.

The number gets worse as the failure rate rises, and that is the danger. A deploy that poisons 5% of a busy queue can consume most of the worker fleet, so a bug affecting a minority of jobs degrades all of them.

Note what the 31 seconds are mostly spent doing, which is waiting rather than working. A worker asleep in a backoff delay is usually not blocking anything, so the practical cost depends on whether the worker can pick up other jobs meanwhile.

That detail is worth knowing because it changes the fix. Workers that process jobs concurrently lose only capacity, and a strictly ordered queue loses everything behind the poison message, which is the far worse case.

What 5,000 dead letters of one job type means

A recent deploy, or an input format change, makes every resize job fail all its attempts.

A DLQ filling with one job type is a deploy-shaped signal. Something changed that makes that handler fail deterministically, and a single job type failing rules out the shared infrastructure that would take everything down with it.

The attempt cap is correctly routing the failures aside, which is worth appreciating in the moment. Five thousand messages in the DLQ is 5,000 jobs that did not loop forever, and the main queue is still moving for every other job type.

Ten minutes is the other clue in that description. A gradual rise suggests a data problem creeping in, and a sudden cliff points at a specific moment, which is almost always a deploy or a change in an upstream producer.

DLQ depth belongs on your dashboards next to queue depth from lesson 6-1. A DLQ that is normally empty makes an excellent alert, since any sustained nonzero value means something needs a human.

The recovery is the full poison life cycle: inspect a dead letter, fix the bug, then replay the DLQ through the main queue. Replay is safe because handlers are idempotent from lesson 6-2, so any job that partially succeeded before failing does not double-apply.

How many handler runs a poison message gets

Exactly 4, which is the attempt cap.

Every attempt fails, so the cap is reached exactly rather than approached. That makes the cap the ceiling on handler runs, and it is the only thing bounding the work a hopeless job can consume.

Without the cap, at-least-once delivery would redeliver the failing message forever. The queue cannot tell a permanent failure from a temporary one, so redelivery is its only reasonable behavior and the worker has to be the one that gives up.

With the cap, the message is preserved in the DLQ for a human while the main queue keeps moving. Both halves of that sentence matter, since discarding the message would keep the queue moving and destroy the evidence.

Picking the number is a judgment call between two costs. Too low and a genuinely transient failure, like a dependency restarting, gets dead-lettered unnecessarily, and too high and each poison message wastes the worker time from the previous block, so 3 to 5 is the usual range.

Cap, dead-letter, inspect, fix, replay is the complete life cycle of a poison message. Being able to name those five steps is a good answer to any interview question about queue reliability.