The iteration protocol

One idea from lesson 3-3 sets up this whole unit. When you write len(p), Python calls p.__len__() behind the scenes, routing the built-in function through a dunder method so that any class can support it.

for loops work the same way. They secretly call two dunder methods, __iter__ and __next__, and that is what this lesson unpacks.

What a for loop really does

This unit looks under the hood of for, and the payoff is threefold: it explains a whole family of real-world bugs (loops that mysteriously run empty), it lets your own classes work with for, and it is the foundation of generators, the tool professionals use to process data too large to hold in memory.

You have looped over lists, strings, and dicts. One for statement handles all of them because they all follow a single contract, the iteration protocol:

1. iter(thing) asks for an iterator, an object that remembers a position.
2. next(iterator) returns the next value.
3. When the values run out, next raises the special StopIteration exception.

So this loop:

for x in [10, 20]:
    print(x)

is really:

it = iter([10, 20])
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

Anything that can hand out an iterator is called an iterable. That is the only requirement, which is why for works on types the language shipped with and on classes you write yourself, with no special cases.

Driving a loop by hand

Calling iter and next directly does the same work a for loop does, just with the machinery visible.

letters = ["a", "b", "c"]

it = iter(letters)
print(next(it))
print(next(it))
print(next(it))

try:
    next(it)
except StopIteration:
    print("exhausted")

Output

a
b
c
exhausted

The fourth next finds nothing left and raises StopIteration, which the try block catches. A for loop catches that same exception for you and quietly ends the loop, which is why you normally never see it.

Iterators are one-shot

An iterator moves forward only. Once exhausted, it stays empty, you must call iter again for a fresh pass. This explains a bug you will absolutely hit someday: looping twice over something that is an iterator and finding the second loop does nothing.

A concrete case. zip(a, b) is a built-in that pairs up two sequences item by item, first with first, second with second. It returns an iterator over those pairs, not a list:

pairs = zip([1, 2], ["a", "b"])
print(list(pairs))   # [(1, 'a'), (2, 'b')]
print(list(pairs))   # [] , already used up

Open files behave the same way: in unit 8 you will see that a file object hands out its lines exactly once per opening.

Lists are iterables but not iterators: each for loop asks them for a brand-new iterator, so you can loop over a list as many times as you like.

An iterator only pays out once

zip returns an iterator, not a list, and that catches people out. The first list(...) call drains it completely, so the second finds nothing left and returns an empty list.

pairs = zip([1, 2, 3], ["a", "b", "c"])

print(list(pairs))
print(list(pairs))

Output

[(1, 'a'), (2, 'b'), (3, 'c')]
[]

This is the source of the empty-second-loop bug mentioned at the top of the lesson. When you need the pairs more than once, store list(zip(...)) in a variable and reuse that list, since a real list can be walked as many times as you like.

Taking just the first two values

iter and next are useful on their own when you want a few values off the front of a sequence without looping over all of it.

words = ["spam", "eggs", "toast", "jam"]

it = iter(words)
print(next(it))
print(next(it))
print(len(words))

Output

spam
eggs
4

it = iter(words) produces the iterator, and each next(it) pulls exactly one value from it. The list itself is never modified by any of this, only the iterator's position moves, which is why len(words) still reports all 4 words at the end.

Iterators never rewind

With it = iter([1]), the first next(it) returns 1 and the second raises StopIteration.

The single value was consumed by the first call, and an iterator has no way to go back or start over. Exhausted is a permanent state for it. That raised StopIteration is not an error to be feared, it is precisely the signal a for loop watches for to know it has reached the end.