Python Cheatsheet
Generators and Iterators
Use this Python reference while you build software engineering projects, review code, or refresh the syntax you reach for most.
yield and Generator Functions
Any def containing yield is a generator function. Calling it runs no code, it
returns a generator object. Each yield hands a value out and freezes the frame
(locals, instruction pointer) until the next request.
def countdown(n): print("starting") while n > 0: yield n n -= 1 g = countdown(3) # nothing printed yet type(g) # <class 'generator'> next(g) # prints "starting", returns 3 next(g) # 2 list(g) # [1] (resumes from where it stopped) list(g) # [] exhausted, and it stays exhausted
Generator Expressions
Same as a list comprehension with parentheses instead of brackets. Use it when you only iterate once.
squares = (x * x for x in range(10)) # generator sum(x * x for x in range(10)) # 285, parens optional as sole arg max((len(w) for w in words), default=0) # Any/all short-circuit, so the generator stops at the first decisive item any(line.startswith("ERROR") for line in lines) all(x > 0 for x in values) # Gotcha: the OUTERMOST iterable is evaluated immediately, the rest lazily data = [1, 2, 3] g = (x for x in data) data = [9, 9] # rebinding does not matter, g already holds the old list list(g) # [1, 2, 3]
Laziness, One-Shot Consumption, next()
A generator is an iterator, so it is its own __iter__ and cannot be restarted
or indexed. len(), g[0], and a second for loop all fail or come back empty.
g = (x for x in range(3)) iter(g) is g # True len(g) # TypeError: object of type 'generator' has no len() g[0] # TypeError: 'generator' object is not subscriptable next(g, "default") # second arg avoids StopIteration on exhaustion next(iter([])) # StopIteration # Need it twice? Materialize, or build the generator from a callable rows = list(row_gen()) def rows_again(): return (r for r in source)
Inside a for loop the StopIteration is caught for you. Inside a generator,
a bare StopIteration from a nested next() call becomes a RuntimeError
(PEP 479, 3.7+), so guard it with a default or a try/except.
return Inside a Generator
return value stops the generator and stashes the value on the exception.
for loops discard it, yield from captures it.
def parse(lines): count = 0 for line in lines: if not line.strip(): return count # becomes StopIteration.value count += 1 yield line.strip() g = parse(["a", "b", "", "c"]) list(g) # ['a', 'b'] g = parse(["a", "b", "", "c"]) next(g); next(g) try: next(g) except StopIteration as e: print(e.value) # 2
yield from Delegation
yield from iterable forwards every item, and for a subgenerator it also
forwards send, throw, and close, and evaluates to that generator's return
value.
def flatten(nested): for item in nested: if isinstance(item, list): yield from flatten(item) # recursion, no manual inner loop else: yield item list(flatten([1, [2, [3, [4]], 5]])) # [1, 2, 3, 4, 5] def inner(): yield 1 yield 2 return "done" def outer(): result = yield from inner() # captures the return value yield result list(outer()) # [1, 2, 'done']
send, throw, close
yield is an expression, so a generator can receive values. This is the classic
coroutine form. Prime it with one next() (or send(None)) before the first
real send.
| Method | Effect |
|---|---|
next(g) / g.send(None) | resume, yield evaluates to None |
g.send(v) | resume, yield evaluates to v, returns the next yielded value |
g.throw(exc) | raise exc at the paused yield |
g.close() | raise GeneratorExit at the yield, then finish |
def accumulator(): total = 0 try: while True: x = yield total # yields total, receives the next x total += x finally: print(f"closing at {total}") g = accumulator() next(g) # 0 priming is required g.send(10) # 10 g.send(5) # 15 g.close() # prints "closing at 15" g.send(1) # StopIteration
Cleanup in a finally (or a with) runs on close() and on garbage collection,
so a generator that opens a file still closes it if abandoned mid-iteration.
The Iterator Protocol and Class-Based Iterators
An iterable has __iter__. An iterator has __iter__ and __next__ and
raises StopIteration when done. Write a class when you need extra state or
multiple independent passes.
class Fib: def __init__(self, limit): self.limit = limit def __iter__(self): # fresh iterator per loop, so reusable a, b = 0, 1 for _ in range(self.limit): yield a a, b = b, a + b list(Fib(7)) # [0, 1, 1, 2, 3, 5, 8] class Countdown: # explicit protocol, single-use def __init__(self, n): self.n = n def __iter__(self): return self def __next__(self): if self.n <= 0: raise StopIteration self.n -= 1 return self.n + 1
Infinite Generators and Memory
itertools.count never ends, so always bound it with islice, takewhile, or a
break.
from itertools import count, islice, takewhile, cycle, repeat list(islice(count(10, 5), 4)) # [10, 15, 20, 25] list(takewhile(lambda x: x < 30, count(0, 7))) # [0, 7, 14, 21, 28] list(islice(cycle("AB"), 5)) # ['A','B','A','B','A']
One million squares, measured with sys.getsizeof and tracemalloc:
| Form | Size |
|---|---|
[i * i for i in range(1_000_000)] | 8.1 MB pointer array, 38.6 MB peak allocation |
(i * i for i in range(1_000_000)) | 200 bytes, constant |
Time to first item is the other win: the generator yields immediately, the list comprehension builds all million entries before you see one.
Pipelines and Streaming Large Files
Chain generators so data flows one item at a time and peak memory stays flat no matter how large the input is.
def read_lines(path): with open(path, encoding="utf-8") as f: yield from f # file objects are already lazy def strip_blank(lines): for line in lines: line = line.rstrip("\n") if line: yield line def only_errors(lines): for line in lines: if "ERROR" in line: yield line def parse(lines): for line in lines: ts, _, msg = line.partition(" ERROR ") yield {"ts": ts, "msg": msg} # Nothing is read until the for loop pulls pipeline = parse(only_errors(strip_blank(read_lines("app.log")))) for record in pipeline: print(record["ts"], record["msg"]) # Fixed-size chunks from a binary stream def chunks(path, size=8192): with open(path, "rb") as f: while block := f.read(size): # walrus, 3.8+ yield block
Never
f.readlines()orf.read()on a file you cannot afford to hold in memory. Iterating the file object directly is the streaming form.