I still remember the first time a generator saved a script from getting OOM-killed. I was processing a multi-gigabyte log file, and the “obvious” version of the code read every line into a list before filtering it. It worked fine on my laptop with a 10MB sample file and then fell over the moment it hit the real dataset in production. Swapping a list comprehension for a generator expression fixed it in about thirty seconds. That asymmetry, huge payoff for a tiny syntax change, is why generators are one of the first things I reach for when data gets big or infinite.

Iterators: the protocol underneath the loop

Every for loop in Python is built on the iterator protocol, two dunder methods:

  • __iter__(self) returns an iterator object (often self).
  • __next__(self) returns the next value, or raises StopIteration when exhausted.

An iterable is anything with __iter__. An iterator is an iterable that also tracks its own position via __next__. Lists, dicts, and strings are iterable, but they aren’t iterators themselves, calling iter() on them produces a fresh iterator object each time.

numbers = [1, 2, 3]
it = iter(numbers)
print(next(it))  # 1
print(next(it))  # 2
print(next(it))  # 3
next(it)  # raises StopIteration

You can write this protocol by hand:

class CountUp:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current

for n in CountUp(3):
    print(n)  # 1, 2, 3

It works, but it’s a lot of ceremony for something conceptually simple: “count up to a limit.” This is exactly the boilerplate generators exist to eliminate.

Generators: functions that pause

A generator function looks like a normal function except it uses yield instead of (or alongside) return. Calling it doesn’t run the function body, it returns a generator object that implements the iterator protocol for you.

def count_up(limit):
    current = 0
    while current < limit:
        current += 1
        yield current

for n in count_up(3):
    print(n)  # 1, 2, 3

Each call to next() resumes the function right after the last yield, runs until the next yield (or the function returns, which raises StopIteration), and hands back the yielded value. Local variables persist between resumes, the function’s stack frame is frozen, not discarded. This is the whole trick: a generator is a function with a bookmark.

You can watch the pausing happen:

def talkative():
    print("starting")
    yield 1
    print("resumed after first yield")
    yield 2
    print("resumed after second yield")

gen = talkative()  # nothing printed yet, no code has run
print(next(gen))
# starting
# 1
print(next(gen))
# resumed after first yield
# 2

Nothing executes until you pull a value. That laziness is the whole point.

Why lazy evaluation matters

A list comprehension builds the entire collection in memory before you use any of it. A generator expression, same syntax with parentheses instead of brackets, produces values on demand:

squares_list = [x * x for x in range(10_000_000)]   # builds a 10M-element list now
squares_gen  = (x * x for x in range(10_000_000))    # builds nothing yet

squares_list allocates real memory immediately (hundreds of megabytes for large ranges). squares_gen is a small object that computes each value only when asked. If you only need the first three squares, or you’re going to filter most of them out anyway, the generator never does the wasted work.

This matters for two overlapping reasons:

  1. Memory: you never hold more than one item (plus whatever state you keep) in memory at a time.
  2. Time-to-first-result: a generator can start producing output before the full computation would even finish, which matters for pipelines and infinite sequences.

Generators can also represent sequences that have no fixed end, an infinite counter, a stream of sensor readings, a live log tail, none of which could ever be materialized as a list.

def natural_numbers():
    n = 1
    while True:
        yield n
        n += 1

evens = (n for n in natural_numbers() if n % 2 == 0)
first_five = [next(evens) for _ in range(5)]
print(first_five)  # [2, 4, 6, 8, 10]

Try that with a list comprehension and you’ll wait forever (and run out of RAM first).

yield from: delegating to sub-generators

When a generator needs to yield everything from another iterable, yield from saves you a manual loop and correctly forwards StopIteration values, sent values, and exceptions:

def chain(*iterables):
    for iterable in iterables:
        yield from iterable

for item in chain([1, 2], "ab", (True, False)):
    print(item)
# 1, 2, 'a', 'b', True, False

This is essentially what itertools.chain does, and it’s the idiomatic way to flatten one level of nested generators without breaking laziness.

Generators are single-use

A generator object is exhausted after one full pass. Unlike a list, you can’t iterate it twice:

gen = (x for x in range(3))
print(list(gen))  # [0, 1, 2]
print(list(gen))  # [] -- already exhausted

If you need to iterate multiple times, either call the generator function again to get a fresh generator, or materialize the results into a list if you genuinely need random access or repeated passes. This trips up almost everyone at least once, I’ve debugged a “mysteriously empty” second loop more than once, and the fix is always the same: remember that a generator is a one-shot iterator, not a reusable collection.

Real-world pattern: streaming a large file without loading it fully

Reading a file line-by-line is already lazy in Python, files are iterators, but combining that with generator pipelines lets you build multi-stage processing without ever holding the full file in memory:

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def parse_log_lines(lines):
    for line in lines:
        parts = line.split(" ", 2)
        if len(parts) == 3:
            timestamp, level, message = parts
            yield {"timestamp": timestamp, "level": level, "message": message}

def filter_errors(records):
    for record in records:
        if record["level"] == "ERROR":
            yield record

def error_report(path):
    lines = read_lines(path)
    records = parse_log_lines(lines)
    errors = filter_errors(records)
    for error in errors:
        print(f"{error['timestamp']}: {error['message']}")

error_report("app.log")

Each stage pulls one line through the whole pipeline before asking for the next. On a 50GB log file this uses roughly the same tiny amount of memory as a 5KB one. Compare that to the tempting-but-wrong version that does lines = open(path).readlines() followed by three list comprehensions, that version needs the whole file, then a full copy of it, then another copy, in memory simultaneously.

Real-world pattern: paginated API results as a single stream

APIs that paginate results are a natural fit for generators, callers shouldn’t have to know or care about page boundaries:

import requests

def fetch_all_users(base_url):
    url = f"{base_url}/users"
    while url:
        response = requests.get(url)
        response.raise_for_status()
        data = response.json()
        yield from data["results"]
        url = data.get("next")  # None when there's no next page

for user in fetch_all_users("https://api.example.com"):
    print(user["email"])
    if user["email"].endswith("@spamdomain.test"):
        break  # stops making HTTP requests immediately

This is the pattern I use most in day-to-day work. The break is doing real work here: because fetch_all_users is lazy, stopping the loop early means the generator never issues the next page’s HTTP request. A version that returned a full list of every user would have to fetch every page up front, even if the caller only wanted the first three matches.

Real-world pattern: generator-based batching

Batching is a recurring need (bulk database inserts, API rate limits, GPU batch inference) and generators make it composable:

from itertools import islice

def batched(iterable, size):
    it = iter(iterable)
    while batch := list(islice(it, size)):
        yield batch

def insert_records(records, batch_size=500):
    for batch in batched(records, batch_size):
        db.bulk_insert(batch)  # one round-trip per batch, not per record

insert_records(read_and_parse("huge_export.csv"))

batched works on any iterable, including an infinite one, and never buffers more than batch_size items at once. (Python 3.12+ ships itertools.batched directly, but the hand-rolled version is worth understanding since you’ll see it in older codebases.)

Generator expressions vs generator functions

Use a generator expression for a single, simple transformation:

total = sum(x * x for x in range(1000) if x % 2 == 0)

Reach for a generator function when there’s real logic, multiple steps, state that persists across iterations, or a name that documents intent:

def running_average(values):
    total = 0
    count = 0
    for value in values:
        total += value
        count += 1
        yield total / count

If you find yourself writing a generator expression with more than one if or a nested comprehension just to avoid “using a function,” stop, a named generator function reads better every time.

Summary

If you take one thing from this: reach for a generator whenever you’re tempted to build a list you’re only going to loop over once. It’s rarely slower, it’s often dramatically lighter on memory, and it composes beautifully with the rest of the pipeline.

Going back to that log file from the intro: the fix really was that small. The original code was a readlines() call feeding a couple of list comprehensions, three full copies of a multi-gigabyte file sitting in memory at once, which is exactly why it worked on a sample file and died on the real thing. Rewriting it as the read_lines to parse_log_lines to filter_errors pipeline from the pattern above dropped peak memory from “more RAM than the box had” to a few megabytes, because at any given moment the process only ever holds one line. No architecture change, no new dependency, just yield instead of return and letting the data flow through one record at a time.