Generator Functions

If you're new to programming

A generator is a function that produces values one at a time instead of building an entire list. Think of it like a vending machine: it holds many items but only gives you one when you press the button. This saves memory because you never have all items out at once. Generators use the yield keyword instead of return.

If you've coded before

Generators are lazy iterators created with yield. They implement __iter__ and __next__ automatically. Generator expressions use parentheses: (x for x in items). This lesson covers generator functions, expressions, memory tradeoffs, and the next() built-in.

James writes a function that returns the word count of every note in a list:

def all_word_counts(notes: list[Note]) -> list[int]:
    return [note.word_count for note in notes]

"This works for 100 notes," Emma says. "What about a million notes?"

"A million integers in a list," James says. "That takes memory."

"How much memory does a list of a million integers take?"

James runs a quick test:

import sys
big_list = list(range(1_000_000))
print(f"List size: {sys.getsizeof(big_list):,} bytes")

Output:

List size: 8,448,728 bytes

"Eight megabytes for just the container," Emma says. "The integers themselves take more. Now imagine a million Note objects, each with a title, body, and tags."

"So I need a way to process notes without storing all results at once."

"You need a generator."

---

Your First Generator

A generator function uses yield instead of return:

from collections.abc import Iterator

def count_up_to(n: int) -> Iterator[int]:
    """Yield numbers from 1 to n."""
    i = 1
    while i <= n:
        yield i
        i += 1

Use it in a for loop:

for number in count_up_to(5):
    print(number)

Output:

1
2
3
4
5

It looks like a normal function that returns a list. But it does not build a list. It produces one value at a time.

---

How yield Works

A normal function runs from start to finish and returns one value. A generator function behaves differently: it pauses at each yield and resumes where it left off when you ask for the next value.

Here is how it works, step by step:

def simple_generator():
    print("Step 1")
    yield "A"
    print("Step 2")
    yield "B"
    print("Step 3")
    yield "C"

gen = simple_generator()

Calling simple_generator() does not run the function. It creates a generator object and waits. Nothing prints yet.

Now call next(gen) to get the first value:

print(next(gen))    # Runs the function until the first yield
print("---")
print(next(gen))    # Resumes from where it paused, runs until the second yield
print("---")
print(next(gen))    # Resumes again, runs until the third yield

Output:

Step 1
A
---
Step 2
B
---
Step 3
C

Follow what happens on each next() call:

1. First next(gen): Python runs the function body from the top. It prints "Step 1", hits yield "A", returns "A", and freezes the function right there. The function's local variables, its position in the code, everything is saved.
2. Second next(gen): Python unfreezes the function and resumes from the line after the first yield. It prints "Step 2", hits yield "B", returns "B", and freezes again.
3. Third next(gen): Same pattern. Resumes, prints "Step 3", yields "C", freezes.

If you call next(gen) a fourth time, there are no more yield statements, so Python raises StopIteration. A for loop handles this automatically: it calls next() repeatedly and stops when StopIteration occurs.

Concept	return	yield
Behavior	Function ends, returns one value	Function pauses, produces one value, resumes later
Memory	All results built at once	One result at a time
Usage	Call once, get result	Iterate, get values one by one
Type returned	The value itself	A generator object

---

Generators with SmartNotes

Now apply generators to SmartNotes. Before writing the function, you need to know the return type. A generator produces values one at a time, so it is not a list. Python calls this pattern an Iterator: something you can call next() on to get the next value. The type Iterator[Note] means "produces Note objects one at a time." You import it from collections.abc:

from dataclasses import dataclass, field
from collections.abc import Iterator  # Type for "produces values one at a time"


@dataclass
class Note:
    title: str
    body: str
    word_count: int
    author: str = "Anonymous"
    is_draft: bool = True
    tags: list[str] = field(default_factory=list)


def search_notes_lazy(
    notes: list[Note],
    keyword: str,
) -> Iterator[Note]:
    """Yield notes that contain the keyword in title or body.

    Unlike a list-returning function, this processes notes
    one at a time without building the full result list.
    """
    keyword_lower = keyword.lower()
    for note in notes:
        if keyword_lower in note.title.lower() or keyword_lower in note.body.lower():
            yield note

Use it:

notes = [
    Note("Python Tips", "Learn basics of coding", 50, "James", tags=["python"]),
    Note("Debug Guide", "How to fix Python errors", 120, "James", tags=["debug"]),
    Note("Cooking", "Boil water and add salt", 30, "Emma", tags=["cooking"]),
]

for note in search_notes_lazy(notes, "python"):
    print(f"Found: {note.title}")

Output:

Found: Python Tips
Found: Debug Guide

This generator yields each matching note as it finds it. If the list had a million notes, it would still use the same amount of memory because it processes one note at a time.

---

Generator Expressions

A generator expression is like a list comprehension but with parentheses instead of brackets:

# List comprehension: builds entire list in memory
word_counts_list = [note.word_count for note in notes]

# Generator expression: produces values on demand
word_counts_gen = (note.word_count for note in notes)

print(type(word_counts_list))
print(type(word_counts_gen))

Output:

<class 'list'>
<class 'generator'>

The generator expression does not compute anything until you iterate over it. This makes it ideal for passing to functions that consume values one at a time:

# sum() consumes the generator one value at a time
total = sum(note.word_count for note in notes)
print(f"Total words: {total}")

# max() finds the largest without building a list
longest = max(note.word_count for note in notes)
print(f"Longest note: {longest} words")

# any() stops at the first True value
has_draft = any(note.is_draft for note in notes)
print(f"Has drafts: {has_draft}")

Output:

Total words: 200
Longest note: 120 words
Has drafts: True

Notice: when passing a generator expression as the only argument to a function, you can skip the extra parentheses. sum(x for x in items) instead of sum((x for x in items)).

---

When to Use Generators vs Lists

Use a list when	Use a generator when
You need to access items multiple times	You only iterate once
You need to know the length	Length is not needed
You need random access (items[5])	Sequential access is enough
The dataset is small	The dataset is large or unknown
You need to pass it to a function that requires a list	You pass it to sum(), max(), any(), all()

A common mistake: trying to iterate over a generator twice.

gen = (note.title for note in notes)

# First iteration works
for title in gen:
    print(title)

# Second iteration produces nothing!
for title in gen:
    print(title)  # Never prints

Generators are exhausted after one pass. If you need to iterate twice, use a list or create a new generator.

---

Chaining Generators

Generators compose naturally. Chain them to build processing pipelines:

def published_notes(notes: list[Note]) -> Iterator[Note]:
    """Yield only published notes."""
    for note in notes:
        if not note.is_draft:
            yield note


def long_notes(notes: Iterator[Note], min_words: int) -> Iterator[Note]:
    """Yield notes with at least min_words words."""
    for note in notes:
        if note.word_count >= min_words:
            yield note


# Chain: published notes → filter by length → get titles
pipeline = long_notes(published_notes(notes), min_words=50)
for note in pipeline:
    print(f"{note.title}: {note.word_count} words")

Output:

Debug Guide: 120 words

Each generator processes one note at a time. No intermediate lists are created. The entire pipeline runs with constant memory regardless of how many notes you have.

---

PRIMM-AI+ Practice: Predict the Generator

Predict [AI-FREE]

Press Shift+Tab to enter Plan Mode.

def evens(limit):
    n = 0
    while n < limit:
        yield n
        n += 2

gen = evens(10)
print(next(gen))
print(next(gen))
print(list(gen))

What does this print? Three separate outputs.

Check your predictions

0
2
[4, 6, 8]

next(gen) gets 0, then 2. list(gen) consumes the remaining values (4, 6, 8). The generator remembers where it left off after each next() call.

Run

Press Shift+Tab to exit Plan Mode.

Create gen_practice.py and verify your predictions.

Investigate

Write one sentence explaining why list(gen) in the Predict exercise returned [4, 6, 8] instead of [0, 2, 4, 6, 8]. Where did the first two values go?

If you want to go deeper, run /investigate @gen_practice.py in Claude Code and ask: "What happens when I call next() on an exhausted generator? What is StopIteration and how does a for loop handle it?"

Modify

Write a generator word_count_stream(notes: list[Note]) -> Iterator[int] that yields the word count of each note. Use it with sum() to compute the total and max() to find the longest note.

Make [Mastery Gate]

Write a generator function read_notes_from_directory(directory: Path) -> Iterator[Note] that scans a directory for .md files, reads each one, parses it into a Note (using your read_note_from_markdown from Chapter 62), and yields it. In Claude Code, type /tdg to guide you through the cycle:

1. Write the stub with types
2. Write 3+ tests (empty directory, one file, multiple files)
3. Prompt AI to implement
4. Verify with ruff, pyright, pytest

---

Try With AI

Opening Claude Code

If Claude Code is not already running, open your terminal, navigate to your SmartNotes project folder, and type claude. If you need a refresher, Chapter 44 covers the setup.

Prompt 1: Memory Comparison

Compare the memory usage of a list comprehension vs a
generator expression for processing 1 million notes.
Show me a benchmark with actual memory measurements.

What you're learning: Generators save memory. The AI quantifies the savings so you understand when generators matter (large datasets) vs. when they do not (small lists).

Prompt 2: Generator Pitfalls

What are the most common mistakes when using generators
in Python? Show me examples of each mistake and how to
fix it.

What you're learning: Generators have traps: single-pass iteration, no length, no indexing. Knowing the pitfalls prevents debugging sessions.

Prompt 3: Infinite Generator

Show me how to write an infinite generator that produces
an endless sequence of IDs (1, 2, 3, ...). How do I use
it safely without running forever? Show me with itertools.islice.

What you're learning: Generators can be infinite because they produce values on demand. This is a powerful pattern for ID generation, pagination, and streaming. You learn to control infinite sequences with islice.

---

James chains two generators together: filter published notes, then filter by word count. The pipeline processes his three notes without building any intermediate lists.

"At the warehouse, we had a conveyor belt," he says. "Packages moved along the belt one at a time. Each station inspected one package, applied a label, and passed it forward. The belt never stopped to collect all packages in one spot. That is what generators do."

Emma nods. "Comprehensions are the warehouse floor: spread everything out, pick what you need, box it up. Generators are the conveyor belt: process items as they flow through, never storing more than one at a time."

"I have comprehensions for building collections and generators for streaming data," James says. "But I keep writing the same patterns: filter, transform, sort. Python must have built-in tools for these."

"It does. map(), filter(), sorted(). Plus lambda for quick one-off functions and lru_cache for remembering results. Lesson 4: functional tools."