Python Iterators and Generators Deep Dive

Why Lazy Evaluation Changes Everything

Before we dive into the mechanics, let's talk about why any of this matters. Most beginners think of computation as eager: you write an expression, Python evaluates it immediately, and you get a result. That's how x = 2 + 2 works, and that's how my_list = [x**2 for x in range(1000)] works, it runs the loop, builds the list, and hands you a fully populated object. For small data, this is totally fine. For real-world data, it can destroy you.

Lazy evaluation flips that model on its head. Instead of computing everything upfront and storing it, you describe how to compute values and defer the actual work until someone asks for it. Think of it like a recipe versus a finished meal. A recipe takes up one page; the meal takes up a whole table. If someone only eats one bite, you wasted everything you cooked. With lazy evaluation, you only cook what gets eaten.

This philosophy runs deep in Python's standard library. The range() function is lazy, it doesn't build a list of a million numbers, it just knows the start, stop, and step and generates each number on demand. The map() and filter() builtins are lazy. Reading lines from a file object is lazy. Python was designed from the ground up to favor this approach because the language's authors understood that the most common bottleneck in data-heavy programs isn't CPU speed, it's memory. A program that runs in constant memory can process a dataset of any size. A program that loads everything into RAM hits a wall the moment your data outgrows your machine.

When you internalize lazy evaluation, you start writing fundamentally different code. You stop thinking "load everything, then process" and start thinking "define the pipeline, then pull values through it." Generators are the tool that makes this mental model concrete and executable. They're not an advanced trick for experts, they're the idiomatic Python way to handle anything that involves sequences of data. By the end of this article, you'll reach for generators automatically, and your programs will be leaner, faster, and more scalable because of it.

The Iterator Protocol: What's Really Happening

Every time you loop in Python, you're relying on the iterator protocol. It's simple but critical:

1. An object implements __iter__(), which returns an iterator object
2. That iterator implements __next__(), which returns the next value
3. When there are no more values, __next__() raises StopIteration

These two methods form a contract between your object and Python's for loop machinery. Any object that satisfies this contract can be iterated, it doesn't matter whether it wraps a list, reads from a network socket, or generates values mathematically. The for loop doesn't care; it just calls the protocol.

my_list = [1, 2, 3]
iterator = iter(my_list)  # Calls __iter__()
 
print(next(iterator))  # Calls __next__() → 1
print(next(iterator))  # 2
print(next(iterator))  # 3
print(next(iterator))  # StopIteration raised

Your for loop does exactly this under the hood. It calls iter() to get an iterator, then repeatedly calls next() until StopIteration is raised.

Here's the insight: the list itself doesn't move forward, the iterator does. The list stays pristine. The iterator maintains state about where you are in the sequence. This design is intentional, it decouples the data container from the traversal mechanism, which gives you enormous flexibility.

my_list = [1, 2, 3]
 
# You can create multiple independent iterators
iter1 = iter(my_list)
iter2 = iter(my_list)
 
print(next(iter1))  # 1
print(next(iter2))  # 1
print(next(iter1))  # 2
print(next(iter2))  # 1 (still independent)

This separation matters. It means iterating doesn't mutate the source, and you can iterate multiple times over the same collection. It also means you can hand the same list to two different consumers and they won't interfere with each other, each holds its own cursor into the data.

Iterator Protocol Internals

Understanding what Python does inside the for loop helps you write better iterators and debug them when they misbehave. When Python executes for item in obj, it first calls iter(obj), which triggers obj.__iter__(). This must return an iterator, an object with a __next__ method. Then Python enters a loop: it calls next(iterator), which triggers iterator.__next__(), assigns the return value to item, and runs the loop body. When __next__ raises StopIteration, Python catches it silently and exits the loop.

This means the StopIteration exception is not an error, it's a signal. You should never suppress it manually in normal iterator code, because that would break the protocol. The one exception is inside generator functions, where raising StopIteration explicitly was once valid but is now deprecated in favor of just letting the function return.

A subtle but important detail: __iter__ and __next__ can live on different objects. A container like a list implements __iter__ to return a separate list_iterator object. That list_iterator implements __next__. This means the list is an iterable but not an iterator, you can get iterators from it, but it doesn't advance itself. An iterator, by contrast, must implement both methods, typically with __iter__ returning self. Why does this matter? Because iterables are reusable and iterators are one-shot. You can loop over a list forever; you can only loop over a list_iterator once.

The practical implication is that when you build a custom class, you need to decide: do I want reusability or single-pass streaming? If reusable, return a new iterator from __iter__. If single-pass, return self from __iter__ and manage your own state. Getting this distinction wrong is one of the most common iterator bugs in production code.

Building Custom Iterators with Classes

Now let's build our own iterator. Say you want to count up from a number to a limit. Writing this as a class makes the protocol concrete, you can see exactly where state lives and how it advances.

class CountUp:
    def __init__(self, max_value):
        self.max_value = max_value
        self.current = 0
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.current < self.max_value:
            self.current += 1
            return self.current
        else:
            raise StopIteration
 
# Usage
counter = CountUp(5)
for num in counter:
    print(num)  # 1, 2, 3, 4, 5

Notice the pattern: __iter__() returns self (because the class itself is the iterator), and __next__() updates state and returns values until it raises StopIteration. The state, self.current, lives on the instance, which is why it persists between calls to __next__.

Let's build something more interesting: a Fibonacci iterator. This is a classic example because the sequence can't be expressed as a simple range, yet it has clean, stateful logic that maps naturally to the iterator pattern.

class FibonacciIterator:
    def __init__(self, limit):
        self.limit = limit
        self.a, self.b = 0, 1
        self.count = 0
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.count >= self.limit:
            raise StopIteration
        value = self.a
        self.a, self.b = self.b, self.a + self.b
        self.count += 1
        return value
 
# Usage
for fib in FibonacciIterator(10):
    print(fib, end=' ')  # 0 1 1 2 3 5 8 13 21 34

This works, but it's verbose. Here's where generators save our sanity.

Generator Functions: The Elegant Alternative

A generator is a function that uses yield instead of return. When you call a generator, it doesn't execute immediately. Instead, it returns a generator object, which is an iterator. Python handles all of the __iter__ and __next__ bookkeeping for you automatically, which means you can focus entirely on the logic of producing values.

def count_up(max_value):
    current = 0
    while current < max_value:
        current += 1
        yield current
 
# Usage
for num in count_up(5):
    print(num)  # 1, 2, 3, 4, 5

Compared to the class version, this is cleaner and far more readable. The yield statement pauses execution, returns a value, and saves state. The next time you call next(), execution resumes right after the yield. The function's entire local frame, every variable, the current line number, everything, is preserved in the generator object while it's suspended.

Let's trace through it carefully, because this suspension behavior is what makes generators so powerful:

def simple_gen():
    print("Starting")
    yield 1
    print("Between 1 and 2")
    yield 2
    print("Between 2 and 3")
    yield 3
    print("Done")
 
gen = simple_gen()
print(next(gen))  # prints "Starting", yields 1
print(next(gen))  # prints "Between 1 and 2", yields 2
print(next(gen))  # prints "Between 2 and 3", yields 3
print(next(gen))  # prints "Done", raises StopIteration

See how the function pauses at each yield? That's the magic. The function's local state (variables, execution position) is preserved between yields. This is suspension, and it's incredibly powerful. The code between yield statements can be as complex as you need: it can call other functions, access databases, perform calculations, or do anything else a regular function can do.

Now let's rewrite Fibonacci as a generator. Compare this side by side with the class version above and notice how much noise disappears:

def fibonacci(limit):
    a, b = 0, 1
    count = 0
    while count < limit:
        yield a
        a, b = b, a + b
        count += 1
 
for fib in fibonacci(10):
    print(fib, end=' ')  # 0 1 1 2 3 5 8 13 21 34

Much cleaner. No class, no __iter__ and __next__, no explicit StopIteration. Python handles all that for you. The generator function reads like a description of the sequence, not like an implementation of a protocol.

Generator Memory Benefits

The memory story for generators is one of the most compelling reasons to use them, and it's worth dwelling on the numbers. When you create a list comprehension like [x**2 for x in range(1_000_000)], Python allocates a contiguous block of memory for one million Python integer objects, plus the list's internal bookkeeping. On a 64-bit system, each Python integer takes at least 28 bytes, and the list pointer array takes 8 bytes per entry. That's roughly 36 megabytes for a million squared integers, just sitting in RAM, even if you only ever look at the first ten.

A generator expression over the same range takes around 128 bytes, regardless of how large the range is. That constant memory footprint is the key insight. Whether you're generating ten values or ten billion, the generator object itself stays the same size because it only holds the recipe for producing values, not the values themselves. The computation happens one step at a time, the value is yielded, and then it's gone from the generator's perspective, your code holds the reference, not the generator.

This becomes critical in machine learning and data pipelines. A dataset of 10 million training samples at even 1KB per sample is 10GB. Loading that into a list before training starts would exhaust the RAM on most machines before a single gradient step runs. But yielding samples one by one, or in small batches, means your training loop can process arbitrarily large datasets with a working set that fits in cache. PyTorch's DataLoader and TensorFlow's tf.data pipelines are both built on this principle, and they expose generator-like interfaces for exactly this reason. When you write a custom dataset class that implements __iter__, you're plugging directly into this pattern.

The other memory benefit is garbage collection. When a generator yields a value and your code processes it, the generator no longer holds a reference to it. If nothing else in your code does either, the garbage collector can reclaim that memory immediately. A list holds references to all its elements until the entire list goes out of scope. For a streaming pipeline processing millions of records, this difference in GC pressure can be the difference between stable, predictable memory usage and a slow memory leak that eventually forces a restart.

Generator Expressions: Lazy List Comprehensions

You know list comprehensions:

squares = [x**2 for x in range(1000000)]

This creates a list with a million items in memory. All at once. If your range is huge, that's wasteful, especially if the next step in your program only needs to sum those values or filter them down to a handful.

A generator expression looks almost identical but uses parentheses:

squares = (x**2 for x in range(1000000))

Now squares is a generator object. It doesn't compute anything until you ask for it. The parentheses signal to Python that you want a lazy iterator, not an eager list. This single syntactic choice, square brackets versus parentheses, determines whether your code loads a million integers into RAM or computes them one at a time.

squares = (x**2 for x in range(1000000))
print(next(squares))  # 0
print(next(squares))  # 1
print(next(squares))  # 4

The differences are dramatic:

import sys
 
# List comprehension
list_comp = [x**2 for x in range(100000)]
print(f"List size: {sys.getsizeof(list_comp)} bytes")  # ~800KB+
 
# Generator expression
gen_expr = (x**2 for x in range(100000))
print(f"Generator size: {sys.getsizeof(gen_expr)} bytes")  # ~128 bytes

The generator expression is tiny because it doesn't store the results. It computes them on demand. This is lazy evaluation, and it's essential when you're working with streaming data or datasets that don't fit in memory. Any time you're writing a list comprehension only to immediately pass it to sum(), max(), any(), all(), or a loop, you should strongly consider converting it to a generator expression instead.

The Memory Trade-off Explained

Here's why this matters in the real world:

List: All data in memory. Fast random access. Big memory footprint.

Generator: One value at a time. Slower (you can't jump to index 999999), but constant memory.

Use generators when:

• Data is too large to fit in memory
• You only need to iterate once
• You're building a pipeline where each step processes one item at a time

Use lists when:

• You need random access (e.g., items[42])
• You need the length (len() doesn't work on generators)
• You iterate multiple times
• The dataset is small enough to fit comfortably in memory

The trade-off is not just about raw memory, it's about the shape of your access pattern. Lists are random-access data structures; generators are sequential streaming structures. Trying to use a generator as a list will frustrate you. Accepting that distinction and designing your code around sequential passes will unlock generators' full power.

Lazy Evaluation: The Philosophy

Lazy evaluation means "don't compute anything until you absolutely need it." It's a core philosophy in Python, especially with generators.

Consider reading a 10GB log file:

# Without generators (bad for huge files)
def read_logs_all():
    with open('huge.log') as f:
        lines = f.readlines()  # Entire file in memory!
    return lines
 
# With generators (good for huge files)
def read_logs_lazy():
    with open('huge.log') as f:
        for line in f:
            yield line.strip()
 
# Usage
for log_entry in read_logs_lazy():
    print(log_entry.strip())  # Process one line at a time

The file reading is already lazy in the second approach. By wrapping it in a generator, you're being explicit about it and enabling chaining (coming next). The with block stays open as long as the generator is alive, which means the file connection is properly managed throughout the iteration.

yield from Explained

The yield from statement, introduced in Python 3.3, is one of the more underappreciated features in the generator toolbox. On the surface it looks like syntactic sugar, but it does something deeper than just save a few lines. When you yield from another iterable, you're establishing a transparent delegation channel between your generator and the inner iterable, not just forwarding values, but also forwarding .send() calls, .throw() calls, and the final return value.

The difference becomes clear when you look at what yield from replaces. The manual delegation approach loops over the inner generator and re-yields each value. It works for simple cases, but it breaks the two-way communication that generators support when used as coroutines. Any values sent into the outer generator via gen.send(value) would need to be explicitly forwarded to the inner generator, and the inner generator's return value, accessible via the StopIteration exception's value attribute, would be silently dropped. yield from handles all of this automatically.

Sometimes a generator needs to delegate to another generator. You could do this:

def inner():
    yield 1
    yield 2
 
def outer_bad():
    for value in inner():
        yield value
 
for val in outer_bad():
    print(val)  # 1, 2

But Python 3.3+ gives us yield from:

def inner():
    yield 1
    yield 2
 
def outer_good():
    yield from inner()
 
for val in outer_good():
    print(val)  # 1, 2

It's the same result, but cleaner. More importantly, yield from handles two-way communication. If the delegated generator returns a value, yield from passes it back:

def inner():
    result = yield 1
    result = yield 2
    return "Done"
 
def outer():
    final = yield from inner()
    print(f"Inner returned: {final}")
 
gen = outer()
print(next(gen))  # 1
print(next(gen))  # 2
try:
    next(gen)
except StopIteration as e:
    print(e.value)  # "Done"

In Python 3.7+, you rarely need this pattern because async/await handles delegation better. But understanding it matters for complex generator chains. You'll encounter yield from in older codebases and in lower-level coroutine implementations, and recognizing what it's doing, transparent bidirectional delegation, will save you hours of debugging.

Infinite Generators and itertools

Generators can be infinite. They just never raise StopIteration. This sounds alarming at first, an infinite loop without a break condition is usually a bug. But with generators, an infinite sequence is a perfectly valid thing to define because you control exactly how many values you consume from the outside.

def infinite_counter(start=0):
    while True:
        yield start
        start += 1
 
# This will run forever if you iterate all of it
for num in infinite_counter():
    print(num)  # 0, 1, 2, 3, ... forever

Obviously, you can't iterate an infinite generator to completion. But you can consume it in chunks using itertools:

from itertools import islice
 
counter = infinite_counter()
first_five = list(islice(counter, 5))
print(first_five)  # [0, 1, 2, 3, 4]
 
# Get items 10-15
next_batch = list(islice(counter, 5))
print(next_batch)  # [5, 6, 7, 8, 9]

Here's a practical example: cycling through a list indefinitely, which is incredibly useful for things like rotating through a pool of API keys, load-balancing across servers, or repeating a color scheme:

from itertools import cycle
 
colors = cycle(['red', 'green', 'blue'])
for _ in range(10):
    print(next(colors))
# red, green, blue, red, green, blue, red, green, blue, red

Chaining generators together lets you combine multiple data sources into a single stream without intermediate storage:

from itertools import chain
 
gen1 = (x for x in [1, 2, 3])
gen2 = (x for x in [4, 5, 6])
 
combined = chain(gen1, gen2)
for val in combined:
    print(val)  # 1, 2, 3, 4, 5, 6

This is memory-efficient because chain doesn't create a list, it just yields from each generator in sequence. The entire itertools module is built on this philosophy: compose small, lazy pieces into powerful pipelines.

Common Iterator Mistakes

The most dangerous thing about iterators is that they often fail silently. You don't get an exception, you just get empty results or subtly wrong output. Knowing the common pitfalls saves you from spending hours debugging code that "looks right" but produces nothing.

The single most common mistake is iterating a generator twice. Because generators are one-shot by definition, the second loop gets nothing. But the code looks identical to a list loop, so the bug can be invisible. The fix is either to convert to a list if you need multiple passes, or to restructure your code so the generator is only consumed once. If you're passing generators between functions, document which function "owns" the consumption, because whoever iterates first wins.

A related mistake is storing a generator in an instance variable and trying to use it across multiple method calls. The generator's state persists, which means the second method call continues where the first left off, not from the beginning. If your class's API promises fresh data on each call, you need to recreate the generator every time, not reuse the stored instance.

Another trap is capturing a loop variable in a generator expression. Consider this: if you write generators = [(x for x in range(i)) for i in range(5)], you might expect each generator to iterate over a different range. But because i is captured by reference, all five generators see the final value of i when they execute. You need to force early binding: generators = [(x for x in range(i)) for i in range(5)] actually works correctly here because the range(i) call is evaluated eagerly at list comprehension time. The more dangerous case is with lambda or yield inside nested loops where the outer loop variable changes.

Finally, never assume a generator is idempotent. Two separate calls to a generator function produce two independent generators, which is great. But one call to a generator function produces one generator that can only be consumed once. Passing that single generator object to a utility function that internally iterates it, and then also iterating it in your calling code, will give you partial results in both places without any error being raised.

Gotcha 1: Generators are one-shot. Once exhausted, they're done:

gen = (x for x in [1, 2, 3])
list(gen)  # [1, 2, 3]
list(gen)  # [] - generator is exhausted!

If you need to iterate multiple times, use a list or recreate the generator. This is a common source of bugs when you pass a generator to multiple functions and only the first one gets data.

Gotcha 2: You can't index generators:

gen = (x for x in range(10))
print(gen[5])  # TypeError

If you need indexing, convert to a list or use itertools.islice():

from itertools import islice
gen = (x for x in range(10))
value = next(islice(gen, 5, 6))  # Get item at index 5

Note that islice(gen, 5, 6) consumes items 0-4 to reach index 5, so this isn't "random access" like lists. It's still O(n) to jump to an index.

Gotcha 3: Generator expressions in function calls can be confusing:

# This works - parentheses count as the generator expression's parentheses
sum(x**2 for x in range(10))
 
# This also works
sum((x**2 for x in range(10)))
 
# This fails - the inner parens are for the function call
sum(x**2 for x in range(10), 1)  # SyntaxError
# Fix it:
sum((x**2 for x in range(10)), 1)

Gotcha 4: Generator functions don't begin executing until you call next():

def my_gen():
    print("Starting!")
    yield 1
 
gen = my_gen()  # Nothing is printed yet
print(next(gen))  # "Starting!" is printed here

This deferred execution is powerful but can surprise newcomers. Functions are lazy to the core.

Tip: Use itertools.tee() to split a generator into independent copies:

from itertools import tee
 
gen = (x for x in [1, 2, 3])
gen1, gen2 = tee(gen, 2)
 
list(gen1)  # [1, 2, 3]
list(gen2)  # [1, 2, 3]

However, tee() buffers consumed values from the first generator in memory, so use it sparingly. It's a tool for the specific case where you genuinely need to iterate the same sequence twice.

Tip: Combine enumerate() with generators when you need both index and value:

data = (x for x in ['a', 'b', 'c'])
for index, value in enumerate(data):
    print(f"{index}: {value}")  # 0: a, 1: b, 2: c

Tip: Use zip() with generators to iterate multiple sequences in parallel:

keys = (x for x in ['name', 'age', 'city'])
values = (x for x in ['Alice', 30, 'NYC'])
 
for key, value in zip(keys, values):
    print(f"{key}: {value}")

If one generator is shorter, zip() stops at the shortest. Use itertools.zip_longest() if you need to consume all items.

Performance Comparison: Real Numbers

Let's put this into perspective with actual measurements. Here's a comparison between lists and generators for a common task, reading and processing CSV data. The numbers here tell a story that pure theory can't fully convey, so run this yourself with your own dataset sizes to see how it scales.

import time
import csv
from io import StringIO
 
# Create sample CSV data
csv_data = '\n'.join([f"{i},{i*2},{i*3}" for i in range(100000)])
 
# Method 1: List comprehension (load everything)
def process_csv_list():
    reader = csv.reader(StringIO(csv_data).readlines())
    data = [row for row in reader]
    return sum(int(row[0]) for row in data)
 
# Method 2: Generator (lazy evaluation)
def process_csv_generator():
    reader = csv.reader(StringIO(csv_data).splitlines())
    return sum(int(row[0]) for row in reader)
 
# Timing
start = time.time()
result = process_csv_list()
list_time = time.time() - start
 
start = time.time()
result = process_csv_generator()
gen_time = time.time() - start
 
print(f"List: {list_time:.4f}s")
print(f"Generator: {gen_time:.4f}s")
print(f"Speedup: {list_time/gen_time:.2f}x")

On a typical system, the generator approach is comparable in speed for first-pass iteration, but uses significantly less memory. The real win is when you only need part of the data:

from itertools import islice
 
# Get only the first 10 rows (generator stops after 10)
first_10_rows = list(islice(process_csv_generator(), 10))

The generator only reads 10 lines. The list version would read all 100,000. That's the power difference.

Advanced Pattern: Generator State Management

Sometimes you need a generator to maintain complex state or interact with external resources (like database connections). Here's a pattern that combines the context manager protocol with generator-based streaming to ensure resources are properly cleaned up even if iteration is interrupted:

import sqlite3
 
class DatabaseIterator:
    def __init__(self, db_path):
        self.db_path = db_path
        self.conn = None
 
    def __enter__(self):
        self.conn = sqlite3.connect(self.db_path)
        return self
 
    def __exit__(self, *args):
        if self.conn:
            self.conn.close()
 
    def fetch_users(self, batch_size=1000):
        offset = 0
        while True:
            rows = self.conn.query(f"SELECT * FROM users LIMIT {batch_size} OFFSET {offset}")
            if not rows:
                break
            for row in rows:
                yield row
            offset += batch_size
 
# Usage
with DatabaseIterator('mydb.sqlite') as db:
    for user in db.fetch_users():
        print(user)  # Process one user at a time

This pattern ensures the database connection is properly opened and closed while yielding results. It's cleaner than managing resources inside a raw generator function.

When Generators Shine: A Decision Tree

Here's a practical guide for when to reach for generators:

Use generators if:

• Processing > 1GB of data
• Iterating through a file or API with pagination
• Building a processing pipeline (read -> transform -> filter -> output)
• You only need one pass through the data
• Memory is a constraint (embedded systems, containerized environments)

Use lists if:

• Data fits comfortably in memory (< 100MB as a rule of thumb)
• You need random access or len()
• You iterate multiple times
• You need to mutate the data
• Simplicity and readability matter more than efficiency

Real-World Pattern 1: Streaming File Reads

Here's a generator for reading large files in chunks. This pattern is essential for processing binary files like images, videos, or compressed archives where you can't use line-based iteration:

def read_file_chunked(filepath, chunk_size=8192):
    with open(filepath, 'rb') as f:
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            yield chunk
 
# Usage
for data_chunk in read_file_chunked('huge_file.bin'):
    print(len(data_chunk))  # Process one 8KB chunk at a time

Or for reading lines (which is so common it's built-in):

# This is already lazy!
with open('data.csv') as f:
    for line in f:
        print(line.strip())

Real-World Pattern 2: Data Pipelines

Build processing pipelines by chaining generators. Each function in the pipeline is a simple transformation that reads from its input and yields to its output, clean, composable, and trivially testable in isolation:

def read_numbers(filename):
    with open(filename) as f:
        for line in f:
            yield int(line.strip())
 
def filter_evens(numbers):
    for num in numbers:
        if num % 2 == 0:
            yield num
 
def double_them(numbers):
    for num in numbers:
        yield num * 2
 
# Chain them together
data = read_numbers('numbers.txt')
evens = filter_evens(data)
doubled = double_them(evens)
 
for result in doubled:
    print(result)

This processes one number at a time through the entire pipeline. No intermediate lists. No memory bloat. Beautiful.

Python's map() and filter() are also lazy by design:

numbers = [1, 2, 3, 4, 5]
evens = filter(lambda x: x % 2 == 0, numbers)
doubled = map(lambda x: x * 2, evens)
 
for result in doubled:
    print(result)  # 4, 8

These return iterators, not lists, so they're memory-efficient.

Real-World Pattern 3: Paging Through APIs

When fetching data from an API that paginates, generators let you fetch pages lazily. The caller doesn't need to know anything about pagination, they just iterate and get items. The complexity is hidden inside the generator:

import requests
 
def paginate_api(endpoint, page_size=100):
    page = 1
    while True:
        response = requests.get(endpoint, params={'page': page, 'limit': page_size})
        data = response.json()
 
        if not data:  # No more pages
            break
 
        for item in data:
            yield item
 
        page += 1
 
# Usage
for user in paginate_api('https://api.example.com/users'):
    print(user)

You don't fetch all pages upfront. You fetch one page at a time and yield its items. If you only need the first 50 users, you only fetch the first page. Efficiency.

Performance Characteristics

Generators don't make everything faster, they make things different. Understanding the trade-offs is crucial:

Computational Cost: Generators defer computation. The first call to next() might trigger significant work. Consider this:

def expensive_computation():
    for i in range(1000000):
        yield i ** 2  # Computation happens here, on demand
 
gen = expensive_computation()  # Instant, no work yet
first = next(gen)  # Now we compute one value

Versus:

data = [i ** 2 for i in range(1000000)]  # All computation upfront
first = data[0]  # Instant access

If you only need a few values, the generator is faster overall. If you need all values, the list might be faster because it can optimize the entire computation at once. Profile your specific use case.

I/O Efficiency: Generators excel when I/O is the bottleneck. Imagine reading from a slow network source:

def fetch_from_slow_api(endpoint):
    page = 1
    while True:
        response = requests.get(endpoint, params={'page': page})
        if not response.json():
            break
        for item in response.json():
            yield item  # Yield immediately after fetching
        page += 1

With this generator, you get results as soon as the first page arrives. You don't wait for all pages. That's a massive difference in perceived responsiveness.

Generators and Memory Profiling

If you're serious about optimization, you should profile memory usage. Python's built-in tracemalloc module gives you precise control over memory measurement, letting you compare the peak usage between the list and generator approaches rather than relying on intuition:

import tracemalloc
import sys
 
tracemalloc.start()
 
# List version
list_version = [x**2 for x in range(1000000)]
current, peak = tracemalloc.get_traced_memory()
print(f"List peak memory: {peak / 1024 / 1024:.2f} MB")
tracemalloc.stop()
 
tracemalloc.start()
 
# Generator version
gen_version = (x**2 for x in range(1000000))
# Consume it
sum(gen_version)
current, peak = tracemalloc.get_traced_memory()
print(f"Generator peak memory: {peak / 1024 / 1024:.2f} MB")
tracemalloc.stop()

For large ranges, the difference is dramatic. The list version uses tens of megabytes; the generator version uses almost nothing. Run this benchmark with your actual data sizes and access patterns before deciding which approach to use, the numbers will make the right choice obvious.

Generators and Async: A Preview

Before we finish, a quick note: generators are the foundation for async/await. When you use async def and await, you're using generator-like suspension and resumption under the hood. Understanding generators now will make async code much clearer in article 57.

Putting It All Together: A Production Log Pipeline

Let's tie everything together with a complete, production-ready example that incorporates everything we've covered. Imagine you're analyzing web server logs that span multiple files, each potentially gigabytes in size. This is exactly the kind of problem where a naive list-based approach would crash a production server:

import gzip
from pathlib import Path
from datetime import datetime
 
def read_log_files(log_directory, pattern='*.log.gz'):
    """Generator: yields individual log lines from all matching files."""
    for filepath in sorted(Path(log_directory).glob(pattern)):
        print(f"Reading {filepath}")
        with gzip.open(filepath, 'rt') as f:
            for line in f:
                yield line.rstrip()
 
def parse_log_lines(lines):
    """Generator: parses raw log lines into structured data."""
    for line in lines:
        try:
            # Assuming Apache Combined Log Format
            parts = line.split(' ')
            if len(parts) >= 9:
                yield {
                    'ip': parts[0],
                    'timestamp': parts[3].strip('['),
                    'method': parts[5].strip('"'),
                    'path': parts[6],
                    'status': int(parts[8]),
                }
        except (ValueError, IndexError):
            continue  # Skip malformed lines
 
def filter_errors(log_entries):
    """Generator: yields only 5xx error entries."""
    for entry in log_entries:
        if entry['status'] >= 500:
            yield entry
 
def aggregate_errors(log_entries):
    """Generator: groups errors by path and yields summaries."""
    error_counts = {}
    for entry in log_entries:
        path = entry['path']
        error_counts[path] = error_counts.get(path, 0) + 1
 
    for path, count in sorted(error_counts.items(), key=lambda x: x[1], reverse=True):
        yield {'path': path, 'error_count': count}
 
# The pipeline: read -> parse -> filter -> aggregate
raw_logs = read_log_files('/var/log/apache')
parsed_logs = parse_log_lines(raw_logs)
error_logs = filter_errors(parsed_logs)
summaries = aggregate_errors(error_logs)
 
# Consume the results
for summary in summaries:
    print(f"{summary['path']}: {summary['error_count']} errors")

Notice the beautiful composability here. Each function is simple, focused, and works with one item at a time. The entire pipeline processes data without ever loading more than a few lines into memory, no matter how many or how large the log files are. Each stage can be tested in isolation, swapped out independently, or extended with new transformations without touching the other stages.

Compare this to a list-based approach:

# List-based (BAD for large files)
all_lines = []
for filepath in Path(log_directory).glob('*.log.gz'):
    with gzip.open(filepath, 'rt') as f:
        all_lines.extend(f.readlines())  # Loads everything!
 
parsed_logs = [parse_line(line) for line in all_lines]
error_logs = [log for log in parsed_logs if log['status'] >= 500]
# ... and so on

If your log directory contains 100GB of compressed logs, the list approach will crash or hang. The generator approach processes it smoothly, yielding results as it goes. The list approach also creates multiple intermediate copies of the data, one for raw lines, one for parsed entries, one for errors, multiplying the memory footprint at each stage.

Conclusion

Iterators and generators represent one of Python's most elegant design decisions: making laziness the default, not the exception. The iterator protocol gives you a standard interface for any sequence of values, whether it comes from a list in RAM, a file on disk, a database query, or a mathematical sequence. Generators make implementing that protocol trivially easy, collapsing dozens of lines of class boilerplate into a few lines of logic with yield.

The memory benefits are real and significant. Constant memory usage means your code scales horizontally to any data size without architectural changes. A script that processes 100MB of logs with generators will process 100GB with exactly the same code, you just let it run longer. That's the kind of scalability that would otherwise require distributed computing frameworks.

Beyond efficiency, generators improve code clarity. A generator function reads like a description of a sequence, not like an implementation of state management. When you chain generators into a pipeline, each stage has a single responsibility and a clean interface. The data flow is obvious from the function names alone. This is idiomatic Python: expressive, composable, and unsurprising.

As you move into the more advanced topics in this series, decorators, async/await, concurrency, you'll encounter the iterator protocol everywhere. Async generators, async context managers, and the entire asyncio event loop are built on the same suspension and resumption model that yield introduced. The intuition you've built here transfers directly. Master generators now, and the advanced concurrency topics will feel like natural extensions rather than foreign concepts.