Understanding the Iteration Protocol

Before diving into generators, you need to understand how Python's iteration works under the hood. When you write a for loop, Python uses a specific protocol to get values one at a time. This protocol is the foundation for all iteration in Python, and understanding it deeply will help you write more efficient and debuggable code.

When you write for x in something: and don't know what Python is doing under the hood, you can't predict why the same iterator produces nothing on a second pass, or why a custom class that "should" loop raises TypeError: not iterable. Teams that skip the protocol end up with generators that mysteriously skip values and helpers that silently process empty sequences — bugs that don't crash, they just quietly drop data. By the end of this lesson you will be able to trace exactly which __iter__ and __next__ calls Python issues for any iteration construct, and you'll know when an iterable can be reused versus when it has been consumed.

• iterable — any object Python can begin iteration over because it implements __iter__(). Lists, dicts, files, and your own classes qualify; this lesson treats __iter__ presence as the defining test.
• iterator — an object that yields values one at a time via __next__(). Every for loop runs over an iterator (not the iterable itself), and iterators get exhausted after one pass — the central pitfall this lesson protects you against.
• iteration protocol — the two-method contract: __iter__() returns an iterator, __next__() returns the next value or raises StopIteration. Comprehensions, sum(), max(), and list() all depend on this exact handshake.
• StopIteration — the exception that signals "no more values." Raising it (not returning a sentinel like None or -1) is the only correct way to end iteration in your own iterators.

What is an Iterator?

An iterator is an object that implements two special methods that together form the iteration protocol:

• iter() — returns the iterator object itself. This method is called when iteration begins.
• next() — returns the next value in the sequence, or raises StopIteration when there are no more values.

This protocol is the contract that every iterable in Python follows. Once you understand it, you also understand how for loops work, how comprehensions work, and how built-in functions like sum(), max(), and list() consume iterables. Generators are a special type of iterator, so this underlying mechanism is what you reach for when a generator behaves unexpectedly or you need to write an efficient custom collection.

Lists, tuples, strings, dictionaries, sets, and open files all implement iter to return an iterator, and that iterator implements next to hand back values one at a time.

The iter() and next() Built-in Functions

Python exposes the protocol through two built-ins. iter(obj) calls obj.__iter__() to get an iterator, and next(iterator) calls iterator.__next__() to fetch the next value. These functions are how you peel back the abstraction: they let you step an iterator forward one value at a time, watch exactly when StopIteration fires, and — with next()'s two-argument form — substitute a default value instead of handling the exception.

Iterables vs Iterators

An iterable is any object that can return an iterator (has iter). An iterator is an object that produces values one at a time (has next). All iterators are iterables — they return themselves from iter — but not all iterables are iterators. The practical consequence is the single most important takeaway from this lesson: a list can be looped over many times because each new loop calls iter to get a fresh iterator, but an iterator object itself is exhausted after one pass and silently yields nothing on every subsequent attempt.

Now that you understand the two-method contract — __iter__() returns an iterator and __next__() either yields the next value or raises StopIteration — the examples below make that handshake concrete.

The first example implements the protocol from scratch. CountUp is a minimal iterator that counts from 1 up to max_value inclusive, holding its own position in self.current and raising StopIteration once the cap is reached.

Code snippetpython
1class CountUp:
2    def __init__(self, max_value):
3        self.max_value = max_value
4        self.current = 0
5
6    def __iter__(self):
7        # Simple iterators that maintain their own state return self.
8        return self
9
10    def __next__(self):
11        if self.current >= self.max_value:
12            raise StopIteration
13        self.current += 1
14        return self.current
15
16counter = CountUp(3)
17for num in counter:
18    print(num)  # 1  2  3

__iter__ returns self because the object both starts iteration and tracks its own cursor — the standard pattern for a single-pass iterator. __next__ increments current and returns it, or raises StopIteration when the cap is hit. The for loop calls __iter__ once to obtain the iterator, then calls __next__ repeatedly until the exception fires, at which point Python catches it silently and exits the loop.

The second snippet drives the same protocol using the iter() and next() built-ins, and exposes the iterable-versus-iterator distinction directly: a list has __iter__ but not __next__, so it is iterable but is not itself an iterator. Calling iter(my_list) produces a separate iterator object each time, which is why a list can be looped more than once while a raw iterator is exhausted after one pass.

Code snippetpython
1my_list = [1, 2, 3]
2
3# A list has __iter__ but not __next__ — iterable, not an iterator.
4print(hasattr(my_list, '__next__'))  # False
5
6# iter() produces a fresh iterator from the iterable.
7iterator = iter(my_list)
8print(next(iterator))  # 1
9print(next(iterator))  # 2
10print(next(iterator))  # 3
11# next(iterator) here would raise StopIteration
12
13# Two-argument next() substitutes a default instead of raising.
14it2 = iter([10, 20])
15print(next(it2, 'done'))  # 10
16print(next(it2, 'done'))  # 20
17print(next(it2, 'done'))  # 'done' — no StopIteration raised
18
19# A list gives a fresh iterator each time; a raw iterator is exhausted.
20iterator = iter(my_list)
21list(iterator)   # [1, 2, 3] — consumes the iterator
22list(iterator)   # []        — already exhausted

The silent empty list on the final call is the visible signature of iterator exhaustion. The two-argument form next(iterator, default) converts a would-be StopIteration into a plain return value — the idiomatic way to express a "peek or fall back" pattern without a try/except block.

You'll know it works when you can predict, before running the code, whether a second list(iterator) call returns the original values or an empty list.

Do's

1. ✓Do raise StopIteration to end a custom iterator — it is the only signal that for loops, comprehensions, and built-ins like sum() and list() recognise as "no more values".
2. ✓Do reach for iter() and next() when debugging — stepping an iterator one value at a time exposes exactly where it exhausts, skips, or returns something unexpected.
3. ✓Do remember that iterators are single-pass — if you need to re-iterate, call iter() on the original iterable again, or materialize the values into a list you can loop over repeatedly.

Don'ts

1. ✗Don't return a sentinel like None or -1 to signal "no more values" — sentinels silently break for loops, comprehensions, and sum() because none of them check for sentinels; they only catch StopIteration.
2. ✗Don't assume a second pass over an iterator yields data — once exhausted, list(it) returns [] and for x in it runs zero times; this is the classic "my function silently processed nothing" bug.
3. ✗Don't conflate iterable and iterator in your API surface — accept iterables when callers may want to re-read, and document clearly when you consume the input so they don't pass in a generator they expected to reuse.