For Loops and Iterators

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

This guide assumes that you are already familiar with the for loop and have a general idea of how a for loop works.

So far, you have seen for loops over many data structures:

odds = range(1, 10, 2)
for num in odds:  # odds is a range object.
print(num)

vowels = 'aeiou'
for vowel in vowels:  # vowels is a string object.
print(vowel)

glookup = { 'hw' : 2, 'proj1' : 17, 'proj2' : 12, 'test1' : 50 }
for key in glookup:  # glookup is a dictionary object.
print('{0} has a maximum score of {1}'.format(key, glookup[key]))

A key principle in program design is that you should use an interface in order to simplify code. (Make sure you know what an interface is before proceeding.) In particular, in the case of for loops, there must be an interface that all of these various different objects implement that the for loop takes advantage of. You might think it is the sequence interface, but looking at that last example, we can see that it is possible to have a for loop over dictionaries, and dictionaries are not sequences. The most commonly used interface in a for loop is the Iterator interface.

Iterators

Conceptually, an iterator is an object that allows you to iterate through some data. In particular, you can keep querying the iterator in order to get more values (at least until it runs out of values). Why would we want this? Don't we already have lists, which can do the same thing? The key point is that iterators are more efficient - they only store enough data so that they can compute the next number. On the other hand, lists have to store all of their values at the same time, which uses up the computer's memory. For example, if we wanted to iterate through the numbers from 1 to a billion, with a list, we would have to use 1 billion slots in the computer to store all of those numbers. With an iterator on the other hand, we only need to store one number - the number we are currently considering.

In order to implement this idea, an iterator just needs to have a single method, that allows you to get the next item from the iterator, or raises a StopIteration error to signal that there are no more elements. This is the __next__ magic method. However, it turns out that you also need an __iter__ method - more on that soon.

Let's look at what happens if we create a class that only has a __next__ magic method and not the __iter__ magic method. We will try to define the RangeIterator class, which will eventually be an Iterator that duplicates (some of) the functionality of the built-in range object:

class RangeIterator:

def __init__(self, start, end):
self.val = start
self.end = end

def __next__(self):
if self.val >= self.end:
raise StopIteration
value_to_return = self.val
self.val += 1
return value_to_return

>>> teens = RangeIterator(13, 20)
>>> teens.__next__()
13
>>> next(teens)  # Equivalent to teens.__next__()
14
>>> for i in range(5):  # Repeat 5 times
...     print(next(teens))
...
15
16
17
18
19
>>> next(teens)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __next__
StopIteration

Okay, so far this seems to be working.

But if we try a for loop, things break:

>>> for i in RangeIterator(0, 10):
...     print(i)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'RangeIterator' object is not iterable

Okay, that was confusing. To understand that, let's take a detour and talk about for loops and what they actually do:

How a For Loop uses Iterators

In general a for loop looks something like this:

for elem in iterable_thing:
do_stuff_with_elem

Internally, Python does the following:

try:
iterator = iterable_thing.__iter__()  # Get an iterator!
except AttributeError:
raise TypeError('not iterable')
try:
while True:
elem = iterator.__next__()
do_stuff_with_elem
except StopIteration:
pass  # Do nothing

Notice the line which gets an iterator. This is a design decision that Python made - Python wants to allow you to use a for loop on iterators as well as things that you can iterate over. So in the loop above, the iterable_thing could be an iterator or something that could give you an iterator (through the __iter__ method).

So, now we can understand why our RangeIterator example didn't work out. The example we had was

>>> for i in RangeIterator(0, 10):
...     print(i)
...

If we look at what a for loop does, we notice that it first calls the __iter__ method, which means that it tries to do RangeIterator(0, 10).__iter__() However, RangeIterator does not have an __iter__ method! That's why we get the not iterable error message.

How can we fix this? Well, an easy fix would be to give RangeIterators an __iter__ method. What should that __iter__ method do? It is supposed to return an iterator, so that the for loop can call __next__ on the iterator to get the various values. However, the RangeIterator is already an iterator, so we can just return the RangeIterator itself!

class RangeIterator:

def __init__(self, start, end):
self.val = start
self.end = end

def __iter__(self):
return self  # self is a RangeIterator object, which already has a __next__ method

def __next__(self):
if self.val >= self.end:
raise StopIteration
value_to_return = self.val
self.val += 1
return value_to_return

>>> for i in RangeIterator(2, 6):
...     print(i)
...
2
3
4
5

In summary - because of how for loops work, an iterator must have both an __iter__ and a __next__ method.

Iterable

Conceptually, an iterable object is something that we can put in a for loop. If we look at what a for does with the iterable_thing, we notice that all it does is call the __iter__ method. So, an object implements the Iterable interface if it has a __iter__ method. (It may or may not have a __next__ method.)

• If something is an iterator, must it also be an iterable?
• If something is an iterable, must it also be an iterator?
• Why do iterators have to have an __iter__ method? Shouldn't they only need a __next__ method?
• Why do we bother using iterators? Couldn't we just use lists all the time? They are easier to use than iterators, after all...
• You have the following interaction with the Python interpreter:
>>> for i in Foo():
...     print(i)
...
TypeError: 'Foo' object is not iterable
>>> for i in Bar():
...     print(i)
...
>>> for i in Baz():
...     print(i)
...
hello
world
>>> next(Garply())
0
>>> next(Garply())
0
>>> next(iter(Garply()))
0
• For each of Foo, Bar, Baz, and Garply, answer the following questions:
• Is it guaranteed to have an __iter__ method? Is it guaranteed to not have an __iter__ method?
• Same thing for the __next__ method.
• Is it guaranteed to be Iterable?
• Is it guaranteed to be an Iterator?
• [Midterm Level - Average] What Would Python Output? (Assume that the RangeIterator class has already been defined, as shown in the previous section.)
>>> rng = RangeIterator(2, 5)
>>> for i in rng:
...     for j in rng:
...         print(i, j)
...
• [Midterm Level - Tricky] What Would Python Output?
class RangeIterator:
def __init__(self, start, end):
self.start = start
self.val = start
self.end = end
def __iter__(self):
self.val = self.start  # Reset to the beginning
return self
def __next__(self):
if self.val >= self.end:
raise StopIteration
value_to_return = self.val
self.val += 1
return value_to_return

class Range:
def __init__(self, start, end):
self.start = start
self.end = end
def __iter__(self):
return RangeIterator(self.start, self.end)

>>> rng = RangeIterator(2, 4)
>>> for i in rng:
...     for j in rng:
...         print(i, j)
...
>>> rng = Range(2, 4)
>>> for i in rng:
...     for j in rng:
...         print(i, j)
...