Iterators and Iterables in Python 3

Lately I've been examining the difference between iterators and iterables in Python more closely. There's a subtle but distinct difference between the two that's easy to overlook (or simply ignore); however, most every time you've typed a for-loop, such as for i in [1, 2, 3, 4, 5]: print(i), you've used both an iterable and an iterator.

According to the Python 3.6 documentation, iterator types define special methods __iter__ and __next__. Together these form something in Python called the iterator protocol. Moreover, __iter__ typically just returns self, so we only have to worry about writing a definition for __next__. While collections.abc already contains an abstract Iterator base class (see e.g. collections.abc.Iterator.__abstractmethods__), instead of importing it I'm going to write it as follows.
from abc import abstractmethod
from typing import Any


class Iterator:
    # c.f. PEP 234, PEP 322

    @property
    @abstractmethod
    def _state(self) -> None:
        raise NotImplementedError

    @_state.setter
    @abstractmethod
    def _state(self, value: Any) -> None:
        raise NotImplementedError

    def __iter__(self) -> 'Iterator':
        return self

    @abstractmethod
    def __next__(self) -> Any:
        raise NotImplementedError
Iterator types are lazy[1] and thus only cough up a value when we ask them to (with the keyword next), so I've added an abstract _state property, to be defined more thoroughly in a subclass (this could just be as simple as a field variable though).

Now, a simple iterator example might look as follows.
class SimpleIterator(Iterator):
    def __init__(self, start: int =0, stop: int =10) -> None:
        self._state = start
        self.stop = stop

    @property
    def _state(self) -> int:
        return self.__state

    @_state.setter
    def _state(self, value: int) -> None:
        self.__state = value

    def __next__(self) -> int:
        if self._state >= self.stop:
            raise StopIteration
        else:
            val = self._state
            self._state += 1
            return val

if __name__ == '__main__':
    itr = SimpleIterator()
    for value in itr:
        print(value)

    # prints 0 ... 9
Note that, when this iterator type raises StopIteration, it continues to do so; i.e., calling next(itr) at any point in the future will raise StopIteration. We can convince ourselves of this with a simple brute-force unittest.
from unittest import main, TestCase


class TestIterator(TestCase):

    def setUp(self) -> None:
        self.itr = SimpleIterator()

    def test_stop_iteration(self) -> None:
        # Run iterator to the end
        for item in self.itr: pass
 
        # Check to see if it raises StopIteration a hundred times or so
        for _ in range(100):
            with self.assertRaises(StopIteration):
                next(self.itr)


main()


# ----------------------------------------------------------------------
# Ran 1 test in 0.002s
# 
# OK
Also, it's not necessary for an iterator to terminate, so the following is fine too.
class SimpleInfiniteIterator(Iterator):
    def __init__(self) -> None:
        self._state = 0

    @property
    def _state(self) -> int:
        return self.__state

    @_state.setter
    def _state(self, value: int) -> None:
        self.__state = value

    def __next__(self) -> int:
        """ Just keep adding one """
        val = self._state
        self._state += 1
        return val
Because a non-terminating iterator never raises a StopIteration, the for-loop continues indefinitely (and hence we don't have to worry about testing that it continues to raise such an exception as it never raises it in the first place).

So that's an iterator. Now what's an iterable? And how is it related to iterators?

From what I've found online, I think the most distinct difference is that iterable types can't be exhausted. In other words, every time a for-loop calls iter(iterable), it gets a fresh new iterator. That's why the following code sample doesn't raise any exceptions.
>>> X = list(range(10))
>>> last = X[-1]
>>> # Loop over X multiple times and print its contents
>>> for _ in range(3):
...     for i in X:
...         print(i, end=', ' if i != last else '\n')
... 
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
In this case, we're looping over a list referenced by X three times. Now, let's try overriding list's __iter__ method so that it cycles over items in the list.
class CyclicIterator(SimpleInfiniteIterator):
    """ Another cyclic iterator that shadows `itertools.cycle`
    (inherits from SimpleInfiniteIterator to avoid some boilerplate
    code)
    """

    def __init__(self, some_list: 'CyclicList') -> None:
        SimpleInfiniteIterator.__init__(self)
        self.some_list = some_list

    def __next__(self) -> int:
        val = self._state
        self._state += 1
        self._state %= len(self.some_list)
        return self.some_list[self._state]


class CyclicList(list):

    def __iter__(self) -> Iterator:
        return CyclicIterator(self)


# X = CyclicList([1, 2, 3, 4, 5])
# for i in X: print(i)
#
# prints 1 ... 5 repeatedly
This seems a little strange, however, because I've passed a reference to the iterator (also, my use of __getitem__ on list to get values).

Sources