I. Iterator:

  • Any object iterated through a for loop is an Iterable
  • Objects evaluated by the next() built-in function are of type Iterator, representing a sequence of lazy computations
  • An Iterable can be converted to an Iterator using the built-in function iter()
  • The internal implementation mechanism of the for loop is to convert an iterable to an iterator using iter(), and to value it from the iterator using next()

1. Understand containers

A container is a data structure that organizes multiple elements together. The elements in the container can be obtained iteratively one by one. The in and not in keywords can be used to determine whether the elements are contained in the container. Usually this type of data structure stores all the elements in memory (there are some special ones). Common Python container objects:

1. List, deque... 2. set, frozensets, ... 3. dict, defaultdict, OrderedDict, Counter, ... 4. tuple, nametuple, ... 5. strCopy the code
Iterables give containers the ability to extract elements. 2. Not all containers are iterableCopy the code

2. Iterable is a colloquial term for iterable.

Given a list or tuple, we can iterate through the list or tuple through the for loop, which we call Iteration.

Iterable alignment is not limited to containers: files, sockets, etc. (open state). Any iterable that returns an iterable can be called an iterable. So let’s look at an example:

x = [1, 2, 3]
y = iter(x)

print(next(y))
print(next(y))

print(type(x), type(y))

>>>
1
2
<class 'list'> <class 'list_iterator'>
Copy the code

Analysis:

Here x is an iterable, which, like a container, is a colloquial term, not a specific data type. List,set, and dict are all iterable objects, while Y is an independent iterator. Inside the iterator, there is a state, which is used to record the position of the current iteration to facilitate the retrieval of the correct elements in the next iteration. Iterators have one specific type: list_iterator,set_iterator,… . Iterables implement __iter__() and __next__() methods (next() in python2 and __next__() in python3.x), which correspond to the built-in functions iter() and next(). The __iter__ method returns the iterable itself, making it both an iterable and an iterator.

3. An iterator is an object

An iterator is an object that can be used to iterate over some or all of the elements in a standard template library container. Each iterator object represents a specified address in the container. Iterators modify the interface of regular Pointers. An iterator is a conceptual abstraction: anything that behaves like an iterator can be called an iterator. While iterators have many different capabilities, they can combine abstract containers with general algorithms.

Compare these two methods:

Object iteration must possess: __iter__ () iteration instruments for: __iter__ () __next__ ()Copy the code

1. For iterators, a __next__() is sufficient. When using the for loop, the program automatically calls the iterator object to be processed and then uses the next() function until a StopIteration exception is detected. 2. The next () built-in function is called the method of object __next__ (), iter () built-in function object is called __iter__ () method

Who can understand the concept? Let’s use the code to verify it:

Call the next() method directly

>>> L = [1, 2, 3]
>>> next(L) 
Traceback (most recent call last):
  File "<input>", line 2, in <module>
TypeError: 'list' object is not an iterator

>>> L = [1, 2, 3]
>>> print(type(L))
<class 'list'>
Copy the code

L lists can be evaluated using a for loop, not next(), so L is an iterable, not an iterator

Call iter() first, then next() and try again

>>> L = [1, 2, 3] >>> I = iter(L) >>> next(I) 1 >>> next(I) 2 >>> next(I) 3 >>> next(I) Traceback (most recent call last): File "<input>", line 1, in <module> StopIteration # exception >>> print(type(I)) <class 'list_iterator'>Copy the code

When the list L is wrapped with iter(), it becomes an iterator and can be called next()

In addition to the above methods, there are corresponding judgment functions:

Iterable judgment

>>> from collections import Iterable
>>> isinstance([], Iterable)
True
>>> isinstance({}, Iterable)
True
>>> isinstance((), Iterable)
True
>>> isinstance('abc', Iterable)
True
>>> isinstance(1000, Iterable)
False
Copy the code

Iterator judgment

>>> from collections import Iterator
>>> isinstance([x for x in range(9)], Iterator)
False
>>> isinstance((x for x in range(9)), Iterator)
True
>>> isinstance(1000, Iterator)
False
>>> isinstance({}, Iterator)
False
>>> isinstance([], Iterator)
False
>>> isinstance((), Iterator)
False
>>> isinstance('abc', Iterator)
False
Copy the code

When iter() is used to transform the iterable, it becomes an iterator, which can meet the judgment conditions:

>>> isinstance(iter('abc'), Iterator)
True
>>> isinstance(iter([]), Iterator)
True
Copy the code

4. Inside the for loop is iter() and next().

Iterable: iterator: iterator: iterator: iterator: iterator: iterator: iterator: iterator: iterator: iterator

It actually looks like this in the code

L = [1, 2, 3] for I in L: pass I = iter(L) # fetch iterator while True: # try: x = next(I) # fetch next value except StopIteration: Break # catches the exception exit loopCopy the code

5. A Python Iterator represents a stream of data

At this point, there is also confusion: why aren’t list, dict, set, STR, etc., iterators? This is because Python’s Iterator represents a stream of data. The Iterator can be called by the next() built-in function and returns the next iteration until there is no more data. The data stream can be thought of as an ordered sequence, but the length of the sequence cannot be known in advance. The next() built-in function continuously evaluates the next data on demand, so the Iterator is lazy and only evaluates when it needs to return the next data. Iterator can be regarded as an infinitely large data stream, eg: all the natural numbers, and list, set, STR, etc. cannot store all the natural numbers.

The extension Iterator inherits from Iterable. Using the built-in help, you can easily see that Iterator contains __iter__() and next() methods, whereas Iteratble contains only __iter__().

>>> from collections  import  Iterator, Iterable
>>> help(Iterator)
Help on class Iterator in module collections.abc:

class Iterator(Iterable)
 |  Method resolution order:
 |      Iterator
 |      Iterable
 |      builtins.object
 |  
 |  Methods defined here:
 |  
 |  __iter__(self)
 |  
 |  __next__(self)
 |      Return the next item from the iterator. When exhausted, raise StopIteration


>>> help(Iterable)
Help on class Iterable in module collections.abc:

class Iterable(builtins.object)
 |  Methods defined here:
 |  
 |  __iter__(self)
Copy the code

Ii. Generator

A generator is a function that returns an iterator. Its greatest use is to return an input object as an iterator. The concept of iteration is used in Python because the traditional memory loading method consumes a lot of memory when iterating over a large object, and is less expensive and faster than reading an element when needed. There are two things to be clear about: any generator is an iterator (otherwise, not true). Any generator is a factory that can delay the creation of values (controllability). In short, a generator object is a special kind of iterator that satisfies the iterator protocol and calls the next() built-in function. For the generator for loop, calling the iter() method returns the generator object, and then iterating over next(), while iter() and next() are both implemented inside yield. Generator creation: two common ways (list generation, function keyword)

1. How do I create a generator

1. The first method: change [] into () in the list generator.

With the list generator, you can create a list directly. However, due to memory limitations, list capacity is definitely limited. Moreover, creating a list of millions of elements not only takes up a lot of memory, for example, we only need to access the first few elements, most of the space taken up by the following elements is wasted.

Therefore, there is no need to create a complete list (saving a lot of memory). In Python, we can use a generator: looping, calculating mechanism – >generator

>>> L = [x*x for x in range(10)]
>>> L
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> g = (x*x for x in range(10))
>>> g
<generator object <genexpr> at 0x0000028F8B774200>
Copy the code

How do we print elements?

We just mentioned that generators are special iterators that can be used to get the next return value of the generator using the next() built-in function:

>>> g = (x*x for x in range(10))
>>> g
<generator object <genexpr> at 0x0000028F8B774200>
>>> next(g)
0
>>> next(g)
1
>>> next(g)
4
>>> next(g)
9
>>> next(g)
16
>>> next(g)
25
>>> next(g)
36
>>> next(g)
49
>>> next(g)
64
>>> next(g)
81
>>> next(g)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
StopIteration
Copy the code

Generator is a stored algorithm that computs the next element each time the next() built-in function is called. That perverted approach will not work, and since the Generator is also iterable, we can use a for loop.

>>> g = (x*x for x in range(10)) >>> for i in g: ... print(i) ... 0 1 49 16 25 36 49 64 81Copy the code

The for loop is used here without exception, just like the iterator, because of the internal mechanics of the for loop.

4. Second method: use the yield keyword in the function

In simple terms, during the execution of a function, the yield statement returns the desired value to the place where the generator was called, exits the function, and the next time the generator function is called, the execution will pick up where it left off, and all variables in the generator will be saved for the next time.

The famous Fibonacci sequence can’t be written using a list generator, but it’s easy to write using a function:

>>> def fib(max):
    n, a, b = 0, 0, 1
    while n < max:
        print(b)
        a, b = b, a + b
        n = n + 1
    return 'done'
>>> fib(6)
1
1
2
3
5
8
'done'
Copy the code

The Fibonacci sequence is very similar to a generator, that is, the above function is infinitely close to the generator by simply changing print(b) to yield B:

>>> def fib(max):
    n, a, b = 0, 0, 1
    while n < max:
        yield b
        a, b = b, a + b
        n = n + 1
    return 'done'

>>> f = fib(6)
>>> f
<generator object fib at 0x0000022C041D4360>
Copy the code

State: Function and Generator execute flow differently. Functions are executed sequentially, terminating when they encounter a return statement or the last line of function statements, while the generator executes next() each time it encounters a yield statement and continues execution from the yield statement returned the previous time.

Here’s a simple generator example:

>>> def odd(): print('step 1') yield 1 print('step 2') yield(3) print('step 3') yield(5) >>> o = odd() >>> next(o) step 1 1 >>> next(o)  step 2 3 >>> next(o) step 3 5 >>> next(o) Traceback (most recent call last): File "<input>", line 1, in <module> StopIterationCopy the code

Third, summary

  1. Iterators are a more abstract concept. Any object whose class has a next or iter method that returns itself can be easily iterated through for container objects such as strings, lists, dict, tuples, and so on. In the background, the for statement calls iter() on the container object. Iter () is python’s built-in function. Iter () returns an iterator object that defines the next() method, which accesses elements in the container one by one. Next () is also a python built-in function. When there are no subsequent elements, next() raises a StopIteration exception.

  2. Generators are simple and powerful tools for creating iterators. Generators automatically implement the “iterator protocol” (that is, the __iter__ and next methods). Generator functions use yield statements when they need to return data. Each time next() is called, the generator returns the position from which it departed (it remembers the last execution of the statement and all the data values). The generator can change the current iteration value during an iteration (send), whereas modifying the current iteration value of a normal iterator often causes an exception. Using generator expressions instead of list parsing can also save memory. In addition to the automatic methods for creating and saving program state, a StopIteration exception is automatically raised when the generator terminates.