Are Python coroutines really that hard? Collection!

Starting today, we’re going to get into the hard part of Python, which is coroutines.

In order to write to understand the knowledge point of coroutine, I looked up a lot of relevant information on the Internet. We find it difficult to have a systematic and comprehensive article, which leads to our learning, often half knowing and half understanding, or a face confused after learning.

The first course to learn coroutines is to know generators. With the foundation of generators, we can better understand coroutines.

If you’re new to iterators, you’re probably familiar with generators. That’s okay. By the end of this series, you’ll be able to go from being a geek to being a Python expert.

Again, all code in this series is written under Python3, and IT is recommended that you jump into Python3 as soon as possible.

Iterable, iterator, generator

When I first learned Python, I was really confused about all three. They’re even thought to be equivalent.

Actually, they are different.

Iterable objects, easy to understand, familiar to us: strings, lists, dict, tuple, deques, etc

To verify this, use the collections. ABC module (Python2 does not have it) and use isinstance() to classify an object as an Iterable, an Iterator, or a Generator.

import collections
from collections.abc import Iterable, Iterator, Generator

# string
astr = "XiaoMing"
print("String: {}".format(astr))
print(isinstance(astr, Iterable))
print(isinstance(astr, Iterator))
print(isinstance(astr, Generator))

# list
alist = [21.23.32.19]
print("List: {}".format(alist))
print(isinstance(alist, Iterable))
print(isinstance(alist, Iterator))
print(isinstance(alist, Generator))

# dictionary
adict = {"name": "Xiao Ming"."gender": "Male"."age": 18}
print("Dictionary: {}.format(adict))
print(isinstance(adict, Iterable))
print(isinstance(adict, Iterator))
print(isinstance(adict, Generator))

# deque
adeque=collections.deque('abcdefg')
print("A deque: {}".format(adeque))
print(isinstance(adeque, Iterable))
print(isinstance(adeque, Iterator))
print(isinstance(adeque, Generator))
Copy the code

The output

The value is XiaoMingTrue
False
FalseList:21.23.32.19]
True
False
FalseDictionary: {'name': 'Ming'.'gender': 'male'.'age': 18}
True
False
FalseA deque: deque (['a'.'b'.'c'.'d'.'e'.'f'.'g'])
True
False
False
Copy the code

As a result, none of these iterables are iterators or generators. One thing they have in common is that they can all use for loops. We know that, so we’re not going to test it.

Extended knowledge: Iterables, which internally implement the __iter__ magic method. You can tell if a variable is iterable by checking for __iter__ in the dir() method.

Next, iterators. In contrast to iterables, iterators are really just one more function. It is __next__(), and we can no longer use the for loop to retrieve elements intermittently. This can be done directly using the next() method.

Iterators are implemented on an iterable basis. To create an iterator, we first have to have an iterable. Now let’s see how to create an iterable and create an iterator based on an iterable.

from collections.abc import Iterable, Iterator, Generator

class MyList(object):  Define an iterable class

    def __init__(self, num):
        self.end = num  # on the border

    Return an instance of an iterator class that implements __iter__ and __next__
    def __iter__(self):
        return MyListIterator(self.end)


class MyListIterator(object):  Define the iterator class

    def __init__(self, end):
        self.data = end  # on the border
        self.start = 0

    Return an instance of the iterator class of this object; Return self because it is an iterator
    def __iter__(self):
        return self

    The iterator class must implement, or next() in the case of Python2
    def __next__(self):
        while self.start < self.data:
            self.start += 1
            return self.start - 1
        raise StopIteration


if __name__ == '__main__':
    my_list = MyList(5)  # get an iterable
    print(isinstance(my_list, Iterable))  # True
    print(isinstance(my_list, Iterator))  # False
    Iteration #
    for i in my_list:
        print(i)

    my_iterator = iter(my_list)  Get an iterator
    print(isinstance(my_iterator, Iterable))  # True
    print(isinstance(my_iterator, Iterator))  # True

    Iteration #
    print(next(my_iterator))
    print(next(my_iterator))
    print(next(my_iterator))
    print(next(my_iterator))
    print(next(my_iterator))
Copy the code

The output

0
1
2
3
4

True
False

True
True

0
1
2
3
4
Copy the code

If there’s too much code up here, you can also look over here, and you’ll understand it better.

from collections.abc import Iterator

aStr = 'abcd'  Create a string, which is an iterable
aIterator = iter(aStr)  The iterable is converted to an iterator by iter()
print(isinstance(aIterator, Iterator))  # True
next(aIterator)  # a
next(aIterator)  # b
next(aIterator)  # c
next(aIterator)  # d
Copy the code

Extended knowledge: iterator, which internally implements the __next__ magic method. (python3.x) can be used to determine if a variable is an iterator by looking for __next__ in the dir() method.

Next, the focus is on generators.

The concept of generators first appeared in Python 2.2, and they were introduced to implement a structure that didn’t waste space when evaluating the next value.

An iterator is an iterator that adds a next() method to the iterable. Generators, on the other hand, implement yield on top of iterators (you can use for loops, you can use next()).

So what’s yield? It’s the equivalent of a return in our function. Each time next() or for is traversed, yield returns the new value and blocks waiting for the next call. It is this mechanism that makes using generators so useful in Python programming. Realize saving memory, realize asynchronous programming.

There are two main ways to create a generator

Use list generators

# use a list generator, not [] but ()
L = (x * x for x in range(10))
print(isinstance(L, Generator))  # True
Copy the code

A function that implements yield

# implements the yield function
def mygen(n):
    now = 0
    while now < n:
        yield now
        now += 1

if __name__ == '__main__':
    gen = mygen(10)
    print(isinstance(gen, Generator))  # True
Copy the code

Iterables and iterators generate all values and store them in memory, while generators require elements to be generated temporarily to save time and space.

How do I run/activate the generator

Since the generator does not generate all elements at once, but executes them once and for all, how do you stimulate the generator to execute (or activate)?

There are two main ways to activate

usenext()
usegenerator.send(None)

Look at each example and you’ll see.

def mygen(n):
    now = 0
    while now < n:
        yield now
        now += 1

if __name__ == '__main__':
    gen = mygen(4)

    The two methods are equivalent by executing them alternately.
    print(gen.send(None))
    print(next(gen))
    print(gen.send(None))
    print(next(gen))
Copy the code

The output

0
1
2
3
Copy the code

. The execution state of the generator

A generator has four states during its life cycle

GEN_CREATED # Waiting for execution to start GEN_RUNNING # The interpreter is executing (this state is only visible in multi-threaded applications) GEN_SUSPENDED # At yield expression GEN_CLOSED # Execution completed

Get a feel for it in the code, but I won’t give you an example of GEN_RUNNING to make it easier to understand. If you’re interested, you can try multithreading. If you have any questions, please reply to me backstage.

from inspect import getgeneratorstate

def mygen(n):
    now = 0
    while now < n:
        yield now
        now += 1

if __name__ == '__main__':
    gen = mygen(2)
    print(getgeneratorstate(gen))

    print(next(gen))
    print(getgeneratorstate(gen))

    print(next(gen))
    gen.close()  Turn off/end the generator manually
    print(getgeneratorstate(gen))
Copy the code

The output

GEN_CREATED
0
GEN_SUSPENDED
1
GEN_CLOSED
Copy the code

. Exception handling of generators

When a generator is working, it will/should raise StopIteration if the generator does not meet the criteria for generating elements.

The generator, built through list generation, already does this for us automatically internally. Let’s take a look.

So when we define a generator ourselves, we should also throw an exception if the generated element condition is not met. Take the code above and modify it.

def mygen(n):
    now = 0
    while now < n:
        yield now
        now += 1
    raise StopIteration

if __name__ == '__main__':
    gen = mygen(2)
    next(gen)
    next(gen)
    next(gen)
Copy the code

Transition from generator to coroutine: yield

From the above, we know that generators give us the ability to suspend function execution (yield). Once the pause function is in place, people want to see if they can send something to the generator while it’s paused (send(None) is also mentioned above). This ability to send messages to paused generators came into Python 2.5 through PEP 342 and led to the creation of coroutines in Python. According to the definition on Wikipedia

A coroutine is a computer program component that generates subroutines for non-preemptive multitasking, allowing different entry points to pause or start execution of the program at different locations.

Note that coroutines are not concepts in the language per se, but in the programming model.

Coroutines are similar to threads, in that they only cross serial execution between multiple coroutines, just like threads; There are differences, too. Threads have to switch, lock, and unlock frequently, which is a pain point compared to coroutines in terms of complexity and efficiency. By using the yield pause generator, coroutines can transfer the execution flow of a program to other subroutines, thus implementing the alternating execution of different subroutines.

Here’s a brief demonstration of how to send a message to a generator.

def jumping_range(N):
    index = 0
    while index < N:
        The message sent via send() will be assigned to jump
        jump = yield index
        if jump is None:
            jump = 1
        index += jump

if __name__ == '__main__':
    itr = jumping_range(5)
    print(next(itr))
    print(itr.send(2))
    print(next(itr))
    print(itr.send(- 1))
Copy the code

The output.

0, 2, 3, 2Copy the code

So let me explain why I output this. Focus on the jump = yield index statement.

Divided into two parts:

yield indexIs to the indexreturnTo the external caller.
jump = yieldCan receive messages sent by external programs via send() and assign values tojump

All of these are the basic knowledge necessary to talk about coroutine concurrency, please be sure to practice and understand it, otherwise the following content will become boring and difficult to understand.

In the next chapter, I’ll cover a new syntax introduced in Python3.5: yield from. The space is also more, so take it out separately.

Are Python coroutines really that hard? Collection!

Iterable, iterator, generator

How do I run/activate the generator

. The execution state of the generator

. Exception handling of generators

Transition from generator to coroutine: yield

Related Posts

Download the Android SDK with Xunlei

Algorithm of the day: 88. Merge two ordered arrays

SQL audit query platform -Archery installation