This paper refers to:

• How the heck does async/await work in Python 3.5?
• PEP 380: Syntax for Delegating to a Subgenerator

The yield and yield the from

Let’s first learn or review the use of yield and yield from. If you are confident that you have understood, you can skip to the next section.

Python3.3 introduces a new syntax: yield from.

yield from iterator
Copy the code

Which is essentially equivalent to:

for x in iterator:
    yield x
Copy the code

In the following example, two yields add up to a large iterable(see 3.3 release) :

>>> def g(x):
...     yield from range(x, 0, -1)
...     yield from range(x)
...
>>> list(g(5))
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
Copy the code

Understanding yield from is crucial for the next section. To fully understand yield from, consider the official example:

def accumulate():
    tally = 0
    while 1:
        next = yield
        if next is None:
            return tally
        tally += next


def gather_tallies(tallies):
    while 1:
        tally = yield from accumulate()
        tallies.append(tally)

tallies = []
acc = gather_tallies(tallies)
next(acc) # Ensure the accumulator is ready to accept values

for i in range(4):
    acc.send(i)
acc.send(None) # Finish the first tally

for i in range(5):
    acc.send(i)
acc.send(None) # Finish the second tally
print(tallies)
Copy the code

I also recorded a video for this, which you can watch along with the text, or you can turn on PyCharm and any debugging tool and try it yourself. Video link

Let’s break down:

Starting with the line acc = Gather_tallies (tallies), since there is a yield in the gather_tallies function, while 1 is not executed immediately (as you can see from the video, ACC is a generator type).

Next (acc) :

Next () runs to the next yield or raises a StopIteration error.

Gather_tallies. There is a yield from accumulate(). Next (acc) does not stop there. It goes into subgenerator accumulate, and then at next = yield, it hits yield, suspends the function, and returns.

for i in range(4):
    acc.send(i)
Copy the code

Acc. Send (value)

• Step 1: Go back to where you paused last time
• Step 2: Assign the value of value toxxx = yieldIn thexxxIn this casenext.

The while loop in the accumulate function determines whether to exit by determining if the value of next is None. In the for loop for I in range(4), I is not None, so the while loop does not break. However, according to what we said earlier: Next () will run and stop at the next yield, and the while loop once again encounters yield, so it will suspend the function and return control to the main thread.

Just to clear things up: for accumulate, its loop does not end, and the next time it resumes with next(), it will still be running its loop. For Gather_tallies, his yield from accumulate() is still running. For the entire program, there are indeed several jumps between the main process and the accumulate body.

Next, the first acc.send(None) : the value of the next variable becomes None, if next is None, and returns tally to the previous level. (To do the math, Tally is 0 + 1 + 2 + 3 = 6). This return value is assigned to Gally in Gather_tallies. Note that the gather_tallies loop has not yet ended, so calling next(ACC) will not raise StopIteration.

for i in range(5):
    acc.send(i)
acc.send(None) # Finish the second tally
Copy the code

This part is the same logic as before. Acc.send (I) will go to gather_tallies and then accumulate, assigning the value to next. Acc. Send (None) Stops the loop. The final tally value is 10 (0 + 1 + 2 + 3 + 4).

The final tallies list is: [6,10].

A brief history of Python Async await

Take a look at wikipedia’s definition of a Coroutine:

Coroutines are computer program components that generalize subroutines for non-preemptive multitasking, by allowing execution to be suspended and resumed.

What is important is that execution to be suspended and resumed easily. In plain English, it means:

Coroutines are functions whose execution you can pause. (How the heck does async/await work in Python 3.5?)

Isn’t that what generators are?

Python2.2 – Generator origin

The concept of Python generators originated in pep255, which was dropped in Python 2.2(2001) and was inspired by the Icon programming language.

One advantage of generators is that they don’t waste space. Look at this example:

def eager_range(up_to):
    """Create a list of integers, from 0 to up_to, exclusive."""
    sequence = []
    index = 0
    while index < up_to:
        sequence.append(index)
        index += 1
    return sequence
Copy the code

If you use this function to generate a 10W list, you need to wait for the while loop to return. Then the sequence list will take up 10W elements. Not to mention time consuming (in terms of how long it takes to first use the Sequence list), it takes a lot of space.

With the yield from the previous section, modify a little:

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    while index < up_to:
        yield index
        index += 1
Copy the code

So you only need to take up one element, and you can use range immediately, instead of waiting for it to be all generated.

Python2.5: send stuff back

Some prescient predecessors thought that if we could take advantage of generators’ ability to pause, and somehow add send Stuff back, wouldn’t that fit wikipedia’s definition of a coroutine?

So pep342.

Pep342 mentions a send() method that allows us to send a “stuff” back to the generator to run again. Consider the following example:

def jumping_range(up_to):
    """Generator for the sequence of integers from 0 to up_to, exclusive. Sending a value into the generator will shift the sequence by that amount. """
    index = 0
    while index < up_to:
        jump = yield index
        if jump is None:
            jump = 1
        index += jump


if __name__ == '__main__':
    iterator = jumping_range(5)
    print(next(iterator))  # 0
    print(iterator.send(2))  # 2
    print(next(iterator))  # 3
    print(iterator.send(-1))  # 2
    for x in iterator:
        print(x)  # 3, 4
Copy the code

Send sends a “stuff” to the generator, assigns a value to jump, and then checks if jump is None to perform the corresponding logic.

Python3.3 yield from

No major improvements have been made to generators since Python2.5, until the introduction of pep380 in Python3.3. This PEP proposal makes writing generators much simpler by introducing yield from, which can be understood as syntactic sugar:

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    def gratuitous_refactor():
        nonlocal index
        while index < up_to:
            yield index
            index += 1
    yield from gratuitous_refactor()
Copy the code

We’ve covered yield from in detail in section 1, so we won’t repeat it here.

Python3.4 asyncio module

Interstitial: Eventloop

If you have js programming experience, you must know something about event loops.

The best and biggest way to understand a concept is to turn to Wikipedia:

An event loop “is a programming construct that waits for and dispatches events or messages in a program” – from event loop – wikipedia

In simple terms, eventLoop implements action B when event A occurs. Take the JavaScript event loop in A browser. When you click on something (A happens), the onclick function is defined (to do B).

In Python, Asyncio provides an eventloop (to recall the example from the previous article). Asyncio focuses on the network request domain, In this case, the “A event occurs” mainly means that sockets can be written and sockets can be read (via selectors).

By this time, Python had acquired the power of asynchronous programming in the form of Concurrent programming.

Concurrent programming executes on only one thread. Concurrency is not Parallelism is a great video for the Go blog.

Asyncio code for this era

The asyncio code for this period looks like this:

import asyncio

# Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html.
@asyncio.coroutine
def countdown(number, n):
    while n > 0:
        print('T-minus', n, '({}).format(number))
        yield from asyncio.sleep(1)
        n -= 1

loop = asyncio.get_event_loop()
tasks = [
    asyncio.ensure_future(countdown("A", 2)),
    asyncio.ensure_future(countdown("B", 3))]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
Copy the code

The output is:

T-minus 2 (A)
T-minus 3 (B)
T-minus 1 (A)
T-minus 2 (B)
T-minus 1 (B)
Copy the code

The asyncio.coroutine decorator is used to indicate that a function can be used in an asyncio event loop.

See yield from asyncio.sleep(1)? By yield from an asyncio.Future object, you give the Future object to the event loop, and while the object is waiting for something to happen (in this case, waiting for Asyncio.sleep (1), Wait 1s), pause the function and start doing something else. When something that the Future object is waiting for happens, the event loop notices and lets it resume running from where it was last paused by calling the send() method.

Break down the code above:

The event loop opens two countdown() coroutine calls and runs until yield from Asyncio.sleep (1), which returns a Future object and then pauses, after which the event loop monitors both Future objects. After 1 second, the event loop sends the Future Object send() to the Coroutine, which then runs again, printing t-minus 2 (A), etc.

Python3.5 async await

The python3.4

@asyncio.coroutine
def py34_coro():
    yield from stuff()
Copy the code

With Python3.5, you can use a more concise syntax:

async def py35_coro():
    await stuff()
Copy the code

This change, grammatically speaking, is not particularly different. What really matters is the rise of coroutines in Python’s philosophical stature. Before python3.4 and asynchronous functions more is a kind of common tag (modifier), after that, it has become a basic abstraction types (the abstract base class) : class collections. The Coroutine.

How the heck does async/await work in Python 3.5? The article also talked about the implementation of async and await underlying Bytecode, I will not go into depth here, after all, space is limited.

Think of async, await as API, not implementation

Python core developer (and one of my favorite PyCon Talkers) David M. Beazley said in this talk at PyCon Brasil 2015 that we should think of Async and await as apis, not implementations. In other words, async and await are not equal to asyncio, and asyncio is just an implementation of async and await. (of course asyncio makes asynchronous programming possible in Python3.4, which drives async, await)

He has also opened source a project github.com/dabeaz/curi… The underlying event loop mechanism is different from asyncio, which uses future objects and Curio, which uses tuples. At the same time, the two libraries have different focus points. Asyncio is a complete framework, while Curio is relatively lightweight and users have more to think about.

How the heck does async/await work in Python 3.5? This article also has a simple event loop implementation example, if you are interested, you can take a look, later if you have time may be implemented together.

To summarize

• Coroutines have only one thread.
• Operating system scheduling processes, coroutines with event loops scheduling functions.
• Async and await elevate the philosophical status of coroutines in Python.

The most important feeling is: Nothing is Magic. You should now be able to get a sense of Python’s coroutines as a whole.

If you love computer science and basic logic like me, welcome to follow my wechat official account: