Understand Python coroutines

Generator (Generator)
- yieldUse of expressions
- Producer and consumer models
- yield fromexpression
Coroutines (Coroutine)
- @asyncio.coroutine
- async/await
conclusion
Refer to the link

Python multithreaded performance is even worse than single-threaded performance due to GIL.

GIL: Global Interpreter Lock (GIL), a mechanism used by computer programming language interpreters to synchronize threads so that only one thread is executing at any one time. [1] Even on multi-core processors, interpreters using GIL only allow one thread to execute at a time.

So there’s this thing called Coroutine.

Coroutines: Coroutines, also called microthreads, fibers, Coroutine. The purpose of A coroutine is to interrupt function A at any time, to execute function B, and then interrupt to continue function A (you can switch freely). But this process is not a function call (there is no call statement), the whole process looks like multithreading, but the coroutine is executed by only one thread.

Because the program controls the switch actively, the coroutine does not have the overhead of thread switch, so the execution efficiency is very high. This is suitable for IO – intensive tasks. If CPU – intensive tasks are performed, multiple processes and coroutines are recommended.

Prior to Python3.4, there was no official support for coroutines, and implementations of tripartites libraries such as gevent and Tornado existed. The asyncio standard library has been built in since 3.4, which officially implements coroutines.

Python supports coroutines, which are generators that follow certain rules, through generators. So before we get to coroutines, we need to learn about generators.

Generator (Generator)

We’ll focus on the yield and yield from expressions, which are closely related to the implementation of coroutines.

Python2.5 introduced inyieldExpression, seePEP342
Python3.3 addyield fromSyntax, seePEP380.

Once a yield expression is included in a method, Python treats it as a generator object rather than a normal method.

`yield`Use of expressions

Let’s look at how this expression is used:

def test(): print("generator start") n = 1 while True: Yield_expression_value = yield N print("yield_expression_value = %d" % yield_expression_value) n += 1 # The generator = test () print (type (generator) print (" \ n -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- \ n ") # 2. Start the generator next_result = the generator. __next__ () Print (" next_result = % d "% next_result) print (" \ n -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- \ n") # 3. Send the value to yield expression send_result = the generator. The send (666). print("send_result = %d" % send_result)Copy the code

Execution Result:

<class 'generator'>

---------------

generator start
next_result = 1

---------------

yield_expression_value = 666
send_result = 2
Copy the code

Method description:

The __next__() method starts or resumes the execution of the generator, equivalent to send(None).
The send(value) method sends a value to a yield expression. To start the generator send(None) is called

Description of execution results:

① Create a Generator object: a function that contains a yield expression is no longer a function and will return a Generator object when called
② Start the generator. Before using the generator, call __next__ or send(None). Otherwise, an error will be reported. After starting the generator, the code will execute to where yield occurs, that is, to yield n, and then pass n to the return value on the line generator.__next__(). (Note that the generator will pause at yield n until the next time the generator is started.)
③ Sending a value to a yield expression: Calling the send method sends a value to the yield expression while resuming the execution of the generator. The generator continues down from where it was last interrupted, and at the next yield, the generator pauses again and switches to the main function to print send_result.

The key to understanding this demo is that the generator starts or resumes execution once and will be paused at yield. The yield_expression_value yield_expression_value yield_expression_value yield_expression_value yield_expression_value yield_expression_value

Producer and consumer models

In the example above, the code interrupts –> switch execution, which is part of the characteristic of coroutines.

Let’s take another producer/consumer example from Liao Xuefeng’s Python tutorial:

The traditional producer-consumer model is that one thread writes messages and one thread fetches messages, controlling queues and waits through locking mechanisms, but can be deadlocked if not carefully.

Instead of using a coroutine, the producer produces the message and then switches directly to the consumer for execution. When the consumer finishes executing the message, the producer switches back to continue production.

def consumer(): print("[CONSUMER] start") r = 'start' while True: n = yield r if not n: print("n is empty") continue print("[CONSUMER] Consumer is consuming %s" % n) r = "200 ok" def producer(c): # start generator start_value = c.end (None) print(start_value) n = 0 while n < 3: n += 1 print("[PRODUCER] Producer is producing %d" % n) r = c.send(n) print('[PRODUCER] Consumer return: %s' % r) # Close generator c.lose () # create generator c = consumer() # pass generator producer(c)Copy the code

Execution Result:

[CONSUMER] start start [PRODUCER] producer is producing 1 [CONSUMER] consumer is consuming 1 [PRODUCER] Consumer return:  200 ok [PRODUCER] producer is producing 2 [CONSUMER] consumer is consuming 2 [PRODUCER] Consumer return: 200 ok [PRODUCER] producer is producing 3 [CONSUMER] consumer is consuming 3 [PRODUCER] Consumer return: 200 okCopy the code

Notice that the consumer function is a generator. Pass a consumer to produce:

First callc.send(None)Start generator;

And then, once you’ve produced something, go throughc.send(n)Switch to consumer execution;

consumerthroughyieldGot the message, processed it, passed ityieldSend the results back;

producegetconsumerProceed to produce the next message as a result of processing;

produceWe’ve decided not to produce. Passc.close()Shut downconsumerThe whole process is over.

The whole process is unlocked and executed by a single thread. Produce and consumer collaborate to complete the task, so it is called “coroutine” rather than preemptive multitasking by threads.

`yield from`expression

Python3.3 added the yield from syntax for delegating part of a generator operation to another generator. In addition, the child generator (that is, the “parameter” after yield from) is allowed to return a value that can be used by the delegate generator (that is, the generator that contains yield from). And in the delegate generator, you can optimize the child generator.

Let’s start with the simplest applications, such as:

Def test(n): def test(n): def test(n): def test(n): def test(n): def test(n): def test(n): print("test_yield_from start") yield from test(n) print("test_yield_from end") for i in test_yield_from(3): print(i)Copy the code

Output:

test_yield_from start
0
1
2
test_yield_from end
Copy the code

Here we just add some printing to the generator, if it is in formal code, you can add normal execution logic.

If there are two yield FROM statements in the test_yield_FROM function above, they are executed serially. For example, rewrite the test_yield_from function above like this:

def test_yield_from(n):
    print("test_yield_from start")
    yield from test(n)
    print("test_yield_from doing")
    yield from test(n)
    print("test_yield_from end")
Copy the code

The output:

test_yield_from start
0
1
2
test_yield_from doing
0
1
2
test_yield_from end
Copy the code

In this case, the effect of yield from is equivalent to the short form of the following

for item in test(n):
    yield item
Copy the code

It doesn’t look like this yield from does anything, but it actually helps us handle exceptions and stuff like that. In practice, what are the main uses for the new “yield from” syntax In Python 3.3?

Coroutines (Coroutine)

Python3.4 added asyncio-related apis, syntax usage@asyncio.coroutineandyield fromImplement coroutines
Python3.5 introduced inasync/awaitSyntax, seePEP492

Let’s look at the Python3.4 implementation first.

`@asyncio.coroutine`

In Python3.4, functions decorated with @asyncio.coroutine are called coroutines. But it’s not strictly syntactically constrained.

For those of you who are not familiar with decorators, check out my last blog, Understanding Python Decorators.

For coroutines supported natively by Python, Python makes some distinctions between coroutines and generators to help disambiguate these two different but related concepts:

The tag@asyncio.coroutineDecorator functions are called coroutine functions,iscoroutinefunction()Method returns True
The object returned by calling a coroutine function is called a coroutine object,iscoroutine()Function returns True

For example, we added @asyncio.coroutine to the yield from demo above:

import asyncio ... @asyncio.coroutine def test_yield_from(n): ... Print # whether coroutines function (asyncio iscoroutinefunction (test_yield_from) # whether coroutines object print (asyncio. Iscoroutine (test_yield_from (3)))Copy the code

No doubt the output is True.

Take a look at the @asyncio.coroutine source code to see what it does. I’ve simplified the source code to look like this:

Import functools import types import inspect def coroutine (func) : # judgment whether the generator if inspect. Isgeneratorfunction (func) : Wraps (func) def coro(*args, **kw): res = func(*args, **kw) res = yield from res return res # Convert generator to coroutine wrapper = types. Coroutine (coro) # For iscoroutinefunction(). wrapper._is_coroutine = True return wrapperCopy the code

Marking this decorator on a generator converts it to a Coroutine.

Then, let’s actually use @asyncio.coroutine and yield from:

import asyncio @asyncio.coroutine def compute(x, y): print("Compute %s + %s ..." % (x, y)) yield from asyncio.sleep(1.0) return x + y @asyncio.coroutine def print_sum(x, y): result = yield from compute(x, y) print("%s + %s = %s" % (x, y, Result) loop = asyncio.get_event_loop() print("start") # Run_until_complete (print_sum(1, 2)) print("end") loop.close()Copy the code

Execution Result:

start
Compute 1 + 2 ...
1 + 2 = 3
end
Copy the code

Print_sum is a coroutine that calls subcoroutine compute and waits for compute to finish before returning the result.

The demo point call flow is shown below:

EventLoop will wrap print_sum as a Task

The flowchart shows the control flow of the demo, but not all the details. For example, the 1s “paused” actually creates a future object, and then wakes up the task 1s later via baseeventloop.call_later ().

It is worth noting that @asyncio.coroutine will be removed in Python3.10.

`async`/`await`

Python3.5 began introducing the async/await syntax (PEP 492) to simplify the use of coroutines and make them easier to understand.

Async /await is really just syntactic sugar for @asyncio.coroutine and yield from:

the@asyncio.coroutineReplace withasync
theyield fromReplace withawait

Can.

Take the example above:

import asyncio


async def compute(x, y):
    print("Compute %s + %s ..." % (x, y))
    await asyncio.sleep(1.0)
    return x + y


async def print_sum(x, y):
    result = await compute(x, y)
    print("%s + %s = %s" % (x, y, result))


loop = asyncio.get_event_loop()
print("start")
loop.run_until_complete(print_sum(1, 2))
print("end")
loop.close()
Copy the code

Let’s look at another example of a Future in Asyncio:

import asyncio

future = asyncio.Future()


async def coro1():
    print("wait 1 second")
    await asyncio.sleep(1)
    print("set_result")
    future.set_result('data')


async def coro2():
    result = await future
    print(result)


loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait([
    coro1()
    coro2()
]))
loop.close()
Copy the code

Output result:

Wait 1 second set_result dataCopy the code

The future object can be called after yield from or await in a coroutine, which suspends the coroutine until it finishes execution or returns a result or throws an exception.

In our example, await future must wait for future.set_result(‘data’) to finish. Coro2 () as the second coroutine may not be obvious enough, but instead the coroutine is called like this:

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait([
    # coro1(),
    coro2(),
    coro1()
]))
loop.close()
Copy the code

The output is still the same as above.

In fact, the keyword async can be used not only for functions, but also async with asynchronous context manager and async for asynchronous iterator. Interested in these and feel useful can look for information on the Internet, here is limited to the length of the spread out more.

conclusion

This article has studied, explored, and summarized generators and coroutines, but has not done much in-depth research. Just to get started with a note, I’ll try to implement the asynchronous API myself, hopefully to help you understand.

Refer to the link

Python coroutines https://thief.one/2017/02/20/Python%E5%8D%8F%E7%A8%8B/

www.dabeaz.com/coroutines/…

Coroutines

How the heck does async/await work in Python 3.5

Python3.4 coroutine documentation

Python3.5 coroutine documentation

Python tutorial by Liao Xuefeng — Coroutines

Generator (Generator)

yieldUse of expressions

Producer and consumer models

yield fromexpression

Coroutines (Coroutine)

@asyncio.coroutine

async/await

conclusion

Refer to the link

Related Posts

Wrapper classes and constant pools for basic data types

Analyze the Coredump file using GDB to troubleshoot occasional SRS memory leaks

Python 3.10 has just been released! I found a terrible feature…

`yield`Use of expressions

`yield from`expression

`@asyncio.coroutine`

`async`/`await`