Today evening what evening old people do not come late lianshandai

I have looked into python coroutines before, but have not explored them in depth. I still use Python2 for most of my work, but my recent project required python3 coroutines, so I got around to learning python3 coroutine syntax. This article focuses on python3.5’s async/await coroutine syntax, which is awkward and difficult to understand. If you are not familiar with the basics of Python coroutines, you are advised to read this article: Python Coroutines.

Coroutine function (asynchronous function)

The most commonly used functions are synchronous functions, where different functions are executed in sequence. So what is an asynchronous function? How do I create asynchronous functions? How do I switch back and forth between asynchronous functions? No hurry, please look down.

Create coroutine functions

Let’s start with ordinary functions:

def test1():
    print("1")
    print("2")
def test2():
    print("3")
    print("4")
a = test1()
b = test2()
print(a,type(a))
print(b,type(b))Copy the code

Run the above code to get the result:

1
2
3
4
None <class 'NoneType'>
None <class 'NoneType'>Copy the code

Test1; test2; test2;

Then use the async keyword to turn ordinary functions into coroutines, i.e. asynchronous functions:

async def test1():
    print("1")
    print("2")
async def test2():
    print("3")
    print("4")
print(test1())
print(test2())Copy the code

Run the above code to get the result:

<coroutine object test1 at 0x109f4c620>
asyncio_python3_test.py:16: RuntimeWarning: coroutine 'test1' was never awaited
  print(test1())
<coroutine object test2 at 0x109f4c620>
asyncio_python3_test.py:17: RuntimeWarning: coroutine 'test2' was never awaited
  print(test2())Copy the code

Note: Ignoring the alarm in the result, you can see that test1 and test2 calls do not enter the function body and execute the function contents, but return a coroutine.

In addition to functions, class methods can also be converted into coroutine methods using the async keyword:

class test:
    async def run(self):
        print("1")Copy the code

Execute coroutine functions

We successfully created the coroutine function and returned a coroutine object when calling the function. How do we enter the function body and execute the function contents? Like a generator, you can use the send method to execute the function and modify the previous code:

async def test1():
    print("1")
    print("2")
async def test2():
    print("3")
    print("4")
a = test1()
b = test2()
a.send(None)
b.send(None)Copy the code

Running the above code results in the following:

1
2
Traceback (most recent call last):
  File "asyncio_python3_test.py", line 19, in <module>
    a.send(None)
StopIteration
sys:1: RuntimeWarning: coroutine 'test2' was never awaitedCopy the code

Test1 coroutine test1 is executed first. When test1 is finished, StopIteration is raised. This is an exception that the coroutine function returns after execution.

async def test1(): print("1") print("2") async def test2(): print("3") print("4") a = test1() b = test2() try: A.end (None) # Call the coroutine function except StopIteration as e: Print (e.value) # Raise a StopIteration exception when the coroutine function is finished, and return the value in the value pass try: B. end(None) # Call the send function except StopIteration: Print (e.value) # Raise a StopIteration exception when the coroutine function is finished, and return value pass in valueCopy the code

Running the above code results in the following:

1
2
3
4Copy the code

Test1 is executed first, and test2 is executed after test1 is completed. From the point of view of the execution process, the current coroutine function is no different from ordinary functions, and there is no asynchronous function, so how to run the coroutine function cross?

Cross execution of coroutine functions (await)

From the above examples, we found that the coroutine functions can be defined using the async keyword, and the function can be executed using the SEND method. So how to switch between two coroutine functions? Here we need to use the await keyword and modify the code:

import asyncio async def test1(): Print ("2") async def test2(): async def test2(): async def test2(): async def test2(): async def test2(): async def test2() Print ("3") print("4") a = test1() b = test2() try: a.end (None) Pass try: b.end (None) # Except StopIteration: pas can be executed by calling sendCopy the code

Running the above function yields the following result:

1, 3, 4,Copy the code

The test1 coroutine function is executed first, and the test1 coroutine function stops executing (blocking) while await execution. The test2 coroutine function is then executed until test2 completes. Print (” 2 “) print(” 2 “) print(” 2 “) print(” 2 “) You can use asyncio’s built-in methods to loop through coroutine functions.

Await with jam

With async you can define coroutine objects, and with await you can suspend for time-consuming operations, just like yield in generators, where the function cedes control. If a coroutine encounters an await, the event loop will suspend the coroutine and execute other coroutines until the other coroutines are also suspended or completed before the next coroutine is executed. The purpose of the coroutine is also to asynchronize time-consuming operations.

Note: “Await” must be followed by an Awaitable object, or an object implementing the corresponding protocol. Looking at the code for the Awaitable abstract class, we can see that whenever a class implements an await method, an instance constructed from it is an Awaitable. The Coroutine class also descends from Awaitable.

Automatic loop execution of coroutine functions

We know from the previous section that we need to use the SEND method to execute coroutines, but once the coroutine function is switched to another function, the function is no longer run, and using the SNED method is not very efficient. So how do you automatically execute all coroutine functions during the entire program execution, just like multithreading, multiprocess, implicitly execute functions instead of explicitly execute functions via send?

Event loop method

Asyncio.get_event_loop (asyncio.get_event_loop, asyncio.get_event_loop)

import asyncio
async def test1():
    print("1")
    await test2()
    print("2")
async def test2():
    print("3")
    print("4")
loop = asyncio.get_event_loop()
loop.run_until_complete(test1())Copy the code

Running the above code results in the following:

1, 3, 4, 2Copy the code

Description: The asyncio.get_event_loop method creates an event loop, then registers the coroutine with run_until_complete and starts the event loop.

Of task

Since the coroutine object cannot be run directly, it is the run_until_complete method that wraps the coroutine as a task object when the event loop is registered. A task object is a subclass of a Future class that stores the state of the coroutine after it is run and is used to obtain the results of the coroutine in the Future. We can also manually define the coroutine object as task by modifying the above code as follows:

import asyncio
async def test1():
    print("1")
    await test2()
    print("2")
async def test2():
    print("3")
    print("4")
loop = asyncio.get_event_loop()
task = loop.create_task(test1())
loop.run_until_complete(task)Copy the code

The state of the coroutine is stored in the task, and the return value of the coroutine function can be retrieved. There are two ways to do this: one involves binding the callback function, and the other prints it directly after the task is run. It is worth mentioning that if the send method is used to execute the function, the return value can be obtained using StopIteration.

Output task results directly

When the coroutine function is finished, we need to get the return value. The first way is to call the result method of the task when the task status is finish.

import asyncio
async def test1():
    print("1")
    await test2()
    print("2")
    return "stop"
async def test2():
    print("3")
    print("4")
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(test1())
loop.run_until_complete(task)
print(task.result())Copy the code

Running the above code results in the following:

1
3
4
2
stopCopy the code

The callback function

The second way to retrieve the return value is by binding a callback function to retrieve the result of the task at the end of execution. The last argument to the callback is a Future object from which the coroutine return value can be retrieved.

import asyncio async def test1(): print("1") await test2() print("2") return "stop" async def test2(): print("3") print("4") def callback(future): Loop = asyncio.get_event_loop() task = print('Callback:',future.result() Asyncio.ensure_future (test1()) # create task, Test1 () is a coroutine object task.add_done_callback(callback) # bind callback function loop.run_until_complete(task)Copy the code

Running the above code results in the following:

1
3
4
2
Callback: stopCopy the code

If the callback function needs to accept more than one argument, it can be imported via a partial function and modify the code as follows:

import asyncio
import functools
async def test1():
    print("1")
    await test2()
    print("2")
    return "stop"
async def test2():
    print("3")
    print("4")
def callback(param1,param2,future):
    print(param1,param2)
    print('Callback:',future.result())
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(test1())
task.add_done_callback(functools.partial(callback,"param1","param2"))
loop.run_until_complete(task)Copy the code

The future object in the callback function is the task object created.

The future object

The future object has several states: Pending, Running, Done, and Cancelled. When a future is created, the task is “pending”. When the event loop is called, it is running, and when the event loop is called, it is done. If you need to stop the event loop, you must cancel the task first.

Coroutines stop

We’ve seen how to use event loops to execute coroutine functions, so how do you stop execution? Before stopping the execution of the coroutine, you need to cancel the Task and then stop the loop event loop.

import asyncio
async def test1():
    print("1")
    await asyncio.sleep(3)
    print("2")
    return "stop"
tasks = [
    asyncio.ensure_future(test1()),
    asyncio.ensure_future(test1()),
    asyncio.ensure_future(test1()),
]
loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(asyncio.wait(tasks))
except KeyboardInterrupt as e:
    for task in asyncio.Task.all_tasks():
        task.cancel()
    loop.stop()
    loop.run_forever()
finally:
    loop.close()Copy the code

Run the above code and press CTRL + C to end the execution.

Some concepts and methods used in this article

  • Event_loop: A program starts an infinite loop in which functions registered to the event loop are called when the event condition is met.
  • Coroutine object: A function defined using the async keyword. The call does not execute the function immediately, but returns a coroutine object that needs to be registered with the event loop and called by the event loop.
  • Task: A coroutine object is a native function that can be suspended, and a task is a further encapsulation of the coroutine that contains the various states of the task.
  • Future: represents the result of a task that will or will not be executed in the future. It is not fundamentally different from a task
  • Async /await keyword: PYTHon3.5 keyword used to define a coroutine, async defines a coroutine, and await is used to suspend the blocking asynchronous invocation interface.

Concurrency and parallelism

Concurrency usually refers to multiple tasks that need to be executed at the same time, while parallelism refers to multiple tasks that need to be executed at the same time. With multi-thread, multi-process, coroutine, coroutine to achieve concurrency, multi-thread and multi-process to achieve parallel.

How do asyncio coroutines implement concurrency

To achieve concurrency asyncio needs multiple coroutines to complete the task, await them whenever a task is blocked, and then the other coroutines continue to work. This requires creating a list of multiple coroutines and registering them in an event loop. Multiple coroutines can be multiple coroutine functions or multiple coroutine objects of a single coroutine function.

import asyncio async def test1(): print("1") await asyncio.sleep(1) print("2") return "stop" a = test1() b = test1() c = test1() tasks = [ asyncio.ensure_future(a), asyncio.ensure_future(b), asyncio.ensure_future(c), ] loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.wait(tasks)) print("task result is ",task.result())Copy the code

Running the above code results in the following:

1
1
1
2
2
2
task result is  stop
task result is  stop
task result is  stopCopy the code

Description: The code defines three coroutine objects, then creates three tasks using the asyncio.ensure_future method, adds all tasks to the task list, and finally adds the task list to the event loop using loop.run_until_complete.

Coroutines crawler

Earlier, you saw how to create coroutine functions using async and await, and create event loops using asyncio.get_event_loop and execute coroutine functions. The example shows the efficiency of coroutine concurrency, but how do you develop coroutines in a real application scenario? Take asynchronous crawlers for example. I tried to use the Requests module and URllib module to write asynchronous crawlers, but the actual operation found that asyncio asynchronous is not supported, so we can use the AIOHTTP module to write asynchronous crawlers.

Aiohttp implementation

import asyncio
import aiohttp
async def run(url):
    print("start spider ",url)
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            print(resp.url)
url_list = ["https://thief.one","https://home.nmask.cn","https://movie.nmask.cn","https://tool.nmask.cn"]
tasks = [asyncio.ensure_future(run(url)) for url in url_list]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))Copy the code

Running the above code results in the following:

start spider  https://thief.one
start spider  https://home.nmask.cn
start spider  https://movie.nmask.cn
start spider  https://tool.nmask.cn
https://movie.nmask.cn
https://home.nmask.cn
https://tool.nmask.cn
https://thief.oneCopy the code

Aiohttp based on asyncio implementation, can be used to write webserver, can also be used as a crawler.

Requests to realize

Because the Requests module blocks the unique thread of client code and asycio event loops, the entire application freezes when the call is made, but if you must use the Requests module, you can use the run_in_executor method of the event loop object, Create a new thread to execute the time-consuming function using the run_in_executor method, so you can modify the code implementation as follows:

import asyncio
import requests
async def run(url):
    print("start ",url)
    loop = asyncio.get_event_loop()
    response = await loop.run_in_executor(None, requests.get, url)
    print(response.url)
    
url_list = ["https://thief.one","https://home.nmask.cn","https://movie.nmask.cn","https://tool.nmask.cn"]
tasks = [asyncio.ensure_future(run(url)) for url in url_list]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))Copy the code

To parameterize requests, use funcTools:

import asyncio import requests import functools async def run(url): print("start ",url) loop = asyncio.get_event_loop() try: response = await loop.run_in_executor(None,functools.partial(requests.get,url=url,params="",timeout=1)) except Exception  as e: print(e) else: print(response.url) url_list = ["https://thief.one","https://home.nmask.cn","https://movie.nmask.cn","https://tool.nmask.cn"] tasks = [asyncio.ensure_future(run(url)) for url in url_list] loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.wait(tasks))Copy the code

Asyncio uses blocking functions

Just as we saw how to use the Requests module in Asyncio, what if we wanted to use other blocking functions in asyncio? There are asynchronous functions that support asyncio, but the real problem is that most IO modules do not support Asyncio.

Problems with blocking functions used in Asyncio

Blocking functions (such as IO reads and writes, Requests network requests) block the unique thread of client code and asyCIO event loops, so the entire application freezes when the call is executed.

The solution

The solution to this problem is to use the run_in_executor method of the event loop object. The asyncio event loop maintains a ThreadPoolExecutor object behind it. We can call the run_in_executor method and send it callable objects to execute. That is, we can create a new thread using the run_in_executor method to execute time-consuming functions.

Run_in_executor method

AbstractEventLoop.run_in_executor(executor, func, *args)Copy the code
  • The Executor argument should be an executor instance. If None, the default executor is used.
  • Func is the function to execute
  • Args is the argument passed to func

A practical example (using time.sleep()) :

import asyncio
import time
async def run(url):
    print("start ",url)
    loop = asyncio.get_event_loop()
    try:
        await loop.run_in_executor(None,time.sleep,1)
    except Exception as e:
        print(e)
    print("stop ",url)
url_list = ["https://thief.one","https://home.nmask.cn","https://movie.nmask.cn","https://tool.nmask.cn"]
tasks = [asyncio.ensure_future(run(url)) for url in url_list]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))Copy the code

Run the above code to get the following function:

start  https://thief.one
start  https://home.nmask.cn
start  https://movie.nmask.cn
start  https://tool.nmask.cn
stop  https://thief.one
stop  https://movie.nmask.cn
stop  https://home.nmask.cn
stop  https://tool.nmask.cnCopy the code

Note: With the run_in_executor method, we can create coroutine concurrency using previously familiar modules, rather than using specific modules for IO asynchronous development.

reference

www.oschina.net/translate/p… www.jianshu.com/p/b5e347b3a… Zhuanlan.zhihu.com/p/27258289 juejin. Cn/post / 684490…