coroutines

1. What is a coroutine?

   Coroutines are characterized by execution in a single thread.

2. Compared to multithreading

   Coroutines have the following advantages over multithreading

   • High execution efficiency of coroutines. The switching of coroutines does not cause thread switching and has no overhead of thread switching. Compared with multi-threading, the more coroutines there are, the more obvious the performance advantage of coroutines.
   • Coroutines do not require multithreaded locking. Because there is only one thread, there is no conflict in writing variables at the same time, the control of shared resources in the coroutine is not locked, only need to judge the state is good, so the execution efficiency is higher than multithreading.

Python support for coroutines is implemented through generators. In a generator, not only can we iterate through a for loop, we can also call next() continuously to get the next value returned by the yield statement. But Python’s yield can not only return a value, it can also receive arguments issued by the caller.

The traditional producer-consumer model is that one thread writes messages and one thread fetches messages, controlling queues and waits through locking mechanisms, but can be deadlocked if not carefully. When does a deadlock occur?

Producer consumers are implemented through coroutines. After the producer produces the message, the yield directly jumps to the consumer to start the execution. After the consumer completes the execution, the switch goes back to the producer to continue production, which is very efficient.

def consumer(a):
    r = ' '
    while True:
        n = yield r
        if not n:
            return
        print('[CONSUMER] Consuming %s... ' % (n,))
        r = '200K'
Copy the code

def produce(c):
    c.send(None)  Call c.end (None) to start the generator
    n = 0
    while n < 5:
        n += 1
        print('[PRODUCER] Producing %s... ' % n)
        r = c.send(n)
        print('[PRODUCER] Consumer return: %s' % r)
    c.close()
Copy the code

if __name__ == '__main__':
    c = consumer()
    produce(c)
Copy the code

Notice that the consumer function is a generator. Pass a consumer to produce:

1. Start the generator by calling c.end (None); C. end(None) The first time it enters the generator, it runs until yield, and the next time it continues from the assignment to n.
2. Then, once something is produced, switch to consumer execution via c. end(n);
3. The consumer uses yield to get the message, process it, and use yield to return the result. The argument n in c.end (n) is assigned to n before the yield statement in consumer, and judgment and output statements continue until either a direct return or the next yield statement is reached.
4. Produce gets the result of consumer processing and continues to produce the next message.
5. Produce decides not to produce, and closes the consumer with c.close(), ending the process.

The whole process is unlocked and executed by a single thread. Produce and consumer collaborate to complete the task, so it is called “coroutine” rather than preemptive multitasking by threads.

asyncio

A few basic concepts

Event_loop: The program starts an infinite loop to which the programmer registers functions (coroutines). When the conditions for the occurrence of an event are met, the corresponding coroutine function is called.

Coroutine: coroutine object. A function defined using the async keyword whose call does not execute the function immediately but returns a coroutine object. Coroutine objects need to be registered with the event loop to be called by the event loop.

Future object: Represents the result of tasks that will or will not be executed in the future. It is not fundamentally different from task.

Task: A coroutine object is a native function that can be suspended, and a task is a further encapsulation of the coroutine that contains the various states of the task. A Task object is a subclass of Future. It associates a Coroutine with a Future and encapsulates a Coroutine as a Future object.

Async /await keyword: PYTHon3.5 keyword used to define a coroutine, async defines a coroutine, and await is used to suspend the blocking asynchronous invocation interface. Its function is somewhat similar to yield.

practice

Asyncio is a standard library introduced in Python 3.4 that provides direct built-in support for asynchronous IO. Asyncio’s programming model is a message loop. We implement asynchronous IO by getting a reference to an EventLoop directly from the Asyncio module and then throwing the coroutine that needs to be executed into the EventLoop.

import asyncio

@asyncio.coroutine
def hello(a):
    print("Hello world!")
    Asyncio.sleep (1):
    r = yield from asyncio.sleep(1)
    print("Hello again!")

# get EventLoop:
loop = asyncio.get_event_loop()
# execution coroutine
loop.run_until_complete(hello())
loop.close()
Copy the code

@asyncio.coroutine marks a generator as a Coroutine and then throws the coroutine into an EventLoop for execution. Hello () first prints Hello World! The yield from syntax then allows us to easily invoke another generator. Since asyncio.sleep() is also a coroutine, the thread does not wait for asyncio.sleep(), but instead interrupts and executes the next message loop. When asyncio.sleep() returns, the thread can get the return value (None in this case) from yield and proceed to the next line. Asyncio.sleep (1) is considered an IO operation that takes 1 second, during which the main thread does not wait to execute any other executable coroutine in the EventLoop, thus allowing concurrent execution.

Asyncio can implement single-thread concurrent I/O operations. If only used on the client, the power is not great. If asyncio is used on the server side, such as web server, since HTTP connection is IO operation, it can achieve high concurrency support of multiple users with single thread +coroutine. Asyncio implements TCP, UDP, SSL and other protocols. Aiohttp is an HTTP framework based on Asyncio.

async

With the @asyncio.coroutine provided by Asyncio, you can mark a generator as a Coroutine type and then invoke another coroutine within the coroutine with yield from for asynchronous operations. To simplify and better identify asynchronous IO, new syntactic async and await were introduced starting with Python 3.5 to make Coroutine code more concise and readable. Note that async and await are the new syntax for coroutine, and to use the new syntax, just do two simple substitutions:

1. the@asyncio.coroutineReplace withasync;
2. theyield fromReplace withawait.

Asynchronous functions (coroutines)

Reference: zhuanlan.zhihu.com/p/27258289

Use asyC qualifiers to wrap normal and generator functions as asynchronous and asynchronous generators. Ordinary and generator functions

def function(a):
    return 1
 
def generator(a):
  yield 1
Copy the code

Use the async modifier to wrap ordinary functions and generator functions into asynchronous functions and asynchronous generators. Asynchronous functions (coroutines)

async def async_function(a):
    return 1
Copy the code

Asynchronous generator

async def async_generator(a):
    yield 1
Copy the code

print("type(func):", type(func) is types.FunctionType)	# True
print("type(generator):", type(generator()) is types.GeneratorType) # True

Calling an asynchronous function directly does not return a coroutine object
print("type(async_func):", type(async_func()) is types.CoroutineType) # True
print("type(async_generator):", type(async_generator()) is types.AsyncGeneratorType) # True
Copy the code

Calling an asynchronous function directly does not return a result, but rather a Coroutine object. Coroutines need to be driven in other ways, so you can use the send method of the coroutine object to send a value to the coroutine object.

async_func().send(None)
Copy the code

But the above call throws an exception.

async_func().send(None)
StopIteration: 1
Copy the code

This is because the generator/coroutine will raise a StopIteration exception upon normal return exit, and the original return value is stored in the StopIteration object’s value property. The true return value of the coroutine can be retrieved by exception catching.

try:
    async_func().send(None)
except StopIteration as e:
    print(e.value)
Copy the code

In coroutine functions, you can suspend your own coroutine with await syntax and wait for another coroutine until the result is returned. Await syntax can only appear in async modified functions, otherwise SyntaxError will be reported. And objects following await need an Awaitable or implement the associated protocol.

Asynchronous generator

Async /await code may not be async/await code. Async /await code is the syntactic sugar of coroutines, which makes the calls between coroutines clearer. Function calls decorated with async will return a coroutine object. Await can only be used within async modified functions. Await must be followed by a coroutine object or Awaitable. Await is intended to wait for the return of the coroutine control flow, while the operation to suspend and suspend a function is yield.

The instance

Asyncio’s programming model is a message loop. We implement asynchronous IO by getting a reference to an EventLoop directly from the Asyncio module and then throwing the coroutine that needs to be executed into the EventLoop.

import asyncio


@asyncio.coroutine
def wget(host):
    print('wget %s... ' % host)
    connect = asyncio.open_connection(host, 80)
    reader, writer = yield from connect
    header = 'GET/HTTP / 1.0 \ r \ nHost: % s \ r \ n \ r \ n' % host
    writer.write(header.encode('utf-8'))
    yield from writer.drain()
    while True:
        line = yield from reader.readline()
        if line == b'\r\n':
            break
        print('%s header > %s' % (host, line.decode('utf-8').rstrip()))
    # Ignore the body, close the socket
    writer.close()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    tasks = [wget(host) for host in ['www.sina.com.cn'.'www.sohu.com'.'www.163.com']]
    loop.run_until_complete(asyncio.wait(tasks))
    loop.close()
Copy the code

Coroutine @asyncio.coroutine marks a generator as a Coroutine type, and we put the coroutine in an event loop to execute. The yield from syntax makes it easier to invoke another generator. So instead of waiting for an IO operation, the thread interrupts and executes the next message loop. When the yield from returns, the thread can get the return value from the yield from and proceed to the next line. During this time, the main thread does not wait, but instead executes any other executable coroutine in the event loop, so three coroutines wrapped in Task can be executed concurrently by the same thread.

To execute three coroutines concurrently in a single thread with async and await syntax:

import asyncio


async def wget(host):
    print('wget %s... ' % host)
    connect = asyncio.open_connection(host, 80)
    reader, writer = await connect
    header = 'GET/HTTP / 1.0 \ r \ nHost: % s \ r \ n \ r \ n' % host
    writer.write(header.encode('utf-8'))
    await writer.drain()
    while True:
        line = await reader.readline()
        if line == b'\r\n':
            break
        print('%s header > %s' % (host, line.decode('utf-8').rstrip()))
    # Ignore the body, close the socket
    writer.close()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    tasks = [wget(host) for host in ['www.sina.com.cn'.'www.sohu.com'.'www.163.com']]
    loop.run_until_complete(asyncio.wait(tasks))
    loop.close()
Copy the code

reference

1. Coroutines and async keyword: zhuanlan.zhihu.com/p/27258289
2. Asynchronous IO:www.liaoxuefeng.com/wiki/101695…
3. Key words: yield blog.csdn.net/mieleizhi05…
4. Asyncio:www.liaoxuefeng.com/wiki/101695…
5. async：zhuanlan.zhihu.com/p/27258289
6. Python asynchronous IO framework aysncio: juejin.cn/post/684490…
7. Coroutines implementation producers consumers: www.liaoxuefeng.com/wiki/101695…