1. The relationship between process, thread and coroutine

1.1 A simple example illustrates the logical relationship among the three

  • There is an owner who wants to open a factory to produce a certain item (scissors, for example)

  • He needs to spend some money and resources to make a production line, and this production line has a lot of devices and materials and all the resources that are needed to be able to make scissors are called processes

  • Only the production line can not produce, so the boss has to find a worker to produce, the worker can use these materials to finally make scissors step by step, this worker to do things called thread

  • To increase productivity, the boss came up with three ideas:

    1. Recruit more workers on this production line to make scissors together, so that the efficiency is doubled, that is, the single-process multi-threaded way
    2. The boss found that the more workers on the production line, the better, because the resources and materials of a production line are limited after all, so the boss spent some financial and material resources to buy another production line, and then recruited some workers so that the efficiency was further improved, that is, the way of multi-process and multi-thread
    3. The boss found that there were already many production lines, and there were already many workers on each production line (i.e., the program was multi-process, and there were multiple threads in each process). In order to improve efficiency again, the boss came up with a trick: If an employee at work or temporary nothing to wait for some conditions (such as waiting for another worker finish production working procedure Before he can work again), then the employees will use this time to do other things, that is to say: if a thread is waiting for certain conditions, can make full use of the time to do other things, in fact this is: Coroutines way

1.2. Core relationships: Processes, threads and coroutines can all implement multitasking mode

  1. A process is a unit of resource allocation
  2. Threads are the unit of operating system scheduling
  3. Process switching requires the most resources and is inefficient
  4. Thread switching requires average resources and efficiency (without the GIL, of course)
  5. Coroutine switching task has small resources and high efficiency
  6. Multi-process, multi-thread depending on the number of CPU cores may be parallel, but the coroutine is in one thread, so it is concurrent

2. Implementation and demonstration of coroutines

Coroutines, also known as microthreads, English name Coroutine, Coroutine (in the same thread task switching collaborative execution). As follows, in the presence of a thread a function, can be in any place to save the current function of some information such as temporary variables, and then switch to another function, pay attention to the way to do it not by calling a function, and the number of switch and when to switch to the original functions are fixed by the developers themselves.

2.1. Implement coroutines using yield.

import time

def work1() :
    while True:
        print("----work1---")
        yield
        time.sleep(0.5)

def work2() :
    while True:
        print("----work2---")
        yield
        time.sleep(0.5)

def main() :
    w1 = work1()
    w2 = work2()
    while True:
        next(w1)
        next(w2)

if __name__ == "__main__":
    main()
''' ----work1--- ----work2--- ----work1--- ----work2--- ----work1--- .... Omit... ' ' '
Copy the code

2. Implement coroutines using greenlet

You can use the Greenlet module in Python to implement coroutine work, if you install the library first, which is done based on PyCharm

Pip3 install Greenlet # If pycharm is used, the module is now installed in the environmentCopy the code
from greenlet import greenlet
import time

def test1() :
    while True:
        print "---A--"
        gr2.switch()
        time.sleep(0.5)

def test2() :
    while True:
        print "---B--"
        gr1.switch()
        time.sleep(0.5)

gr1 = greenlet(test1)
gr2 = greenlet(test2)

# Switch to run in GR1
gr1.switch()

'''
---A--
---B--
---A--
---B--
---A--
---B--
---A--
---B--
'''
Copy the code

3. Use the ultimate module gEvent to implement coroutines

Gevent is a coroutine based Python network library that uses a high-level synchronization API from Greenlet that encapsulates the Libevent event loop. It lets developers write asynchronous I/O code synchronously without changing their programming habits.

Greenlet already implements coroutines, but this is still switched using switch. Python has a module that is even more powerful than Greenlet and can automatically switch tasks: GEvent. The idea is that when a Greenlet encounters an IO(input, output, file, etc.) operation, such as accessing the network, it will automatically switch to another Greenlet, wait until the IO operation is complete, and then switch back to continue at an appropriate time. Since IO operations are time-consuming and often leave the program in a wait state, having gEvent automatically switch coroutines for us ensures that greenlets are always running instead of waiting for IO

import gevent

def f1(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
def f2(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
def f3(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
print("Start executing ------")
print("------f1 ------")
g1 = gevent.spawn(f1, 5) Create a coroutine task that is actually creating a Greenlet object
print("------f2 ------")
g2 = gevent.spawn(f2, 5)
print("------f3 ------")
g3 = gevent.spawn(f3, 5)
g1.join()  #join() starts and waits for G1 execution to complete
g2.join()
g3.join()

"' start -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- f1 f2 -- -- -- -- - -- -- -- -- -- -- -- the f3 -- -- -- -- -- - < Greenlet at 0 x23a40197a48: f1(5)> 0 
      
        1 
       
         2 
        
          3 
         
           4 
          
            0 
           
             1 
            
              2 
             
               3 
              
                4 
               
                 0 
                
                  1 
                 
                   2 
                  
                    3 
                   
                     4'''
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
Copy the code

Summary: The above three greenlets are run sequentially rather than interchangeably. The first one is executed before the second one is executed.

1. Alternate execution is implemented through gevent.sleep(). When G1 is in sleep state, switch to G2 or G3 for execution.

import gevent ,time

def f1(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
        gevent.sleep(0.5)

def f2(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
        gevent.sleep(0.5)
def f3(n) :
    for i in range(n):
        print(gevent.getcurrent(), i)
        gevent.sleep(0.5)  # to simulate a time-consuming operation, note that it is not sleep in the time module
print("Start executing ------")
print("------f1 ------")
g1 = gevent.spawn(f1, 5) Create a coroutine task that is actually creating a Greenlet object
print("------f2 ------")
g2 = gevent.spawn(f2, 5)
print("------f3 ------")
g3 = gevent.spawn(f3, 5)
g1.join()  #join() waits for G1 execution to complete
g2.join()
g3.join() # You can use joinall([]) instead

"' start -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- f1 f2 -- -- -- -- - -- -- -- -- -- -- -- the f3 -- -- -- -- -- - < Greenlet at 0 x2226aa27a48: f1(5)> 0 
      
        0 
       
         0 
        
          1 
         
           1 
          
            1 
           
             2 
            
              2 
             
               2 
              
                3 
               
                 3 
                
                  3 
                 
                   4 
                  
                    4 
                   
                     4 '''
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
Copy the code

2. Of course, in actual code, we won’t use itgevent.sleep()Instead of switching coroutines, gEvent switches automatically when an IO operation is performed, such as:

from gevent import monkey
import gevent
import random
import time

Patch_all () is equivalent to gevent.sleep().
monkey.patch_all()  # Replace the code of time-consuming operation used in the program with the module implemented by itself in GEvent to achieve coroutine switching

def ct_work(args) :
    for i in range(4) :print(args, i)
        time.sleep(random.random())

gevent.joinall([ # replaces the join-by-join () operation.
        gevent.spawn(ct_work, "work1"),
        gevent.spawn(ct_work, "work2")])'''' Note that if the time-consuming operation of patch_all() is not added, it is output sequentially. work1 0 work2 0 work2 1 work1 1 work1 2 work2 2 work2 3 work1 3 '''
Copy the code

Scream tips:

The performance of using Gevent is indeed higher than using traditional threads, even much higher. But here’s one pit:

  1. Monkey-patching is what we call a Monkey patch because, if used, Gevent directly modifies most of the blocking system calls in the standard library, including the socket, SSL, threading, and SELECT modules, to run cooperatively. However, we can’t guarantee where you can use these libraries in a complex production environment and have strange problems due to patching
  2. Third-party library support. Make sure that other network libraries used in your project also use pure Python or explicitly support Gevent

Unified statement: About the original blog content, there may be some content reference from the Internet, if there is an original link will be quoted; If can not find the original link, in this statement if there is infringement please contact to delete ha. About reprint blog, if have original link will declare; If can not find the original link, in this statement if there is infringement please contact to delete ha.