Concurrent Programming in Python: A Performance Perspective

. Contents of this article

• preface
• The basic concepts of concurrent programming
• Single thread vs. multi-thread vs. multi-process
• Summary of performance comparison results

Preface.

As a branch of the advanced series “Concurrent programming,” I think this is something every programmer should be able to do.

I prepared this series of concurrent programming for nearly a week, from combing the knowledge points to thinking about what examples to use to make it easier for people to understand these points. Hope to present the effect really can be as imagined in that way, also as friendly to small white.

I roughly sorted out the contents of this series yesterday (it may be adjusted later) :

Course outline

For concurrent programming, Python implementation, summarized, there are roughly the following three methods:

• multithreading
• Multiple processes
• Coroutines (generators)

In the following chapters, we will gradually introduce these three knowledge points.

The basic concepts of concurrent programming

Before we get into the theory, let’s go over some basic concepts. Even though this is an advanced tutorial, I want it to be smaller and easier to understand.

Serial: a person can only do one thing at a time, such as watching TV after eating dinner; Parallel: a person can do more than one thing at a time, such as eating while watching TV;

In Python, multithreading and coroutines, while strictly serial, are much more efficient than regular serial programs. Ordinary serial programs can do nothing but wait while the program blocks. It’s like, you know, after the TV show, it’s a commercial break, and we can’t go eat during the commercial break. This is obviously very inefficient and unreasonable for the program.

Of course, after this course, we learned to use the commercial time to do other things, flexible time. This is also what we multithreaded and coroutine to help us to accomplish, internal reasonable scheduling of tasks, to maximize the efficiency of the program.

Although multithreading and coroutines are pretty smart. But it’s still not productive enough. The most productive would be multitasking, eating and talking while watching TV. That’s what our multi-process can do.

To help you understand more intuitively, I found two pictures on the Internet that vividly explain the difference between multi-threading and multi-process. To delete (cut)

• Multithreading, alternate execution, another sense of serial.

  

• Multi-process, parallel execution, true concurrency.

  

Single thread vs. multi-thread vs. multi-process

Words are powerless, and a thousand words are not as powerful as a few lines of code.

First of all, my experimental environment configuration is as follows

Pay attention to the following code, if you want to understand, there are following knowledge requirements for xiaobai:

1. The use of decorators
2. The basic use of multithreading
3. Basic use of multiple processes

Of course, do not understand it does not matter, the main final conclusion, can let you have a general clear understanding of the single thread, multi-thread, multi-process in the realization of the effect, to achieve this effect, the mission of this article is completed, until the last, learn the complete series, may wish to come back to understand maybe there will be a deeper understanding.

Let’s take a look at which one is better than the other.

To begin the comparison, define four types of scenarios

• CPU intensive
• Disk IO intensive
• Network IO intensive
• IO intensive [simulation]

Why are these scenarios? It has to do with the applicable scenarios of multi-threading and multi-process. I’ll show you in the conclusion.

# CPU intensive
def count(x=1, y=1):
    Make the program complete 1.5 million calculations
    c = 0
    while c < 500000:
        c += 1
        x += x
        y += y


Disk reads and writes are IO intensive
def io_disk():
    with open("file.txt"."w") as f:
        for x in range(5000000):
            f.write("python-learning\n")


Network IO intensive
header = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'}
url = "https://www.tieba.com/"

def io_request():
    try:
        webPage = requests.get(url, headers=header)
        html = webPage.text
        return
    except Exception as e:
        return {"error": e}


IO intensive
def io_simulation():
    time.sleep(2)
Copy the code

The index of competition, we use time to measure. The less time you spend, the more efficient you are.

I’m going to define a decorator here that’s a simple time timer, just to make this code look a little bit cleaner. If you don’t know a lot about decorators, that’s fine, as long as you know that they are used to calculate the running time of functions.

def timer(mode):
    def wrapper(func):
        def deco(*args, **kw):
            type = kw.setdefault('type', None)
            t1=time.time()
            func(*args, **kw)
            t2=time.time()
            cost_time = t2-t1
            print("{}-{} take time: {} seconds".format(mode, type,cost_time))
        return deco
    return wrapper
Copy the code

So step one, let’s look at single-threaded

@timer("[Single thread]")
def single_thread(func, type="") :for i in range(10):
              func()

# single thread
single_thread(count, type="CPU intensive")
single_thread(io_disk, type="Disk IO intensive")
single_thread(io_request,type="Network IO intensive")
single_thread(io_simulation,type="Analog IO intensive")
Copy the code

Look at the results

[Single thread] -CPU CPU intensive time: 83.42633867263794 SEC [single thread] - Disk IO intensive time: 15.641993284225464 SEC [single thread] - Network IO intensive time: 1.1397218704223633 seconds [single thread] - Analog IO intensive time: 20.020972728729248 secondsCopy the code

Step two, let’s look at multithreaded

@timer([Multithreading])
def multi_thread(func, type=""):
    thread_list = []
    for i in range(10):
        t=Thread(target=func, args=())
        thread_list.append(t)
        t.start()
    e = len(thread_list)

    while True:
        for th in thread_list:
            if not th.is_alive():
                e -= 1
        if e <= 0:
            break

# multithreaded
multi_thread(count, type="CPU intensive")
multi_thread(io_disk, type="Disk IO intensive")
multi_thread(io_request, type="Network IO intensive")
multi_thread(io_simulation, type="Analog IO intensive")
Copy the code

Look at the results

CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU [Multithreading] - Analog IO intensive cost time: 2.0288875102996826 secondsCopy the code

Step 3. Finally, look at multiple processes

@timer("[Multi-process]")
def multi_process(func, type=""):
    process_list = []
    for x in range(10):
        p = Process(target=func, args=())
        process_list.append(p)
        p.start()
    e = process_list.__len__()

    while True:
        for pr in process_list:
            if not pr.is_alive():
                e -= 1
        if e <= 0:
            break

# multiple processes
multi_process(count, type="CPU intensive")
multi_process(io_disk, type="Disk IO intensive")
multi_process(io_request, type="Network IO intensive")
multi_process(io_simulation, type="Analog IO intensive")
Copy the code

Look at the results

Multi-process -CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU 0.13074755668640137 seconds [multi-process] - Analog IO intensive cost: 2.0076842308044434 secondsCopy the code

. Summary of performance comparison results

Put the results together and tabulate them.

Let’s analyze this table.

First is cpu-intensive, multi-threaded contrast to single thread, not only advantages, apparently also due to global lock, lock release GIL switch threads and time-consuming, low efficiency, and multiple processes, is due to multiple CPU to calculate work at the same time, the equivalent of ten men doing one’s homework, obviously efficiency is multiplied.

Then IO intensive, IO intensive can be disk I/O, network I/O, database I/O, etc., all belong to the same category, the amount of calculation is small, mainly the WASTE of I/O waiting time. Through observation, it can be found that we disk IO, network IO data, multithreading compared to single thread also did not reflect a great advantage. This is because the IO task of our program is not heavy enough, so the advantage is not obvious.

So I also added a “simulation IO intensive”, with sleep to simulate THE IO waiting time, is to reflect the advantages of multi-threading, but also to make you more intuitive understanding of the working process of multi-threading. A single thread requires each thread to sleep(2), 10 threads is 20 seconds, and multithreading, when it sleeps (2), switches to other threads, so that 10 threads sleep(2) at the same time, and eventually 10 threads only have 2 seconds.

The following conclusions can be drawn

• Single threads are always the slowest and multiple processes are always the fastest.
• Multithreading is suitable for use in IO intensive scenarios, such as crawlers, website development, etc
• Multi-process is suitable for scenarios with high CPU requirements, such as big data analysis and machine learning
• Multiple processes, while always the fastest, are not necessarily the best choice, as they require CPU resources to take advantage