Python multiprocess programming

Multithreading in Python is not really multithreading, and in most cases it is necessary to use multiprocessing if you want to make full use of multi-core CPU resources. Python provides a very useful multiprocess package called Multiprocessing. You only need to define a function and Python will do everything else. With this package, you can easily convert from single process to concurrent execution. Multiprocessing supports child processes, communication and sharing of data, various forms of synchronization, and provides components such as Process, Queue, Pipe, and Lock.

Example 1.2: Create a function and treat it as multiple processes

import multiprocessing import time def worker_1(interval): print "worker_1" time.sleep(interval) print "end worker_1" def worker_2(interval): print "worker_2" time.sleep(interval) print "end worker_2" def worker_3(interval): print "worker_3" time.sleep(interval) print "end worker_3" if __name__ == "__main__": p1 = multiprocessing.Process(target = worker_1, args = (2,)) p2 = multiprocessing.Process(target = worker_2, args = (3,)) p3 = multiprocessing.Process(target = worker_3, args = (4,)) p1.start() p2.start() p3.start() print("The number of CPU is:" + str(multiprocessing.cpu_count())) for p in  multiprocessing.active_children(): print("child p.name:" + p.name + "\tp.id" + str(p.pid)) print "END!!!!!!!!!!!!!!!!!"Copy the code

The results of

?

Example 1.3: Define a process as a class

import multiprocessing
import time

class ClockProcess(multiprocessing.Process):
    def __init__(self, interval):
        multiprocessing.Process.__init__(self)
        self.interval = interval

    def run(self):
        n = 5
        while n > 0:
            print("the time is {0}".format(time.ctime()))
            time.sleep(self.interval)
            n -= 1

if __name__ == '__main__':
    p = ClockProcess(3)
    p.start()      Copy the code

Note: run() is automatically called when process P calls start().

The results of

?

Example 1.4: Daemon comparison results

#1.4-1 No daemon attributes

import multiprocessing
import time

def worker(interval):
    print("work start:{0}".format(time.ctime()));
    time.sleep(interval)
    print("work end:{0}".format(time.ctime()));

if __name__ == "__main__":
    p = multiprocessing.Process(target = worker, args = (3,))
    p.start()
    print "end!"Copy the code

The results of

?

#1.4-2 plus daemon properties

import multiprocessing
import time

def worker(interval):
    print("work start:{0}".format(time.ctime()));
    time.sleep(interval)
    print("work end:{0}".format(time.ctime()));

if __name__ == "__main__":
    p = multiprocessing.Process(target = worker, args = (3,))
    p.daemon = True
    p.start()
    print "end!"Copy the code

The results of

?

Note: Factor processes set daemon properties and end when the main process ends.

#1.4-3 Setting the method to end daemon execution

import multiprocessing
import time

def worker(interval):
    print("work start:{0}".format(time.ctime()));
    time.sleep(interval)
    print("work end:{0}".format(time.ctime()));

if __name__ == "__main__":
    p = multiprocessing.Process(target = worker, args = (3,))
    p.daemon = True
    p.start()
    p.join()
    print "end!"Copy the code

The results of

?

Function explanation:

• Apply_async (func[, args[, KWDS [, callback]]]) is non-blocking, and apply(func[, args[, KWDS]]) is blocking.
• Close () closes the pool so that it does not accept new tasks.
• Terminate () terminates the work process and does not process unfinished tasks.
• The main join() process blocks, waiting for the child process to exit, and the join method is used after close or terminate.

Execution instructions: Xrange (4) creates a pool with 3 processes. Xrange (4) creates a pool with 3 processes. Xrange (4) creates 4 objects in succession. When one of the executions is complete, a process is left to process object 3, so the output “MSG: hello 3” appears after “end”. Since it is not blocking, the main function will execute its own, ignoring the execution of the process. Therefore, after running the for loop, it directly prints “mMsg: hark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~”, and the main program waits for the end of each process at pool.join ().

Example 7.2: Using a process pool (blocking)

#coding: utf-8 import multiprocessing import time def func(msg): print "msg:", msg time.sleep(3) print "end" if __name__ == "__main__": pool = multiprocessing.Pool(processes = 3) for i in xrange(4): MSG = "hello %d" %(I) pool.apply(func, (MSG,)) Print "Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~" pool.close() pool.join() # Print "sub-process (es) done." print" sub-process (es) done."Copy the code

The result of an execution

?

Example 7.3: Use process pools and focus on the results

import multiprocessing
import time

def func(msg):
    print "msg:", msg
    time.sleep(3)
    print "end"
    return "done" + msg

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=4)
    result = []
    for i in xrange(3):
        msg = "hello %d" %(i)
        result.append(pool.apply_async(func, (msg, )))
    pool.close()
    pool.join()
    for res in result:
        print ":::", res.get()
    print "Sub-process(es) done."Copy the code

Result of one execution

?

Example 7.4: Using multiple process pools

#coding: utf-8 import multiprocessing import os, time, random def Lee(): Print "\nRun task Lee-%s" %(os.getpid()) #os.getpid() start = time.time() time.sleep(random.random() * 10) End = time.time() print 'Task Lee, runs %0.2f seconds.' %(end-start) def Marlon(): print "\nRun task Marlon-%s" %(os.getpid()) start = time.time() time.sleep(random.random() * 40) end=time.time() print 'Task Marlon runs %0.2f seconds.' %(end-start) def Allen(): print "\nRun task Allen-%s" %(os.getpid()) start = time.time() time.sleep(random.random() * 30) end = time.time() print 'Task Allen runs %0.2f seconds.' %(end-start) def Frank(): print "\nRun task Frank-%s" %(os.getpid()) start = time.time() time.sleep(random.random() * 20) end = time.time() print 'Task Frank runs % 0.2 f seconds. The' % (end - start) if __name__ = = "__main__ ': function_list= [Lee, Marlon, Allen, Frank] print "parent process %s" %(os.getpid()) pool=multiprocessing.Pool(4) for func in function_list: Apply_async (func) # pool.apply_async(func) # pool.apply_async(func) # pool.apply_async(func) # pool.apply_async(func) # pool.apply_async(func) # pool.apply_async(func) # pool.apply_async 'pool.close() pool.join() # Call close() before calling join Print 'All subprocesses done 'print 'All subprocesses done.'Copy the code

Result of one execution

?