1/ Use the Pool class of the Multiprocessing module
The pool class provides a specified number of processes (determined by processes) that the user can invoke. When a new request is submitted to the pool, if the pool is not full, a new process is created to execute the request. If the pool is full, the requests are told to wait until any of the processes in the pool have finished before a new process is created to execute the requests. <1> Apply () synchronously executes the pattern function prototype: Apply (func[, args=()[, KWDS ={}]]) this function is used to pass indefinite arguments. Like apply in Python, the main process is blocked until the end of the function execution (not recommended, <2>apply_async() Apply_async (func[, args=()[, KWDS ={}[, callback=None]]]) is the same as apply, but it is non-blocking and supports callback after result return. Map (func, iterable [, chunksize = None]) the Pool in the class map method, and the built-in map function usage behavior, it can make the process block until the results back note: Although the second argument is an iterator, in practice the program must run the subprocess <4>map_async() prototype after the entire queue is ready: Map_async (func, iterable[, chunksize[, callback]]) is the same as map, but it is non-blocking <5>close() to close the process pool, <7> Join () The main process blocks and waits for the child process to exit. The join method is used after close or terminateCopy the code# -*- coding:utf-8 -*-
import os,time
from multiprocessing import Pool
def worker(arg) :
print("Child process starts execution >>> PID ={}, ppID ={}, no. {}".format(os.getpid(),os.getppid(),arg))
time.sleep(0.5)
print("Child process terminated >>> PID ={}, ppID ={}, no. {}".format(os.getpid(),os.getppid(),arg))
def main() :
print("Main process starts execution >>> PID ={}".format(os.getpid()))
process_pool = Pool( processes = 5 )
for i in range(10) :# process_pool.apply(worker,args=(I,)) # process_pool
process_pool.apply_async(worker,args=(i,)) # async execution
Close the process pool and stop accepting other processes
process_pool.close()
# block process
process_pool.join()
print("Main process terminated")
# Test it out
Apply (function,args=(I,))
The result is as follows, which shows that the synchronous execution mode blocks and does not achieve the purpose of parallelism.The main process starts >>> PID =6688The child process starts >>> PID =6689,ppid=6688That number0Child process terminated >>> PID =6689,ppid=6688That number0The child process starts >>> PID =6690,ppid=6688That number1Child process terminated >>> PID =6690,ppid=6688That number1The child process starts >>> PID =6691,ppid=6688That number2Child process terminated >>> PID =6691,ppid=6688That number2The child process starts >>> PID =6692,ppid=6688That number3Child process terminated >>> PID =6692,ppid=6688That number3The child process starts >>> PID =6693,ppid=6688That number4Child process terminated >>> PID =6693,ppid=6688That number4The child process starts >>> PID =6689,ppid=6688That number5Child process terminated >>> PID =6689,ppid=6688That number5The child process starts >>> PID =6690,ppid=6688That number6Child process terminated >>> PID =6690,ppid=6688That number6The child process starts >>> PID =6691,ppid=6688That number7Child process terminated >>> PID =6691,ppid=6688That number7The child process starts >>> PID =6692,ppid=6688That number8Child process terminated >>> PID =6692,ppid=6688That number8The child process starts >>> PID =6693,ppid=6688That number9Child process terminated >>> PID =6693,ppid=6688That number9Main process terminatedAsynchronous execution
# apply_async()
The asynchronous execution mode does not block and achieves the purpose of parallelism.The main process starts >>> PID =8449The child process starts >>> PID =8451,ppid=8449That number0The child process starts >>> PID =8452,ppid=8449That number1The child process starts >>> PID =8453,ppid=8449That number2The child process starts >>> PID =8454,ppid=8449That number3The child process starts >>> PID =8455,ppid=8449That number4Child process terminated >>> PID =8451,ppid=8449That number0The child process starts >>> PID =8451,ppid=8449That number5Child process terminated >>> PID =8454,ppid=8449That number3Child process terminated >>> PID =8452,ppid=8449That number1Child process terminated >>> PID =8453,ppid=8449That number2Child process terminated >>> PID =8455,ppid=8449That number4The child process starts >>> PID =8455,ppid=8449That number6The child process starts >>> PID =8454,ppid=8449That number7The child process starts >>> PID =8452,ppid=8449That number8The child process starts >>> PID =8453,ppid=8449That number9Child process terminated >>> PID =8451,ppid=8449That number5Child process terminated >>> PID =8454,ppid=8449That number7Child process terminated >>> PID =8455,ppid=8449That number6Child process terminated >>> PID =8453,ppid=8449That number9Child process terminated >>> PID =8452,ppid=8449That number8Main process terminated# -*- coding:utf-8 -*-
import os,time
from multiprocessing import Pool
def worker(arg) :
print("Child process starts execution >>> PID ={}, ppID ={}, no. {}".format(os.getpid(),os.getppid(),arg))
time.sleep(0.5)
print("Child process terminated >>> PID ={}, ppID ={}, no. {}".format(os.getpid(),os.getppid(),arg))
def main() :
print("Main process starts execution >>> PID ={}".format(os.getpid()))
process_pool = Pool( processes = 5 )
data = range(5)
# process_pool.map(worker,data)
process_pool.map_async(worker,data)
Close the process pool and stop accepting other processes
process_pool.close()
# block process
process_pool.join()
print("Main process terminated")
Copy the code2 / concurrent. Futures. ProcessPoolExecutor process pool
The concurrent.futures module, which provides the ProcessPoolExecutor and ThreadPoolExecutor classes, further abstracts from threading and Multiprocessing. There are two ways to submit a task: synchronous submission and asynchronous submission. <1> Submit a task synchronously. Submit a task, wait for the completion of the task, get the result of the task, and then execute the next task. Result () Advantages: Can be decouple disadvantages: Slow, because you need to wait for the result before executing the next taskCopy the code import datetime
from concurrent.futures import ProcessPoolExecutor
import time, random, os
import requests
def f(name) :
print('%s %s is running'%(name,os.getpid()))
#print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
if __name__ == '__main__':
process_pool = ProcessPoolExecutor(4) Set the number of processes in the process pool
for i in range(10) :Synchronous call mode, call and equivalent
Pass the parameter (task name, parameter) using position or keyword parameter
obj = process_pool.submit(f,"Process PID:")
The #.result() function returns the result of the process and the result of the return
# if f() does not return, obj.result() is None
res = obj.result()
Wait for tasks in the pool to finish before executing the main process
process_pool.shutdown(wait=True)
print("Main thread")
Copy the code<2> Asynchronous task submission only calls function F, which is not equivalent. Advantage: Fast speed. Disadvantage: coupling existsCopy the code import datetime
from concurrent.futures import ProcessPoolExecutor
import time, random, os
import requests
def f(name) :
print("%s %s is running" %(name,os.getpid()))
time.sleep(random.randint(1.3))
if __name__ == '__main__':
Set the process in the process pool
process_pool = ProcessPoolExecutor(4)
for i in range(10) :# Asynchronous commit, call only, not equivalent
process_pool.submit(f,'Process pid:')
# Pass parameter (task name, parameter), parameter using position parameter or keyword parameter
Wait for tasks in the pool to finish before executing the main process
process_pool.shutdown( wait=True )
print('Main thread')
Copy the code