directory

  • Introduction to Python thread pools
  • ThreadPoolExecutor () ¶
    • 1. The thread pool as_completed function is used
    • 2. Use the thread pool map function
    • 3. Use the thread pool WAIT function
  • Guess you like it

Recommended path for learning Python: Python Learning Directory >> Python Basics

Introduction to Python thread pools

ThreadPoolExecutor (ThreadPoolExecutor) Thread pool (ThreadPoolExecutor) Thread pool (ThreadPoolExecutor)

  • 1. Threadpool — An old module that is still in use, but not mainstream anymore;
  • Concurrent. futures — currently this module is used mainly by thread pools, the main module;

ThreadPoolExecutor () ¶

In addition to the Python thread pool ThreadPoolExecutor function submit/cancel/done/result, there are several additional functions that need to be explained today:

1. The thread pool as_completed function is used

Although the done function provides a way to determine whether a task is finished, it is not very useful because we do not know when the thread is finished, so we need to always determine whether each task is finished or not. At this point, you can use the AS_COMPLETED method to fetch the results of all the tasks at once.

The as_completed method is a generator that blocks when no task is completed and continues execution when a task is completedThe for loopThe following statement then continues to block, looping until all tasks are completed.

#! Usr /bin/env python # -* -coding :utf-8 _*- "" www.codersrc.com @file :Python thread pool threadpoolexecutor. py @time :2021/05/05 07:37 @Motto: A long journey without a small step, a river without a small stream, The wonderful program life needs to accumulate unremittingly! """ from concurrent.futures import ThreadPoolExecutor, Def download_video(index): time.sleep(2) print("download video {} finished at {}".format(index,time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime()))) return index executor = ThreadPoolExecutor(max_workers=2) urls = [1, 2, 3, 4, 5] all_task = [executor.submit(download_video, (url)) for url in urls] for task in as_completed(all_task): Data = task.result() print(" task {} down load success". Format (data))" Download Video 1 Finished at 2021-05-05 07:10:00 task 1 Down Load Success Download Video 2 Finished at 2021-05-05 07:10:00 Task 2 Down Load Success Download video 3 Finished at 2021-05-05 07:10:02 Task 3 Down Load success Download video 4 Finished At 2021-05-05 07:10:02 task 4 Down Load Success Download Video 5 Finished at 2021-05-05 07:10:04 task 5 Down Load Success"Copy the code

Code analysis:

5 tasks, 2 threads, since the thread pool is constructed to allow a maximum of 2 threads to be executed at the same time, so execute task 1 and task 2 at the same time. According to the output of the code, after task 1 and task 2 are executed, the for loop enters the blocking state. For does not continue task 3 / task 4 until task 1 or task 2 ends, and ensures that at most two tasks are executed simultaneously (see the Python time module for custom time formats).

2. Use the thread pool map function

Unlike the AS_completed method, the map method ensures that tasks are ordered, as an example: If 5 videos are downloaded at the same time, even if the second video is downloaded before the first one, it will be blocked until the first video is downloaded and the main thread is notified, and the second video will be notified to ensure that the tasks are completed in sequence. Here is an example:

#! Usr /bin/env python # -* -coding :utf-8 _*- "" www.codersrc.com @file :Python thread pool threadpoolexecutor. py @time :2021/05/05 07:37 @Motto: A long journey without a small step, a river without a small stream, The wonderful program life needs to accumulate unremittingly! """ from concurrent.futures import ThreadPoolExecutor, Def download_video(index): time.sleep(index) print("download video {} finished at {}".format(index,time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime()))) return index executor = ThreadPoolExecutor(max_workers=2) urls = [3, 2, 1, 4, 5] for data in executor.map(download_video,urls): Print (" task {} down load success". Format (data)) Download Video 2 Finished at 2021-05-05 07:10:55 Download Video 3 Finished at 2021-05-05 07:10:56 task 3 Down Load Success Task 2 Down Load Success Download video 1 Finished at 2021-05-05 07:10:56 Task 1 Down Load Success Download video 4 Finished At 2021-05-05 07:10:00 task 4 Down Load Success Download Video 5 Finished at 2021-05-05 07:10:01 task 5 Down Load Success"Copy the code

Code analysis:

Even if task 2 is completed before task 3, the for loop still prompts you to complete task 3 first and then task 2, in order of the urls listed to keep the tasks sequenced!

3. Use the thread pool WAIT function

** The wait method is similar to a thread’s join and blocks the main thread until all threads in the thread pool have completed their operation. ** Example code is as follows:

#! Usr /bin/env python # -* -coding :utf-8 _*- "" www.codersrc.com @file :Python thread pool threadpoolexecutor. py @time :2021/05/05 07:37 @Motto: A long journey without a small step, a river without a small stream, The wonderful program life needs to accumulate unremittingly! """ from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED, Def download_video(index): time.sleep(2) print("download video {} finished at {}".format(index,time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime()))) return index executor = ThreadPoolExecutor(max_workers=2) urls = [1, 2, 3, 4, 5] all_task = [executor.submit(download_video,(url)) for url in urls] wait(all_task,return_when=ALL_COMPLETED) Print ("main ")  download video 2 finished at 2021-05-05 07:10:22 download video 1 finished at 2021-05-05 07:10:22 download video 3 finished at 2021-05-05 07:10:24 download video 4 finished at 2021-05-05 07:10:24 download video 5 finished at 2021-05-05  07:10:26 main '''Copy the code

** The wait method takes three parameters, the waiting task sequence, the timeout, and the wait condition. Wait condition return_WHEN defaults to ALL_COMPLETED, indicating that all tasks are to be completed. As you can see, the main line prints main after all tasks are completed. The wait condition can also be set to FIRST_COMPLETED to stop waiting when the first task is completed. 支那

3.Guess you like

  1. Python conditional derivations
  2. Python list derivations
  3. Python dictionary derivations
  4. Python function declarations and calls
  5. Python variable argument *argc/**kargcs
  6. Python anonymous function lambda
  7. Python return logic determines expressions
  8. Python string/list/tuple/dictionary conversions
  9. Python local and global variables
  10. The Python type function is different from the isinstance function
  11. Python is differs from ==
  12. Python mutable and immutable data types
  13. Shallow and deep copies of Python
  14. Read and write Python files
  15. Python exception Handling
  16. Python module import
  17. Python __name__ == ‘__main__’ explained in detail
  18. Python thread creation and parameter passing
  19. Python thread mutex Lock
  20. Python thread time Event
  21. The Python thread Condition variable Condition
  22. Python thread Timer Timer
  23. Python thread Semaphore
  24. Python thread Barrier object Barrier
  25. Python thread Queue Queue – FIFO
  26. Python thread queue LifoQueue – LIFO
  27. Python thread PriorityQueue PriorityQueue

Python thread pool ThreadPoolExecutor

This article is published by the blog – Ape Say Programming Ape Say programming!