First, pre-knowledge

1. Queue basics

  • If not specifiedThe queueWhat is it, please check for yourself
  • Queues are the most common method of communication between threads in Python because they are thread-safe
From queue import queue # create queue # limit the number of elements in the queue to maxsize Q = Queue(100) q1 = Queue() # q.puff (1) q.puff (True) q.puff (' ABC ') # print(q.qsize()) # Check if Queue is full Print (q.void ()) print(q.void ()) # print(q.void ()) # print(q.void ()) # print(q.void ()) # print(q.void ()) print(q.get())Copy the code

2. Multithreading

(1) The relationship between process and thread

A process can have multiple threads, but must have one main thread. Processes do not affect each other, and resources are not shared. 3. Resources can be shared between threads, but the content of the process to which the thread belongs is shared. Threads must depend on the process to exist.Copy the code

(2) Create a thread — method 1

The from threading import Thread def add (n1, and n2) : print (' results as follows: '+ n1 + n2) def main () : # create a thread # -- the name of the target function # -- args as a tuple T = Thread(target=add, args=(1, 2,)) # start Thread.Copy the code

(3) Create a thread — Method 2

1. Create a Thread by inheriting Thread steps (1) define a class (2) inherit Thread (3) override the run() method (4) write logic code in the run() method (2). (1) When a subclass inherits Thread, it automatically executes the run() method in its parent class when instantiating the object, so we can override run() and execute our own code in run(). (2) A subclass inherits Thread, Then it must make sure that the base class's constructor is called before it can do anything else to the thread -- i.e., the parent class's constructor needs to be called when passing arguments.Copy the code
Def __init__(self, n1, n2) from threading import Thread class MyThread(Thread): # threading.thread.__init__ (self) # call the constructor of the parent class: Self = n1 self.n2 = n2 # def run(self): Print (self.n1 + self.n2) def main(): T1 = MyThread(1, 1) # set thread name t1.setName(' T1 ') # start thread t1.start() if __name__ == '__main__': main()Copy the code

(4) The use of locks

  • It is important to ensure that all related threads use the same lock, otherwise locking is meaningless
# # before locking -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the from threading import Thread num = 0 # def statement to a Shared resource For I in range(10000000): num+=1 print(num) def main(): T1 = Thread(target=Jia) T2 = Thread(target=Jia) t3 = Thread(target=Jia) t1.start() t2.start() t3.start() t3.start() t3.start() t3.start() if __name__ == '__main__': main()Copy the code
Lock after # # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the from threading import Thread from threading import Lock Lock = Lock() num = 0 def Jia(Lock) def Jia(Lock) Global num # for I in range(10000000): Num +=1 print(num) # release() def main(): T1 = Thread(target=Jia, args=(lock,)) T2 = Thread(target=Jia, args=(lock,)) t3 = Thread(target=Jia, args=(lock,)) Args = (lock), t1) # open threads. The start (t2), start (t3), start () if __name__ = = "__main__ ': the main ()Copy the code

3. Advanced

(1) Thread.join()

  • Action: Blocks the current thread until the join() thread terminates

  • Analyze the following code:

    • Before blocking: there is a sentence in the main threadPrint (' Done '), intended to be in thefnAfter the function is executed, it is printedThe end of theBut since the master thread and t1 thread are synchronized, they are executing at the same time, soPrint (' Done ')The output position of is not necessarily the last, but may be atfnPrint it halfway through executionThe end of the
    • After the block:T1 threadCall thejoin(), block the current thread, that is, block the main thread, so the main thread needs to waitT1 threadYou can continue executing the main thread only after the completion, so it is implementedPrint (' Done ')infnOutput content requirements after execution
# before blocking: Also is not call the join () # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the import time from when the import Thread def Fn (): for I in range(10): print(I) time.sleep(1.5) def main(): T1 = Thread (target = fn) t1. The start () print (' over ') if __name__ = = "__main__ ': the main ()Copy the code
After blocking: Call the join () # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the import time from when the import Thread def Fn (): for I in range(10): print(I) time.sleep(1.5) def main(): T1 = Thread (target = fn) t1. Start (t1). The join () print (' over ') if __name__ = = "__main__ ': the main ()Copy the code

(2) Daemon process

Processes are classified as master, daemon, and non-daemon processes. 2. Daemon and non-daemon are relative to the main process. The daemon process is an unimportant process. When the main process ends, the daemon forcibly terminates. 4. Non-daemons. A process that is more important than the daemons. The daemons are not forced to terminate when the main process terminates.Copy the code
# t1 process is not a daemon: T1 process into dead circulation # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the from threading import Thread def fn () : While True: print(1) def main(): t1 = Thread(target=fn) t1.start() print(' end ') if __name__ == '__main__': main()Copy the code
# t1 is a daemon: T1 process will be for the process of the end, be forced to end # -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- the from threading import Thread def fn () : while True: print(1) def main(): T1 = Thread(target=fn) t1.start() t1.setdaemon (True) Illustrate this process is the "daemon" (the default is False 】 print (' over ') if __name__ = = "__main__ ': the main ()Copy the code

(3) queue communication between threads

  • Be sure to think about it in future code, especially in phase 5
# queue.join () "blocks the producer thread when the producer is finished, and unblocks the producer thread only when the consumer sends a message that the Queue has finished." Send a message to the producer when the queue has been consumed, also send a message to the producer indicating that the queue has been consumed, indicating that the producer thread can be unblocked.Copy the code

Producer and consumer model

  • The pattern has two objects, producer and consumer, which operate simultaneously
  • The following is divided into five stages and explained slowly

Phase 1: Blocking of the consumer thread

Def produce(q): for I in range(1, 11): def produce(q): for I in range(1, 11): Def consumer(q): while True: TMP = q.get() print(f' get -- {I}') # def main(): q = Queue() pro = Thread(target=produce, args=(q,)) con = Thread(target=consumer, args=(q,)) pro.start() con.start() if __name__ == '__main__': main()Copy the code

  • Analysis of the

    • The producer thread and consumer thread are created and started in the main thread. The producer produces 10 products in total
    • As producers produce products, consumers invoke themq.get()Method consumes the product. When the producer has finished producing the product, the producer thread terminates and the consumer continues the callq.get()Method consumes the product, which the consumer invokes when there is no product to consumeq.get()“, causes the consumer thread to block until more data is added, but the producer has finished producing the product and will not produce any more, so the consumer thread willIt's blocked all the time

Stage 2: Endless consumption of products

Def produce(q): for I in range(1, 11): def produce(q): for I in range(1, 11): Def consumer(q): while True: TMP = q.get() print(f' get -- {I}') # def main(): q = Queue() pro = Thread(target=produce, args=(q,)) con = Thread(target=consumer, Args = (q)) con. SetDaemon # Settings daemon thread (True) pro. The start () con. The start () if __name__ = = "__main__ ': the main ()Copy the code

  • For the code in phase 1, just add one line of code that calls the consumer thread “daemon thread.

  • Analysis of the

    • When the producer has finished producing, the producer thread ends, and then the main thread ends, and thenConsumer threadBeing forced to exit as a daemon thread solves the problem of consumer thread blocking
    • However, as can be seen from the following figure, although the problem of consumer thread blocking was solved, the consumer only consumed 5 products this time, and the products produced by the producer were not consumed completely. Please take a look at this problemPhase 3

Stage 3: Small, perfect code

Def produce(q): for I in range(1, 11): def produce(q): for I in range(1, 11): Def consumer(q) def consumer(q): while True: def consumer(q) def consumer(q) def consumer(q): while True: TMP = q.get() print(f' consumer product -- {TMP}') q.task_done() q = Queue() pro = Thread(target=produce, args=(q,)) con = Thread(target=consumer, args=(q,)) con.setDaemon(True) pro.start() con.start() if __name__ == '__main__': main()Copy the code

  • Only two lines of code have been added for phase 2

    • q.join()
    • q.task_done()

  • Analysis:

    • When the producer has finished producing the product, the producer thread is executedq.join()Blocked, only receive the consumer sent to have consumed the last product, unblocked
    • The consumer thread executes as it consumes the productq.task_done()A message is sent to the producer thread until the full part of the product is consumed. When a message is sent to the producer, the producer is notified that the full part of the product has been consumed
    • The producer receives the message to consume the entire product, the block is released, and the producer thread terminates
    • Then the main thread ends
    • Then, the daemon thread of the consumer thread is forced to close

Phase 4: Questions about thread execution order

Def produce(q): for I in range(1, 11): def produce(q): for I in range(1, 11): Def consumer(q) def consumer(q): while True: def consumer(q) def consumer(q) def consumer(q): while True: TMP = q.get() print(f' consumer product -- {TMP}') q.task_done() q = Queue() pro = Thread(target=produce, args=(q,)) con = Thread(target=consumer, Args = (q)) con. SetDaemon (True) pro. Start (con). The start () print (' over ') if __name__ = = "__main__ ': the main ()Copy the code

  • Compared to phase 3, only one output statement is added to the main thread

  • Analysis of the

    • What we want is to print the output after the two child threads have finishedProducers and consumers are all over!!But obviously, this is not the case. Let’s start with the analysis
    • The program has 1 main thread and 2 child threads, all of which are executed at the same time. Therefore, the execution time of the output statement in the main thread is random, so the output position is also random
    • Solution: Block the current thread, that is, the main thread, as shown in phase 5

Phase 5: Resolution of the thread execution order problem

Def produce(q): for I in range(1, 11): def produce(q): for I in range(1, 11): Def consumer(q) def consumer(q): while True: def consumer(q) def consumer(q) def consumer(q): while True: TMP = q.get() print(f' consumer product -- {TMP}') q.task_done() q = Queue() pro = Thread(target=produce, args=(q,)) con = Thread(target=consumer, Args =(q,)) con.setdaemon (True) pro.start() con.start() pro.join() # block the current thread print(' end ') if __name__ == '__main__': main()Copy the code

  • Compared to phase 4, only one line of code is added to meet the need to block the main thread

  • Analysis:

    • The program has 1 main thread, 2 child threads, all three will execute simultaneously
    • To the main threadpro.join(), the current thread is blocked, that is, the main thread is blocked, until all products are produced, all products are consumed, and the producer thread terminates
    • The main thread is unblocked
    • The main thread then terminates and the consumer thread is forced to terminate

Three, reference

  • Take an in-depth look at the specific role of task_done() in the production consumer model
  • The Java implementation of join lets the main thread wait for all child threads to finish executing before continuing
  • Why does join block the main thread?