Common pitfalls in Java thread pool configuration

Official account: Java Xiaokaxiu, website: Javaxks.com

Author: Zhenbianshu, link: Zhenbianshu.github. IO /

Thread pooling is a very important part of the Java language. Doug Lea’s thread pooling package is very convenient for us to use. However, it is also possible to misunderstand the configuration parameters of the thread pool because you do not know the implementation.

preface

We often see in technical books and blogs that when submitting a task to a thread pool, the thread pool’s execution logic is as follows:

When a task is submitted, the thread pool first checks whether the number of running threads reaches the core number and creates a thread if not.
If the number of threads running in the thread pool reaches the core thread count, the task will be placed in BlockingQueue.
If BlockingQueue is full, the thread pool will attempt to expand the number of threads to the maximum thread pool capacity.
If the number of threads in the current thread pool reaches the maximum thread pool capacity, a reject policy is executed to reject the task submission.

The process is shown below (from Meituan technology blog) :

The process description is fine, but it can lead to misunderstandings if some points are not well thought out, and the scenarios in the description are too idealistic, and some very weird things can happen if you configure without considering the runtime environment.

Reprint at will, the article will continue to revise, please note the source address: Zhenbianshu.github. IO.

Core pool

The core pool is the permanent part of the thread pool. The internal threads are not destroyed, and the majority of our submitted tasks should be executed by the threads in the core pool.

Misunderstanding of thread creation timing

One of the most common misconceptions about core pooling is not knowing when threads are created in the core pool. I don’t think it’s too much of a mistake to throw 10% of the blame at Doug Lea, Because he wrote in the document “If fewer than corePoolSize Threads are running, Try to start a new thread with the given command as its first task. In our understanding, running means that the current thread has been scheduled by the operating system, has an operating system time slice, or is understood to be performing a task.

Based on the above understanding, it is easy to assume that if the QPS of the task is very low, the number of threads in the thread pool will never reach coreSize. That is, if we set coreSize to 1000, but the QPS is only 1 and a single task takes 1s, then the core pool size will always be 1, even if there is traffic jitter, the core pool will only be expanded to 3. Because a thread executes one task per second, it’s just enough to handle 1QPS without creating a new thread.

The creation process

But if you simply design a test using JStack to print out the thread stack and count the number of threads in the thread pool, you will see that the number of threads in the thread pool increases as the task is submitted until it reaches coreSize.

Since the core pool is designed to serve as a resident pool for daily traffic, it should be initialized as soon as possible. Therefore, the thread pool logic is that each task will create a new thread before reaching coreSize.

public void execute(Runnable command) { ... int c = ctl.get(); If (workerCountOf(c) < corePoolSize) {if (addWorker(command, true)) return; c = ctl.get(); }... }Copy the code

The running state in the documentation also means that the thread has been created, and we know that the thread will try to get the task from BlockingQueue in a while loop after it is created, so it’s running.

With this in mind, we are warming up some of the high concurrency services not to expect jIt-optimizations for hot code, but to warm up thread pools, connection pools, and local caches.

BlockingQueue

BlockingQueue is another important component within a thread pool. It serves as an intermediary for the producer-consumer model of the thread pool, and it can buffer large bursts of traffic, but it is also prone to errors in understanding and configuring it.

Operating model

The most common mistake is not understanding the running model of thread pools. The first thing to be clear about is that the thread pool does not have an accurate scheduling capability, that is, it cannot sense which threads are idle and dispatch the submitted tasks to the idle threads. Thread pools are producer-consumer. All tasks go to BlockingQueue, except for the tasks that trigger thread creation (the thread’s firstTask), and wait for threads in the pool to consume them. Which thread consumes the task depends entirely on the scheduling of the operating system.

The corresponding producer source code is as follows:

public void execute(Runnable command) { ... If (isRunning(c) &&workqueue.offer (command)) {isRunning() = ctl.get(); if (! isRunning(recheck) && remove(command)) reject(command); else if (workerCountOf(recheck) == 0) addWorker(null, false); }... }Copy the code

The corresponding consumer source code is as follows:

private Runnable getTask() { for (;;) {... Runnable r = timed ? workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS) : workQueue.take(); if (r ! = null) return r; . }}Copy the code

Buffering by BlockingQueue

Based on the producer-consumer model, we might assume that if enough consumers are configured, the thread pool won’t have any problems. No, we also have to consider the amount of concurrency.

Consider the following situation: There are 1000 tasks to be submitted to the thread pool for concurrent execution. In the case that the thread pool is initialized, all of them are placed in the BlockingQueue waiting to be consumed. In the extreme case, the consuming thread does not complete any tasks. All 1000 requests need to exist in BlockingQueue. If the Size of the BlockingQueue is smaller than 1000, additional requests will be rejected.

So what’s the probability of this limiting case happening? The answer is very large, because the operating system has a very high priority for SCHEDULING I/O threads. Normally, our tasks are started by the preparation or completion of I/O (such as Tomcat accepting HTTP requests), so it is likely that tomcat threads will be scheduled to submit requests to the thread pool. The consumer thread could not be scheduled, causing requests to pile up.

The service I was in charge of had this kind of abnormal rejection of requests. During the pressure test, the average response time of QPS 2000 was 20ms. Under normal circumstances, 40 threads could balance the production speed without piling up. However, at BlockingQueue Size 50, requests can still be rejected by the thread pool, even if the thread pool coreSize is 1000.

In this case, the significance of BlockingQueue is that it is a container that can store tasks for a long time, providing buffer for the thread pool at little cost. The maximum number of tasks that can be submitted at the same time is called the number of concurrent tasks. When configuring a thread pool, it is important to know the number of concurrent tasks.

Calculation of concurrency

QPS is often used to measure service stress, so it is often used when configuring thread pool parameters. However, sometimes the correlation between QPS and concurrency is not that high, and QPS also calculates peak concurrency with task execution time.

For example, what is the peak concurrency of an interface with strictly identical request intervals, with an average QPS of 1000? There is no way to estimate, because if the task takes 1ms, it has only 1 concurrency; If the task execution time is 1s, the concurrency peak is 1000.

But if we know the execution time of the task, can we calculate the concurrency? No, because if the interval of the request is different, the request within 1min May be sent in 1 second, then the concurrency has to be multiplied by 60, so the above says that we know the QPS and task execution time, the concurrency can only be calculated.

My general rule of thumb for concurrency is QPS* average response time, with double the redundancy, but if the business is important, it’s ok to set BlockingQueue Size to a larger Size (1000 or more) because each task takes up a limited amount of memory.

Consider runtime

GC

In addition to the various situations mentioned above, GC is also an important influence factor.

We all know that GC stops the World, but the World here refers to the JVM, and the preparation and completion of a request I/O is done by the operating system. The JVM stops, but the operating system still accepts the request and executes it after the JVM resumes. So GC stacks requests.

The concurrency calculation mentioned above must take into account the simultaneous processing of the accumulated requests within the GC time. The number of accumulated requests can be calculated simply by QPS*GC time, and it must be remembered to leave redundancy.

Business peak

In addition, it is important to consider business scenarios when configuring thread pool parameters.

If the bulk of the interface traffic is coming from a timed application, then the average QPS is meaningless and the thread pool design should consider setting the Size of BlockingQueue to a larger value. However, if the traffic is very uneven, only a certain period of time in a day with high traffic, and thread resources are tight, consider leaving a large redundancy for the maxSize of the thread pool. You can also set up a larger BlockingQueue to allow tasks to pile up to some extent when traffic spikes are obvious and response times are less sensitive.

Of course, in addition to experience and calculation, regular pressure measurement of the service can undoubtedly help grasp the real situation of the service.

summary

When summarizing the thread pool configuration, my biggest feeling is to read the source code! Read the source code! Read the source code! You won’t get the most out of important concepts just by reading the summaries of a few books and articles, and even if you do get most of them, you’ll easily stumble in a few corners. After a deep understanding of the principles, in the face of complex situations, the ability to flexibly configure.

There are many points to discuss about thread pools, and this article should continue to be revised. Stay tuned.

If you have any questions about this article, please leave a comment below. If you find this article helpful, please follow me on Weibo or GitHub. You can also subscribe to my blog by clicking On Watch in the upper right corner of my blog REPO and selecting Releases Only to notify you as soon as new posts are released.

References:

Implementation principle of Java thread pool and its practice in Meituan business