Hello~ readers happy New Year! Here little black brother downstairs to everyone to pay a New Year, I wish everyone thriving hot hot, year after year more tun tun tun.

That year the Bug

Spring Festival holiday, little black brother sitting on the high-speed train home, suddenly thought of a production problem. It was the first year that the little black elder brother took part in the work. During the National Day holiday of that year, the little black elder brother asked for leave one day ahead of schedule to go home and do a passport. At that time, I just started to take charge of a production system, so I was a little worried about asking for leave during the working day. I was afraid that the problem would be solved by the absence of little black brother.

Alas, the more real fear of what, what.

Halfway through the high-speed train, the colleague feedback system could not get the latest flow information (flow information was pulled periodically by Spring timing task). The little black brother was surprised, immediately pulled out the computer, connected to the VPN, ready to board the production machine, check the system. However, we all know that the high-speed railway network is very unstable, and it has been a long time since I could not connect to the VPN, so I had to remotely command my colleagues to look at the system log. Through the log feedback from colleagues, it was found that the scheduled task of drawing water was not executed. Further check, the black brother found that other scheduled tasks of the whole system had also stopped…

This is a strange question. How can a perfectly good scheduled task suddenly stop?

Unable to find a solution, we had to restart the application in advance. After the restart, the problem is solved temporarily, the scheduled task starts to execute again, and the latest payment flow information is obtained.

Troubleshoot problems

When he got home, he immediately boarded the production machine, checked the system logs, and found that a certain time before the restart of the task was half finished, and after that time no other scheduled tasks were executed.

Through the system logs, the problematic code was located.

The retry compensation strategy is adopted to prevent the occasional failure of querying flow information due to network problems. This strategy is fine in the face of occasional failures, but if the bank line lookup service continues to fail, the code gets stuck in an endless loop. It happened that there were some problems in the network during that time, which led to the failure of the query here.

Increase the maximum retry times and fix the Bug.

Immediately after the fix, the latest version of the code was deployed to the production system, temporarily fixing the problem.

In the face of some failures, you can use a compensation strategy of retry, re-execution, most likely to ensure success, but remember to set the appropriate significant number of times.

Further screening

Although the problem was solved, there was still a doubt in the mind of black brother. Why the blocking of one scheduled task would affect the execution of other scheduled tasks? At first, I understood that different scheduled tasks should be isolated from each other and not affect each other.

Thought of here, small black elder brother decided to write a Demo, reproduce the problem, and then in-depth source investigation.

Start the program, the log output is as follows:

According to the logs, the fixDelayMethod method falls into sleep after execution. The cronMethod scheduled task cannot be executed until the sleep ends. As shown above, the two scheduled tasks are executed by the pool-1-thread-1 thread. From this you can see that Spring scheduled tasks will be handed over to the thread pool for execution.

The default naming policy of a thread pool is pool-%poolNumber-thread-%num.

If there is only one worker thread in the thread pool, once that thread is blocked for a long time, there is no chance for other tasks piled up to be executed.

Is this problem causing the Sping scheduled task to stop? Let’s keep working our way down.

Log the green part on the drawing ScheduledAnnotationBeanPostProcessor output is an important information:

No TaskScheduler/ScheduledExecutorService bean found for scheduled processing
Copy the code

Looking at the Spring documentation, Spring internally calls TaskScheduler to perform scheduled tasks, and another ScheduledExecutorService provides the JDK with an executor to perform scheduled tasks. Remember those two

Through this period of logs, using the IDEA of powerful keyword search functions, positioning to ScheduledAnnotationBeanPostProcessor# finishRegistration method.

This is a long method, so let’s focus on some of the points marked in the diagram.

Spring will start to scan all the Bean methods with the @Scheduled annotation and then package them as Task subclasses to ScheduledTaskRegistrar.

This code is located in ScheduledAnnotationBeanPostProcessor# processScheduled, interested can browse view

If the ScheduledTaskRegistrar does not exist with the timing task or the TaskScheduler in ScheduledTaskRegistrar does not exist, FinishRegistration will call ScheduledAnnotationBeanPostProcessor# resolveSchedulerBean method is used to search TaskScheduler/ScheduledExecutorService.

The obtained Bean will then be injected into the ScheduledTaskRegistrar via setScheduler.

If the ScheduledExecutorService type is obtained, it will be wrapped into the taskScheduler.

The last is not found, will be the first to see the output of the log. Spirng will then create a single-threaded scheduled task executor ScheduledExecutorService on ScheduledTaskRegistrar#afterPropertiesSet. It is injected into ConcurrentTaskScheduler and then executes scheduled tasks through the taskScheduler.

Scheduled tasks assigned to TaskScheduler actually end up being executed through ScheduledExecutorService.

One conclusion can be drawn here:

Spring scheduled tasks are actually performed through the ScheduledExecutorService provided by the JDK. By default, Spring generates a single-threaded ScheduledExecutorService to perform scheduled tasks. Therefore, once a scheduled task blocks the execution thread for a long time, other scheduled tasks are affected and have no chance to be executed by the execution thread.

Spring’s default configuration can be a pit in cases where multiple scheduled tasks need to be performed. We can change the configuration to make Spring multithreaded for scheduled tasks.

Custom Configuration

Spring can change the default configuration in a number of ways.

The XML configuration

Configure the number of TaskScheduler threads using XML.

<task:annotation-driven  scheduler="myScheduler"/>
<task:scheduler id="myScheduler" pool-size="10"/>
Copy the code

Using the TaskScheduler subclass ThreadPoolTaskScheduler, Spring uses the pool-size number of internal threads. This number of threads directly sets the number of ScheduledExecutorService threads.

Annotation configuration

In screening on the above issue, we know that Spring will find TaskScheduler/ScheduledExecutorService, if there will be used. So here we can generate beans for these classes.

Choose one of the above two ways

SpringBoot configuration

The above two configurations apply to normal Spring and are cumbersome. In contrast, SpringBoot configuration will be very simple, just add the following configuration to the boot configuration file.

spring.task.scheduling.pool.size=10
spring.task.scheduling.thread-name-prefix=task-test
Copy the code

Technical summary

Let’s start with the technical summary:

  1. SpringThe scheduled task execution principle is actually usedJDKbuilt-inScheduledExecutorService
  2. SpringBy default, a single thread will be usedScheduledExecutorService
  3. A single thread executes scheduled tasks. If a scheduled task takes a long time to execute, other scheduled tasks are affected
  4. If multiple scheduled tasks exist, you can modify the default configuration to use multiple threads to execute scheduled tasks to ensure the accuracy of the scheduled task execution time
  5. In the face of occasional failures, we can use a retry compensation strategy, but remember to set an appropriate maximum number of retries

Just have a chat

For common open source frameworks, we should not only know how to use them, but also be familiar with the relevant configuration, and finally, we should understand the internal use of the principle. When something goes wrong, we can quickly locate the problem and find the actual cause of the problem.

Help document

Spring scheduling-task-scheduler

Welcome to pay attention to my public account: procedures to get daily dry goods push. If you are interested in my topics, you can also follow my blog: studyidea.cn