preface

In the previous analysis of the principle of XXL-Job actuator, we mentioned that xxL-Job framework contains two core modules: the scheduling center and the actuator. The scheduling center is mainly responsible for task scheduling, while the actuator is responsible for task execution. Then we analyze the internal structure of the actuator by drawing a picture, and sort out the execution process of Job.

In this paper, we continue to analyze the scheduling center around the task scheduling process, the content is still referring to xx-Job V2.x version of the source code.

The body of the

To look againxxl-jobArchitecture diagram:The scheduling center provides two functions: system management and task scheduling. The rest are ancillary functions.

  • System management as shown in the figure, including task management, executive management, log management. An administrative interface is also provided.
  • Task scheduling is responsible for pulling tasks from the data center and delivering tasks to the executor according to the execution time.

Composition structure of the scheduler

Two core threads

When the scheduling center is started, the following two threads are started:

  1. schedulerThread

The scheudlerThread does two things:

  • From the data center (DB), which isxxl_job_infoA match was scanned in the tableCondition 1The task,Condition 1The restrictions are as follows:
    • Task execution time less than (Current time + 5 seconds)
    • Limit the number of scans. This value is dynamic, depending on the number of threads in the fast or slow thread pool mentioned later.
    count = treadpool-size * trigger-qps  (each trigger cost 50ms, qps = 1000/50 = 20) 

treadpool-size = (getTriggerPoolFastMax() + getTriggerPoolSlowMax()) * 20
// After the introduction of the fast and slow thread pool, it will be easier to come back here
Copy the code
  • The scanned tasks are divided into the following three categories:
  1. ringThread

RingThread is used to continuously read tasks from the container that need to be executed at the current point in time, and these tasks are handed over to something called a fast or slow thread pool that passes the tasks to the scheduler for execution.

Time round

Together, the ringthreads and containers described above form a time wheel.

As for the time wheel, if you expand it, there will be a lot of content. If you need to know more about it, you can refer to this article or search by yourself. This article will only provide a brief understanding.

To put it simply, the time wheel implements the function of delay execution. Its role in XXl-job is to send the tasks that have not reached the execution time to the executor one by one through the fast and slow thread pool according to the expected time.

  • The data structure of a time wheel is usually an array + linked list, similar to jdk1.7’s HashMap, where each node in a linked list is a task to be executed.

  • The time wheel in XXL-job can be graphically described as a clock with only a second hand in the figure below.

  • During the running of ringThread, the thread sweeps one scale every second. Assuming that the current scale exists in the job list, it takes out all jobs in the linked list and finally throws them to the fast and slow thread pool.

  • Of course, xxL-job crosses the scale and checks one more scale forward to avoid long processing time. That is, when the pointer points to 2s, it reads both the 1s and 2s tasks.

Fast and slow thread pools

As mentioned above, when a task is scanned from the data center, it is then dropped into the fast and slow thread pool, which is defined as follows:

      fastTriggerPool = new ThreadPoolExecutor(
                10,
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax(),
                60L,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(1000),
                new ThreadFactory() {
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-"+ r.hashCode()); }}); slowTriggerPool =new ThreadPoolExecutor(
                10,
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax(),
                60L,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(2000),
                new ThreadFactory() {
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-slowTriggerPool-"+ r.hashCode()); }});Copy the code

As can be seen from the above, the fast and slow thread pool contains two thread pools, fast and slow. When a job is submitted to the fast and slow thread pool, the fast and slow thread pool will select one of the thread pools to perform subsequent operations based on some conditions.

Fast and slow thread pools are used as follows:

Implement thread pool isolation: Scheduling thread pools are separated and split, and Slow tasks are automatically demoted to the "Slow" thread pool to avoid running out of scheduling threads and improve system stability.

What is a slow task?

A task is classified as slow if it executes more than 10 timeouts in a minute

When a specific fast or slow thread pool receives a scheduled task, an RPC remote call triggers the executor to complete the execution logic of the task.

When the actuator receives a scheduled task, see xxL-job actuator Principle Analysis for details on how to execute the task.

The source entry

com.xxl.job.admin.core.thread.JobScheduleHelper#start

com.xxl.job.admin.core.thread.JobTriggerPoolHelper#addTrigger

com.xxl.job.core.biz.client.ExecutorBizClient#run
Copy the code

conclusion

Based on xxL-job v2.x source code, this paper analyzes the structure of XXL-job scheduler and how the dispatching center triggers tasks.

The scheduler mainly contains the following modules:

  • schedulerThread: Is responsible for scanning tasks to be performed from the data center
  • ringThread: Responsible for precise control of tasks expected to be performed
  • Fast and slow thread pools: Schedules fast and slow tasks separately by wrapping two thread pools.

The main difficulty here is to introduce the concept of a time wheel, which is not described in detail in this paper.

In addition, the use of time wheel is used in many places, such as Netty, Dubbo, Kafka, etc.