Quartz principle decryption

Author: Dorae Date: July 17, 2018 15:55:02

1. Quartz overview

Quartz is an open source task scheduling framework implemented in Java that can be used to create simple and complex task scheduling. It also provides many enterprise-level features, such as JTA and clustering. It is a popular Java task scheduling framework.

1. What can it be used for

Quartz is a task scheduling framework when encountering the following problems:

• I want to repay the debt automatically on the 25th of every month.
• Want to send an anonymous greeting card to the goddess of his unrequited love on April 1 every year;
• I want to back up my data every hour.

So it boils down to doing something at a regular point in time, and it can be so complex that it needs a framework to help us. Quartz is designed to solve this problem by defining a trigger condition that triggers the corresponding job at a specific point in time.

Characteristics of 2.

• Powerful scheduling functions, such as a variety of scheduling methods, can meet all kinds of general and special needs;
• Flexible application mode, such as support task scheduling and task combination, support a variety of data storage (DB, RAM, etc.;
• Support for distributed clustering, which was further modified after being acquired by Terracotta.

Basic principles of Quartz

1. Core elements

Core elements of Quartz include Scheduler, Trigger, Job and JobDetail. Trigger, Job and JobDetail are metadata, and Scheduler is the controller that actually conducts scheduling.

• Trigger

Trigger is used to define the time rules for scheduling tasks. There are four main types of Trigger in Quartz: SimpleTrigger, CronTrigger, DataIntervalTrigger and NthIncludedTrigger.

• Job&Jodetail

Quartz divides tasks into Job and JobDetail. Job is used to define the execution logic of a task, while JobDetail is used to describe the definition of Job (such as the implementation class of the Job interface and other related static information). For Quartz, there are two main types of jobs: StateLessJob and StateFulJob

• Scheduler

The controller that actually performs the scheduling logic, Quartz provides factory classes such as DirectSchedulerFactory and StdSchedulerFactory to support the generation of scheduler-related objects.

2. Relationships between core elements

3. Main thread

In Quartz, there are two types of threads, execution threads and scheduling threads, where threads that perform tasks are usually maintained by a thread pool. Figure 1-2 shows the relationship between threads.

Figure 1-2

In Quartz, there are two main Scheduler threads: regular Scheduler threads (to perform regular scheduling) and Misfire Scheduler threads (to perform missed tasks). The Regular Thread polls the Trigger, and if there is a Trigger to be fired, it obtains an idle Thread from the task Thread pool and executes the job associated with the Trigger. Misfire Thraed scans all triggers to see if any are missing, and if so, treats them according to a certain strategy.

4. Data storage

Quartz triggers and jobs need to be stored before they can be used. There are two kinds of Quartz in the method of storage: RAMJobStore and JobStoreSupport, one was RAMJobStore to trigger and job stored in memory, and JobStoreSupport is based on JDBC will trigger and job stored in the database. RAMJobStore is very fast to access, but because all data is lost when the system is stopped, JobStoreSupport must be used in clustered applications. Table 1-1 shows the structure of the table.

Three, quartz cluster principle

Each node in a Quartz cluster is an independent Quartz application, which in turn manages other nodes. This means that you have to start and stop each node individually. In a Quartz cluster, independent Quartz nodes do not communicate with another node or management node, but are aware of another Quartz application through the same database table, as shown in Figure 1-3.

Iv. Quartz’s main process

1. Start the process

If Quartz is configured in Spring, the associated beans are loaded when the server starts. SchedulerFactoryBean implements the InitializingBean interface, so the afterPropertiesSet method is executed when the bean is initialized. This method will call the SchedulerFactory(DirectSchedulerFactory or StdSchedulerFactory, usually StdSchedulerFactory) to create the Scheduler. During the process of creating quartzScheduler, SchedulerFactory will read configuration parameters and initialize each component. The key components are as follows:

1. ThreadPool: general is the use of SimpleThreadPool SimpleThreadPool created a certain amount of WorkerThread examples to make the Job in the thread for processing. WorkerThread is an inner class defined in the SimpleThreadPool class, which is essentially a thread. There are three lists in the SimpleThreadPool: workers- holds all thread references in the pool; availWorkers- holds all idle threads; busyWorkers- holds all working threads; The thread pool configuration parameters are as follows:

   org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount=3 org.quartz.threadPool.threadPriority=5

2. JobStore: Divided into RAMJobStore stored in memory and JobStoreSupport stored in database (including JobStoreTX and JobStoreCMT two implementations, JobStoreCMT is dependent on containers for transaction management. If you want to use a cluster, use JobStoreSupport.

3. Paused =true,halted=false, paused=true, paused=true, paused=true, paused=true, paused=true Until the start method sets paused to false;

In addition, SchedulerFactoryBean implements the SmartLifeCycle interface, so after initialization, the start() method is executed, which performs the following actions:

1. Create and start the ClusterManager thread: This thread is used for cluster fault detection and handling, discussed in more detail below;
2. Create the MisfireHandler thread and start the thread: This thread is used to process the misfire task, discussed in more detail below;
3. Paused =false and the scheduler thread will start scheduling.

Figure 1-4 shows the startup process for Quartz.

2. QuartzSchedulerThread threads

QuartzSchedulerThread is the thread that actually performs the task scheduling. The main code is as follows.

while (!halted.get()) {
	int availThreadCount = qsRsrcs.getThreadPool().blockForAvailableThreads();
	triggers = qsRsrcs.getJobStore().acquireNextTriggers(now + idleWaitTime,
			Math.min(availThreadCount, qsRsrcs.getMaxBatchSize()), qsRsrcs.getBatchTimeWindow());
	long triggerTime = triggers.get(0).getNextFireTime().getTime();
	long timeUntilTrigger = triggerTime - now;
	while (timeUntilTrigger > 2) {
		now = System.currentTimeMillis();
		timeUntilTrigger = triggerTime - now;
	}
	List<TriggerFiredResult> bndle = qsRsrcs.getJobStore().triggersFired(triggers);
	for (int i = 0; i < res.size(); i++) {
		JobRunShell shell = qsRsrcs.getJobRunShellFactory().createJobRunShell(bndle);
		shell.initialize(qs);
		qsRsrcs.getThreadPool().runInThread(shell);
	}
}
Copy the code

1. Get the number of available threads in the thread pool (if none are available, it blocks until one is available).
2. AcquireNextTriggers acquireNextTriggers: Obtain the trigger lock by selecting… For Update implementation; Access (configurable) to be executed within 30 m triggers (need to ensure that the cluster nodes at the same time), if the @ ConcurrentExectionDisallowed and list exist which trigger skip, Otherwise, the updated trigger state is ACQUIRED(initially WAITING); Insert the firedTrigger table in the ACQUIRED state; (note: in the RAMJobStore, there is a timeTriggers, sorting by trigger time nextFireTime; When JobStoreSupport was removed from the database triggers was sorted by nextFireTime);
3. Wait until the first trigger executed in the retrieved trigger is within 2ms;
4. TriggersFired:
   1. Update firedTrigger status=EXECUTING;
   2. Update trigger when the next trigger is triggered;
   3. Update trigger state: Stateless trigger->WAITING, stateful trigger->BLOCKED, if nextFireTime==null ->COMPLETE;
   4. Commit connection, release lock;
5. For each trigger to execute, create a JobRunShell and put it into the thread pool:
   1. The execute: carry out the job
   2. Acquire TRIGGER_ACCESS lock
   3. If the job is stateful, update the trigger state: BLOCKED->WAITING,PAUSED_BLOCKED->BLOCKED
   4. If @ PersistJobDataAfterExecution updateJobData
   5. Delete firedTrigger
   6. Commit connection, release the lock

Figure 1-5 shows the process for scheduling threads.

Figure 1-6 shows the Trigger status change during the scheduling process.

3. MisfireHandler threads

Here are some reasons why misfired jobs can happen:

1. The system was restarted for some reason. During the time between system shutdown and restart, some tasks may be misfired;
2. Trigger is suspended for a period of time, some tasks may be misfire;
3. All threads in the thread pool are occupied, so the task cannot be triggered to execute, causing misfire.
4. When the next triggering time of stateful task arrives, the last execution is not finished. To deal with misfired jobs, Quartz defines two handling strategies for trigger:
   • MISFIRE_INSTRUCTION_FIRE_ONCE_NOW: Execute this once for the misfired job;
   • MISFIRE_INSTRUCTION_DO_NOTHING: Ignore the misfired job and wait for the next trigger; The default is MISFIRE_INSTRUCTION_SMART_POLICY, which is executed once every minute in the =MISFIRE_INSTRUCTION_FIRE_ONCE_NOW thread by default. In a transaction, a maximum of 20 recovery packets are recovered by default.

Execution process:

1. If set to true, check whether there are trigger events requiring recovery before obtaining a lock, and get misfitell a series of events.
2. Get TRIGGER_ACCESS lock;
3. HasMisfiredTriggersInState: obtain the misfired trigger, the default in a transaction can only 20 largest misfired trigger (configurable), misfired judgment basis: Status =waiting,next_fire_time < current_time-misfirethreshold(available, default 1min)
4. notifyTriggerListenersMisfired
5. UpdateAfterMisfire: Get misfire policy (default is MISFIRE_INSTRUCTION_SMART_POLICY, which in CronTrigger =MISFIRE_INSTRUCTION_FIRE_ONCE_NOW), Update nextFireTime according to policy;
6. Update nextFireTime etc to trigger table;
7. Commit connection, release lock 8. If there is more misfired, sleep short time (for cluster load balancing), otherwise sleep misfirethreshold time, then continue polling;

Figure 1-7 shows the misfireHandler thread execution process.

4. ClusterManager Cluster management threads

Initialization:

FailedInstance =failed+self+firedTrigger

Thread execution:

Each server will regularly (org. Quartz. JobStore. ClusterCheckinInterval this time) update LAST_CHECKIN_TIME SCHEDULER_STATE tables, if this field is far beyond the time of the update, The server instance is considered hung;

Note: Each server instance has a unique ID, which if set to AUTO is hostname+current_time

Specific processes executed by threads:

1. Check whether there are failedInstances that timeout.
2. Update the server instance’s LAST_CHECKIN_TIME; If there is a timeout instance:
3. Obtain the STATE_ACCESS lock;
4. Get failedInstances that are timed out;
5. Get TRIGGER_ACCESS lock;
6. clusterRecover:
   • For each failedInstances, get the firedTriggers for each instance via instanceId;
   • For each firedTrigger:
     • Update trigger state:
       • BLOCKED->WAITING
       • PAUSED_BLOCKED->PAUSED
       • ACQUIRED->WAITING
     • If firedTrigger is not in the ACQUIRED state (in execution state) and jobRequestRecovery=true: Create a SimpleTrigger and store itin the trigger table with status=waiting,MISFIRE_INSTR=MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY.
     • Delete firedTrigger

Figure 1-8 shows the clusterManager thread execution sequence.

Five, pay attention to the problem

1. Time synchronization problem

Quartz doesn’t really care if you’re running nodes on the same or different machines. When clusters are placed on different machines, they are called horizontal clusters. When nodes run on the same machine, it is called a vertical cluster. For vertical clusters, there is the problem of single points of failure. This is unacceptable for high-availability applications because once the machine crashes, all nodes are terminated. For horizontal clusters, there are time synchronization issues.

A node uses a timestamp to inform other instances of its own last check-in time. If a node’s clock is set to a future time, the running Scheduler will no longer be aware that the node is down. On the other hand, if a node’s clock is set to the past time, another node may decide that node is down and try to take over its Job. The simplest way to synchronize your computer’s clock is to use an Internet Time Server ITS.

2. Description Nodes compete for jobs

Because Quartz uses a random load balancing algorithm, jobs are executed by different instances in a random manner. Currently, there is no way to assign (pin) a Job to a specific node in a cluster, Quartz notes.

3. Obtaining the Job list from the cluster

Currently, there is no easy way to get a list of all jobs being executed in a cluster without directly accessing the database. Requesting a Scheduler instance yields only a list of jobs running on that instance. Quartz recommends that you write some JDBC code to access the database to retrieve all Job information from the corresponding tables.

Vi. References

1. www.cnblogs.com/drift-ice/p…
2. www.cnblogs.com/zhenyuyaodi…
3. Blog.csdn.net/u014427391/…
4. Blog.csdn.net/moon_yang_b…
5. My.oschina.net/songhongxu/…
6. Wangtianzhi. Cn / 2016/01/03 /…
7. www.quartz-scheduler.org/documentati…