The design principle of Quartz for scheduling system

Scheduling system is an important part of distributed system technology, so it is essential to understand its technical principles. Scheduling systems used in different systems have different names, but their general functions are similar. As a classic open source enterprise scheduling system, how can we not study it?

Why learn quartz source code?

• Scheduling systems are important and common, and Quartz is well known and widely used in the enterprise
• Learning a good system design can improve their system design ability, the subsequent task scheduling related functions, do easier and more stable

concept

• Job represents a task instance. The Job is generated by instance information configured with Jobdetail.
• JobDetail represents a task configuration detail.
• Trigger represents the configuration of scheduling parameters, when to initiate the call, and scheduling of time policies.
• Scheduler: Scheduling container. A Scheduler can register multiple Jobdetails and triggers. Only when JobDetail and Trigger are combined can they be scheduled by Scheduler.
• JobStore: A place where jobdetails and triggers are stored and read. They can be stored in memory or in a database.

Demo

To get a feel for how Quartz is used, take a look at some code:

1 If MVN depends, mysql and HikariCP are used for persistent task configuration.

<! -- https://mvnrepository.com/artifact/org.quartz-scheduler/quartz --> <dependency> < the groupId > org. Quartz - the scheduler < / groupId > < artifactId > quartz < / artifactId > < version > 2.3.0 < / version > < / dependency > < the dependency > < groupId > org. Quartz - the scheduler < / groupId > < artifactId > quartz - Steve < / artifactId > < version > 2.2.1 < / version > </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> The < version > 5.1.35 < / version > < / dependency > <! -- https://mvnrepository.com/artifact/com.zaxxer/HikariCP --> <dependency> <groupId>com.zaxxer</groupId> < artifactId > HikariCP < / artifactId > < version > 2.2.5 < / version > < / dependency >Copy the code

2 Prepare Demo code

Create a simple Job interface class
public class HelloJob implements Job {
    @Override
    public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
        System.out.println("hello quartz!"); }}// 1. Create a Scheduler using a factory
JobDetail specifies Job as HelloJob
// 3. Trigger Execution policy Executes a scheduling job repeatedly for every 10s
public class SchedulerTest {
    private static SchedulerFactory factory = new StdSchedulerFactory();

    public static void main(String[] args) throws SchedulerException {
        Scheduler scheduler = factory.getScheduler();
        scheduler.start();

        // JobDetail
        JobDetail job = JobBuilder.newJob(HelloJob.class)
            .withIdentity("myJob"."group")
            .build();

        // Trigger
        Trigger trigger = TriggerBuilder.newTrigger()
            .withIdentity("myTrigger"."group")
            .startNow()
            .withSchedule(simpleSchedule()
                .withIntervalInSeconds(10)
                .repeatForever())
            .build();

        / / schedulingscheduler.scheduleJob(job,trigger); }}Copy the code

3 By default, JobDetail and Trigger are stored in memory. If you want to persist them in the database, you can add Quartz. Properties and modify the configuration to prepare the database script.

• Database scripts: data table script: raw.githubusercontent.com/quartznet/q…
• Quartz configuration:

# quartz database table prefix org. Quartz. JobStore. TablePrefix = # QRTZ_ persistent classes used, Commit and rollback JobStoreTX support things org. Quartz. JobStore. Class = org. Quartz. Impl. Jdbcjobstore. JobStoreTX Org. Quartz. JobStore. DriverDelegateClass = org. Quartz. The impl. Jdbcjobstore. # StdJDBCDelegate data source tag, # Search StdSchedulerFactory for String[] dsNames = cfg.getPropertyGroups(PROP_DATASOURCE_PREFIX); See this code org. Quartz. JobStore. The dataSource = # myDS configuration database org. Quartz. The dataSource. MyDS. Driver = com. Mysql.. JDBC driver org.quartz.dataSource.myDS.URL = jdbc:mysql://localhost:3306/quartz-test? characterEncoding=utf-8 org.quartz.dataSource.myDS.user = root org.quartz.dataSource.myDS.password = org.quartz.dataSource.myDS.maxConnections = 5 #org.quartz.dataSource.myDS.connectionProvider.class=org.quartz.utils.HikariCpPoolingConnectionProvider Org. Quartz. The dataSource. MyDS. The provider = hikaricp org. # quartz the rest of the default configuration. Quartz scheduler. InstanceName: DefaultQuartzScheduler org.quartz.scheduler.rmi.export: false org.quartz.scheduler.rmi.proxy: false org.quartz.scheduler.wrapJobExecutionInUserTransaction: false org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount: 10 org.quartz.threadPool.threadPriority: 5 org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread: true org.quartz.jobStore.misfireThreshold: 60000Copy the code

Running results:

Data table interpretation:

The principle of design

UML class diagrams

• There are two main threads: QuartzSchdulerThread and MisfireHandler
  • The core execution logic for scheduling tasks is in QuartzSchdulerThread
  • MisfireHandler is used to resolve the problem of the task not firing.
• JobStore adds, deletes, changes, and checks jobs and triggers, and JobRunShell associates the framework with our customized business jobs

Quartz is the primary priming process

The sequence diagram shows when most of Quartz’s core classes were created.

1. Create the scheduling factory class first. StdSchedulerFactory is generally used to create Scheduler through the factory class. Scheduler properties can be configured via Quartz. Properties

2. StdScheduler, the standard implementation of Scheduler, is the proxy class of QuartzScheduler, and its main behavior is realized through QuartzScheduler.

The 3 QuartzScheduler is instantiated in the StdSchedulerFactory, which uses two objects.

• QuartzSchedulerResources is instantiated with StdSchedulerFactory, which contains the main resources that Scheduler creates and runs processes, such as JobStore and ThreadExecutor.
• The QuartzSchedulerThread is responsible for triggering triggers and interacting with SchedulerSignaler

Quartz task scheduling process

How are the tasks we create scheduled? It is mainly implemented in the scheduling thread QuartzSchedulerThread, and its rough logic

Get the number of available threads in the thread pool. If there are no available threads, the pool will block until there are available threads. Configuration: org. Quartz. ThreadPool. XXX

2 Use JobStore to obtain the trigger to be executed in the next 30 seconds. org.quartz.spi.JobStore#acquireNextTriggers

3 Trigger time of loop and waiting to task configuration

Get TriggerFiredResult from JobStore. TriggerFired

5 For each TriggerFiredResult to execute, create a JobRunShell and put it into the thread pool

• JobRunShell calls the initialization method to create the Job and JobExecutionContext to execute this time. Job = jobdetail.getJobClass ().newinstance (), JobExecutionContext contains JobDetail and Trigger information of this Job.
• Drop the JobRunnerShell into the thread pool and run one of the available workerThreads from the pool.
• Run JobRunnerShell’s run method. job.execute(jec); Execute the Job instance code, before and after the execution can do some listening through listner.

The Quartz mission Misfire process

The Quartz scheduler normally retrieves tasks to be triggered in the future and then loops until a specified time to execute, but the configured task may not be executed at the specified point in time. Reasons for this:

• The system restarts, and during the restart, some tasks are misfired

• Trigger is suspended (suspendXXX) for a period of time during which some tasks are misfired

• The task cannot be executed due to insufficient thread pool resources

• For some tasks, the last task that was executed at the trigger time has not yet finished.

So how does Misfire work?

• Suppose there is a task that needs to be executed at time 0, but at the current time, 80 in the figure, the task has not been executed,
  • If the tasks to be performed at the current time and time 0 are greater than misfireThreshold, the tasks at time 0 are regarded as misfire tasks.
• Tasks at time 0 are then detected by MisfireHandler and its next_trigger_time is set to 90(after the current time).
• Since the task’s next_trigger_time is set to the current time, the scheduling thread redetects the task and fires.

Internal run method execution flow:

1 Scan for triggers that have not been executed between misfireThreshold and this point in time. First of all to count: countMisfiredTriggersInState (conn, STATE_WAITING, getMisfireTime ())

2 If count is greater than 0, the lock is acquired to prevent concurrent access. Then get the Misfire trigger that needs to be triggered.

3 Update trigger next_fire_time based on misfireInstruction. The main method is located at: SimpleTriggerImpl#updateAfterMisfire

4 to submit the connection

5 If there are more Misfire missions, rest for the shortest 50ms. If not, the sleep time is misfireThreshold

The Trigger condition

If you see a picture of Trigger state flow on the Internet, please refer to the following:

Some of the problems

Expected problems with Quartz:

1 The data table structure is fixed, must we follow the official table structure?

• You can implement JobStore and customize the table structure by referring to the JobStoreSupport class

2 Quartz uses database as distributed lock by default, poor performance, how to optimize?

• Custom LockHandler class, using Redis to achieve distributed lock
• Trigger batch processing is used
• Change the order in which tasks are performed
• Reduce context switching

Reference: tech.ebayinc.com/engineering…

The last

I am uneducated, if the process is improper, I hope the big guy can point out mistakes, if there is a discussion about its design principle, also welcome to tease.

Will continue to be updated…

reference

• Quartz principle decryption
• Quartz enterprise application
• Quartz 英 文 版

Quarz uses MongoDB or Redis for storage: github.com/michaelklis… Github.com/jlinn/quart…

Process finished with exit code 0