Introduction to the
Along with the development of the system size, timing task number increasing, the task becomes more complex, especially in the distributed environment, there are multiple business system, every business system has the task of timing requirements, if in their own system scheduling, on the one hand, increase the complexity of the business systems, on the other hand also not convenient management, Therefore, a task platform is required for unified management and scheduling of scattered tasks. Based on the current situation, the task platform needs to support the following aspects: 1. Unified task management, which provides a graphical interface for task configuration and scheduling. 2. Task concurrency control. Only one task can be executed at the same time. 3. Flexible capacity expansion of tasks, which can dynamically increase or decrease server allocation pressure according to busy conditions, and fragment processing of large tasks. 4. Task dependency problem, which can handle the situation that the task contains sub-tasks and trigger the execution of sub-tasks after the completion of the previous one. 5, support multiple types of tasks, support Spring Bean, Shell and so on. 6. The task node is highly available and can be transferred to other nodes when the task node is abnormal or busy. 7. The scheduling center is highly available and supports cluster deployment to avoid single points of failure. 8, execution status monitoring, easy to view the task execution status, abnormal alarms, support multi-channel notification.Copy the codeThe history of
With the development of technology, scheduled task has changed from single-thread scheduling to multi-thread scheduling, from single-machine deployment to cluster deployment, from independent execution to multi-task cooperative execution. ! [file](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2020/1/21/16fc7042cc6838a8~tplv-t2oaga2asx-image.i mage)Copy the codeThe first stage is single-thread scheduling. Prior to Java1.5, thread-based wait (sleep or wait) mechanism was implemented regularly, requiring developers to implement scheduling logic. It is wasteful for a single Thread to handle a single task, but it is easy for a Thread (Timer) to handle multiple tasks because one task is busy and other tasks are blocked.
The ScheduledExecutorService scheduling thread pool is provided in Java1.5. The scheduling thread pool supports fixed delay and fixed interval mode. It is not convenient to execute the thread pool on a certain day or a certain month. It’s a lot of trouble to deal with.
The third stage is Spring task scheduling. Spring simplifies task scheduling by supporting timed execution of a Bean method through the @scheduled annotation. In addition to supporting fixed delay and fixed interval modes, it also supports CRon expressions, making the development of Scheduled tasks extremely simple.
The fourth stage is Quartz task scheduling. In the cluster deployment of task service, Quartz realizes the scheduling and concurrent control of tasks through database lock to avoid the simultaneous execution of the same task. Quartz provides a task scheduling API through Scheduler, on which developers can develop their own task scheduling management platform.
The fifth stage distributed task platform provides a unified platform, without the need to do the development related to scheduling. The business system only needs to realize the specific task logic, automatically register with the task scheduling platform, and complete the development of scheduled tasks through relevant configuration.
The solution
There are many solutions for distributed task scheduling. You can develop task management platforms based on Quartz or use open source task scheduling platforms such as XXl-Job and Elastice-Job.Copy the codeXXL-JOBThe distributed task scheduling platform, released in 2015 by Xu Xueli, an employee of Dianping, is a lightweight distributed task scheduling framework. Its core design goal is rapid development, simple learning, lightweight and easy to expand. The official address: https://www.xuxueli.com/xxl-job/
ELASTIC-JOBThe Elastic distributed task scheduling system developed by Dangdang is rich and powerful. Zookeeper is used to realize distributed coordination, high availability and sharding of tasks, and can support Cloud development. It consists of two independent sub-projects, Elastic-Job-Lite and Elastic-Job-Cloud. The official address: http://elasticjob.io/docs/elastic-job-lite/00-overview/
Scheme comparison
Integration of the sample
The following uses the integration of XXL-job as an example. Xxl-job divides scheduled tasks into two parts: 1. 2. Actuator. Therefore, the integration of XXL-Jobs is divided into two steps: 1. Deploy the scheduling center; 2. Interconnect service systems (actuators). The architecture diagram is as follows: [file](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2020/1/21/16fc704382fc800e~tplv-t2oaga2asx-image.i mage)Copy the codeDeployment and Scheduling Center
Determine the deployment scheme before deployment. In the test environment, you can use one scheduling center and one mysql service. In the production environment, you are advised to use two scheduling centers and the primary and secondary mysql services to ensure high availability. Ensure that the following preparations have been made before the deployment: Jdk1.8, Maven, and mysqlCopy the codeDownload the XXL – job source: http://gitee.com/xuxueli0323/xxl-job/, using maven build packaging, generate the deployment of XXL – job – admin. The jar.
Create a database and initialize related tables. For details, see doc/db/ tablesxxLJob.sql
Properties file in the deployment directory or in a separate application. Properties file in the deployment directory (recommended, Spring Boot loads the configuration in the boot directory first, avoiding the need to repackage the source code when changing configurations such as the database later.
Run the management platform (please ensure that the Java execution environment has been configured, Jdk1.8 or above)
Specific steps in the reference Xu Xue blog: https://www.cnblogs.com/xuxueli/p/5021979.html, deploy the following script for reference:
# download source CD/root/wget https://github.com/xuxueli/xxl-job/archive/v2.0.1.zip unzip v2.0.1. zip mv XXL - job - 2.0.1 XXL - job # modified vim configuration file/root/XXL - job/XXL - job - admin/SRC/main/resources/application properties # # # modify mysql, E-mail and other configuration compilation CD/root/XXL - job Mkdir -p /xxl-job CD /root/xxl-job/xxl-job-admin/target/ cp /root/xxl-job/xxl-job-admin/target/xxl-job-admin-2.0.1.jar /xxl-job/xxl-job-admin-2.0.1.jar Utf8mb4) mysql -u root-proot -e "source /root/xxl-job/doc/db/tables_xxl_job. SQL;" Admin 123456 curl = /dev/null; /dev/null = /dev/null http://localhost:8080/xxl-job-adminCopy the codeThe default user name is admin and the default password is 123456. The password can be changed in the configuration file.
Service System Interconnection
The connection between the business system and the scheduling center needs to be configured according to the framework of the current project. You can refer to the example under XXl-job-executor-samples in the source code. The following is the integration of the business system based on the Spring Boot framework.Copy the codeConfiguration actuator
@Configuration public class XxlJobConfig { @Value("${spring.application.name:}") private String springAppName; @Value("${xxl.job.admin.addresses}") private String adminAddresses; @Value("${xxl.job.executor.appname:}") private String appName; @Value("${xxl.job.executor.ip:}") private String ip; @Value("${xxl.job.executor.port:9999}") private int port; @Value("${xxl.job.accessToken:}") private String accessToken; @Value("${xxl.job.executor.logpath:job-logs}") private String logPath; @Value("${xxl.job.executor.logretentiondays:7}") private int logRetentionDays; @Bean public XxlJobSpringExecutor xxlJobExecutor() { XxlJobSpringExecutor xxlJobSpringExecutor = new XxlJobSpringExecutor(); xxlJobSpringExecutor.setAdminAddresses(adminAddresses); if (StringUtils.isEmpty(appName)) { if (StringUtils.isEmpty(springAppName)) { throw new IllegalStateException("missing xxl-job appname config"); } appName = springAppName; } xxlJobSpringExecutor.setAppName(appName); xxlJobSpringExecutor.setIp(ip); xxlJobSpringExecutor.setPort(port); xxlJobSpringExecutor.setAccessToken(accessToken); xxlJobSpringExecutor.setLogPath(logPath); xxlJobSpringExecutor.setLogRetentionDays(logRetentionDays); return xxlJobSpringExecutor; }}Copy the code1. Spring Boot applications generally have an AppName. By default, spring App Name is used. The InetUtils utility class is provided in Spring-Cloud-Commons, which can help obtain IP3. The InetUtils utility class can be specified by default. If multiple services are deployed on the same server, the InetUtils port can be specified by default. Allocations can be obtained through detection or planning. 4. The logPath must be in the application directory. Do not use an absolute path to avoid conflicts with the application. 5. Logretentiondays The retention period of logs is not too long. Set the retention period as required.
Developing scheduled Tasks
There are two ways to define a scheduled task: 1. In 2.1.2 or later, you can add @xxlJob to a method to declare a task. 2. Prior to 2.1.2, each task required a separate Bean to implement the IJobHandler interface and annotate the class with @JobHandler annotations. The second way is more troublesome, recommended to use the first way (there is no stable version).
Based on @xxlJob annotation code (it is recommended to set the name, and the same as the scheduling center configuration)
@Component public class SampleXxlJob { private static Logger logger = LoggerFactory.getLogger(SampleXxlJob.class); /** * 1. Simple task example (Bean mode) */ @xxlJob ("demoJobHandler") public ReturnT<String> demoJobHandler(String param) throws Exception { XxlJobLogger.log("XXL-JOB, Hello World."); for (int i = 0; i < 5; i++) { XxlJobLogger.log("beat at:" + i); TimeUnit.SECONDS.sleep(2); } return ReturnT.SUCCESS; } /** * 2. @xxlJob ("shardingJobHandler") public ReturnT<String> shardingJobHandler(String param) throws Exception {/ / shard parameter ShardingUtil. ShardingVO ShardingVO = ShardingUtil. GetShardingVo (); Xxljoblogger.log (" Sharding parameter: current sharding number = {}, total number of slices = {}", shardingvo.getIndex (), shardingvo.getTotal ()); For (int I = 0; i < shardingVO.getTotal(); I ++) {if (I == shardingvo.getIndex ()) {xxljoblogger.log (" {} slice, hit fragment start processing ", I); } else {xxljoblogger. log(" {} slice, ignore ", I); } } return ReturnT.SUCCESS; }}Copy the code@Component Public class demoJobHandler extends IJobHandler {@component public class demoJobHandler extends IJobHandler { @Override public ReturnT<String> execute(String param) throws Exception { XxlJobLogger.log("XXL-JOB, Hello World."); for (int i = 0; i < 5; i++) { XxlJobLogger.log("beat at:" + i); TimeUnit.SECONDS.sleep(2); } return SUCCESS; }}Copy the codeConfiguring a Scheduled Task
To configure scheduled tasks, you need to configure actuators first. Automatic registration is recommended to avoid changing machine addresses during cluster deployment. The page for adding scheduled tasks is as follows (Note that appName must be the same as that configured in the service system) :
JobHandler must be the same as the name configured in the code. The cluster deployment of the actuator can be controlled by configuring the routing mode. Only cron expressions are supported for xxL-job scheduling.
3. Start or execute tasks, query execution logs, and register nodes
Integrated trampling record
Clock synchronization must be performed on the task server. The executor clock cannot be scheduled for 180 seconds. Otherwise, scheduling fails (restricted by the RPC framework). 3, try to avoid short tasks, such as second tasks, which can lead to a large number of database locks affecting performance; 4. The query is slow due to a large number of scheduling logs. Logs are recorded in the database and need to be cleared periodically. 5. Automatic registration fails due to multiple network adapters on the server. You must specify the NETWORK adapter IP address during registration.
Reference: http://blog.freshfood.cn/article/39https://www.cnblogs.com/xuxueli/p/5021979.htmlhttps://www.xuxueli.com/page/projects.h tml