Introduction: This paper briefly describes the resource definition, visual management and control capability and distributed batch processing capability of task scheduling platform, and provides some basic reference cases based on SchedulerX capability combined with actual business scenarios. It is hoped that the above content can make you easily familiar with the overview of the access and use of the task scheduling platform. For existing users, the platform resources can be controlled and isolated based on the characteristics of their own teams, and the processing efficiency can be improved through the distributed batch processing capability when the product business volume increases.

Author: Qian Xi

preface

In all kinds of business system scenarios, there are a large number of demand scenarios for periodic triggering and periodic triggering to run specified business tasks, and the significance of the existence of distributed task scheduling middleware platform is to support the above scenarios. Crontab in Linux, Timer in Java, etc., are involved in periodic scheduling and running tasks to complete a series of automatic processing business scenarios.

Periodic periodic operation is the most basic feature of task scheduling, but with the expansion and development of business, the capability of task scheduling middleware platform will be higher. In traditional business systems, Quartz and Spring Scheduling packages provide a lot of convenience for business development scheduled tasks in the form of framework integration. However, with the distributed and micro-service deployment of each business application, A large number of scattered scheduled tasks are scattered in various business application systems, so it is difficult to monitor, operate and manage all tasks in a unified way. The distributed task scheduling middleware platform will effectively manage and control all scheduled tasks in a unified way.

The distributed task scheduling platform is based on highly reliable scheduled task scheduling and visualized management and control as the core value. Combined with the business development trend, the platform is developed around these two basic elements.

SchedulerX overview

Platform Resource Management

From the perspective of unified O&M management and control for all service applications globally, the “space” concept is abstracted from the team and deployment environment for resource management isolation, and a layer of “application group” is abstracted from service applications and machine clusters. Each application group can be created on the scheduling platform to interconnect with the actual application. In this way, each service team can independently manage the tasks of its own service team on a unified platform. Moreover, reasonable resource permission control and isolation can be carried out uniformly on Ali Cloud based on RAM permission policy.

Based on the above resource model, service platform architecture managers can reasonably plan space and applications and configure permission policies according to their team’s organizational characteristics, and clearly achieve task scheduling resource access control isolation and global control.

Platform visual control

Task is the basic unit of task scheduling platform to control scheduling operations. The visualization of task operation enables the hidden tasks running silently in various business applications to be exposed to the light of day, and enables the running status and execution results of each task to be displayed and reminded. In the absence of visualization, task running status and results will be unknown or difficult to detect, and even after a large number of business iterations, the number of scheduled tasks in the application system cannot be standardized management.

Distributed shard batching

With the development of business volume, distributed batch processing of large amount of data will be required in some scheduled task scenarios. It has also become an important feature of task scheduling platform to coordinate multiple machines to process large quantities of data on a regular basis. Through the distributed batch processing model, users can simply improve the efficiency of processing large quantities of data.

Simple use scenario

The following lists a few simple service usage scenarios based on the task scheduling platform to get a preliminary understanding of what can be done on the task scheduling platform.

Based on broadcast cluster O&M scenario

In broadcast mode, the created scheduling task is sent to the service application cluster or ECS cluster with Agent installed at a specified period. The newly added resources are also broadcast dynamically. Based on this feature, users can construct many customized scenarios, for example:

  • Periodically clearing logs and temporary data: Periodically clearing logs and temporary files generated by services.
  • Server detection: Users can use broadcast shell scripts to quickly construct simple server disk, memory, and CPU health detection scenarios.
  • Refresh service cache: Periodically refresh the cache and preheat specified service applications in service scenarios where local cache exists.
  • Service service detection: Users can customize the collection and judgment of various service indicators and flexibly build lightweight service monitoring based on the scheduling platform.

Timing Service Scenario

In various business scenarios, scheduled task processing on the task scheduling platform is the most extensive service form. For example:

  • Send message reminder regularly: payment reminder, credit expiration reminder, birthday greeting reminder of customer and employee, transaction order processing notification, etc.
  • Regular data synchronization processing: employee organizational structure information synchronization, basic business information synchronization, regular daily business data batch liquidation, transaction order timeout processing, etc.
  • Regular data generation push: monthly/quarterly/annual report generation push, regular activity announcement push, regular consumption bill generation push and so on.

When the volume of data to be processed gradually expands, the processing efficiency bottleneck will gradually appear in the original single-machine processing mode. At this time, the distributed cluster batch processing capability of task scheduling can make full use of the cluster collaboration capability to process large quantities of data in parallel.

Distributed batch processing scenario

Based on the service batch processing capability, when the real-time traffic is large, users can construct regular service batch compensation tasks in coordination with the service degradation policy to improve the peak service processing capability and ensure the core service throughput during peak service periods. As an architect of business system platform, if the timing distributed batch processing capability can be effectively combined with specific business, the overall business processing capability and stability of the system will be fully brought into play. The following is an abstract summary of relevant service usage scenarios for reference.

In a typical scenario, the core transaction service might handle the invocation of downstream services via RPC/MQ. However, in this mode, there will be some problems when the business volume increases. RPC downstream service capacity will affect the throughput capacity of the core service, and MQ dependency needs to ensure that the message delivery is normal and the consumer side processes correctly, which will become the bottleneck for the improvement of the processing capacity of the core service. Therefore, within the scope of business to accept, take complete decoupling between business services regularly batch compensation way will greatly improve the processing efficiency, and through the idempotence repeatable execution will improve the downstream service reliability, security, and combining with distributed batch model can be bulk loading, bulk processing of data in order to reduce the DB interaction stress. At the same time, the compensation mode can be integrated with micro-service degradation in architecture design.

Quick access to SchedulerX

Creating an Application Group

On the task scheduling console, choose Public Network Region, and choose Application Management > Create an application. The application information will be the core element for the service application, Agent, and task scheduling platform to establish the connection relationship in the following steps.

The SpringBoot project introduces SchedulerX

Add the following dependencies to the locally constructed SpringBoot project and the application group configuration information created on the corresponding console in Properties. After the application is started, the relationship between the service application and the task scheduling platform can be established.

<dependencies> <dependency> <groupId>com.aliyun.schedulerx</groupId> <artifactId> Schedulerx2-spring-boot-starter </artifactId> <version>1.3.2</version> </dependency> </dependencies> </dependencies> # Public clouds spring public environment. Schedulerx2. Namespace = aad167f6-8 bee - 41 a7 - ba41 - * * * * * * * * * spring.schedulerx2.endpoint=acm.aliyun.com spring.schedulerx2.groupId=qianxi.text spring.schedulerx2.appKey=lYgR6qq**********Copy the code

Other instructions

After completing the preceding steps, you can create and develop scheduled tasks for specific services in subsequent applications. For details, see the official manual.

conclusion

This paper briefly describes the resource definition, visual management and control ability and distributed batch processing ability of task scheduling platform, and provides some basic reference cases based on the ability of SchedulerX combined with actual business scenarios. It is hoped that the above content can make you easily familiar with the overview of the access and use of the task scheduling platform. For existing users, the platform resources can be controlled and isolated based on the characteristics of their own teams, and the processing efficiency can be improved through the distributed batch processing capability when the product business volume increases.

In the future, SchedulerX will continue to improve its visual management and control ability to meet the needs of enterprise-level scheduled task management, and the platform’s management and control and scheduled scheduling service capabilities will gradually be compatible with common open source component clients in the market. We will also combine with the actual scene, continue to analyze the explanation one by one, stay tuned.

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.