Series navigation:
Familiar with The Registry and nodes of the Elastic Job series
Skilled use of The Elastic Job Series introductory Demo (3)
Elastic-job is a distributed scheduling solution consisting of two independent subprojects, elastic-Job-Lite and Elastic-Job-Cloud.
Elastik-job-lite is a lightweight, decentralized solution that uses JAR packages to coordinate distributed tasks and relies on only Zookeeper.
Project source address ==> ShardingSphere-ElasticJob
Operating Environment Requirements
| The framework | version |
|---|---|
| Java | JDK1.7 + |
| Zookeeper | 3.4.6 + |
| Maven | 3.0.4 + |
Directory Structure description
Elastic-job ├── Elastic-Job Lite Java Support Module ├─ Elastic-Job Lite Java Support Module ├ Exercises - Elastic Job-lite-Lifecyle Lite Do not use └ exercises ─elastic- Job-lite-console Lite ├──elastic job-example ├─ Elastic job-example Zk Built-in ZK ├─ Elastic job-example Job ├─ Elastic job-example Job ├──elastic- Job -example- Lite ├─ Elastic - Job -example- Lite - Spring ├─ Elastic - Job -example- Lite - Spring ├─ Elastic Job-example-Lite-SpringBoot Based On SpringBoot Application Examples ├─ elastic job-doc MarkDown └ exercises - - Elastic - Job-lite-doc LiteCopy the codeFunction is introduced
- Distributed job scheduling coordination
- Flexible capacity expansion and reduction
- Failed job transfer
- Missed re-firing of executed jobs
- Job shard consistency ensures that the same shard has only one execution instance in distributed environment
- Self-diagnose and fix problems caused by distributed instability
- Supports parallel job scheduling
- Support job life cycle operations
- Rich job types
- Spring integration and namespace provision
- Operation and maintenance platform management
- .
This section describes the concept of Elastic-Job
Shard concept
In the distributed execution of tasks, a task needs to be split into multiple independent tasks, and then distributed servers execute one or several fragments respectively.
For example, there is a job traversing a table in a database with two servers. To execute jobs quickly, each server should perform 50% of the jobs. To meet this requirement, the job can be divided into two pieces, one for each server. The logic of job traversal data should be: Server A traverses data whose ID ends in odd; Server B iterates over data whose ids end in even numbers. If the data is divided into 10 pieces, the logic of the job traversal data should be as follows: the shard item assigned to each piece should be ID%10, and server A is assigned to shard item 0,1,2,3,4. Server B is allocated to shard items 5,6,7,8,9. The direct result is that server A traverses the data whose ids end with 0-4. Server B traverses the data whose ids end in 5-9.
The shard items are decoupled from the service
Pexe-job does not directly provide data processing functions. The framework only distributes shard items to each running Job server. Developers need to handle the relationship between shard items and real data by themselves.
Application scenarios of personalization parameters
The personalization parameter, shardingItemParameter, can match the corresponding relationship with the shard item and is used to convert the number of the shard item into a more readable business code.
For example, split the database by region level. Database A is the data of Beijing. Database B is the data of Shanghai; Database C is the data of Guangzhou. If the configuration is based only on shard items, developers need to know that 0 stands for Beijing; 1 indicates Shanghai. 2 is Guangzhou. The proper use of personalization parameters can make the code more readable. If 0= Beijing,1= Shanghai,2= Guangzhou is configured, the enumeration values of Beijing, Shanghai, Guangzhou can be directly used in the code to complete the corresponding relationship between the sharding items and the business logic.
Distributed scheduling
There is no central Job scheduling node in elastice-Job-Lite. Instead, applications that deploy the Job framework trigger Job scheduling when they reach a certain point in time.
The registry is used only for job registration and monitoring information storage. The main job node is only used for functions such as sharding and cleaning.
Operational high availability
Elastic-Job-Lite provides the most secure way to execute jobs. Set the total number of shards to 1 and use more than one server to execute the job. The job will be executed in 1 master and n slave mode.
Once the server executing the job crashes, the server waiting to execute the job takes over the next time the job starts. If failover is enabled, the failover function works better. In this case, if a job crashes during execution, the standby machine immediately starts backup execution.
Make the most of resources
Elastice-job-lite also provides the most flexible way to maximize the throughput of executing jobs. By setting the number of shard items to be greater than the number of servers, preferably greater than a multiple of servers, the job will make proper use of distributed resources and dynamically allocate shard items.
For example, if three servers are divided into 10 slices, the partition item allocation result is server A=0,1,2. Server B = three, four, five; Server C=6,7,8,9. If server C crashes, the result of fragment entry allocation is server A=0,1,2,3,4. Server B=5,6,7,8,9. Maximize the use of existing resources to improve throughput without losing shard entries.
