This article is intended for those with basic Java knowledge

Author: HelloGitHub – Salieri

The introduction

HelloGitHub introduces the Open Source project series.

Project Address:

https://github.com/KFCFans/PowerJob

PowerJob is designed to be an enterprise-level distributed task scheduling platform, that is, to become the scheduling middleware within the company. Powerjob-server is deployed in the entire enterprise. All applications of its line of business rely on Powerjob-Worker to access task scheduling and distributed computing capabilities.

Therefore, PowerJob consists of powerJob-server and Powerjob-worker. Powerjob-server is responsible for providing Web services and scheduling tasks. Powerjob-worker is responsible for executing the task code written by the user and providing distributed computing capability.

Here’s a quick tutorial for setting up and trying out PowerJob in a local development environment.

STEP1: initialize the project

  1. git clone https://github.com/KFCFans/PowerJob.git

  2. To import the IDE, the source structure is as follows. We need to start the Scheduling server (PowerJob-Server) and write our own processor code in the Samples project


STEP2: start the scheduling server

  1. Create database (no need to create table, just create database) powerjob-daily

  2. Modify the configuration file. The description of the configuration file is very detailed in the official document. Since we set up the test environment in the local IDE, we chose to use the development environment configuration file application-daily.properties to modify and use it. Where you need to modify the database configuration:

    • spring.datasource.core.jdbc-url
    • spring.datasource.core.username
    • spring.datasource.core.password
    • Of course, students with mongoDB can also modifyspring.data.mongodb.uriTo get the full experience.

  3. After the configuration file is modified, you can directly start the scheduling server using the startup class OhMyApplication and observe the startup logs to check whether the startup is successful. After the startup is successful, visit http://127.0.0.1:7700/. If the Web page is displayed successfully, The scheduling server is successfully started.

  4. Register the app: Click the home app register button, fill in the Powerjob-agent-test and console password (for access to the console), and register the sample app (you can also register other appNames, but don’t forget to synchronize them in the sample app).


STEP3: write sample code

Go to the sample project (PowerJob-worker-samples), modify the configuration file to connect to PowerJob-Server and write your own processor code.

  1. Modify the startup configuration class OhMySchedulerConfig for PowerJob-worker-samples to change appName to the name just registered on the console.

  2. Write your own processor: Create a new class in the sample project that inherits the processor you want to use (see the official documentation for each processor). For a simple demonstration, use BasicProcessor, a stand-alone processor. The following code example demonstrates the online logging function.

    @Slf4j
@Component
public class StandaloneProcessorDemo implements BasicProcessor {

    @Override
 public ProcessResult process(TaskContext context) throws Exception {   OmsLogger omsLogger = context.getOmsLogger();  omsLogger.info("start process,context is {}.", context);  System.out.println("jobParams is " + context.getJobParams());   return new ProcessResult(true."process successfully~");  } } Copy the code

  3. Start the sample program, that is, run the main class SampleApplication directly and observe the console output to determine whether it is successfully started.

STEP4: task configuration and operation

After the scheduling server and sample project are started, go to the Web page (http://127.0.0.1:7700/) again to configure and run the task.

  1. Enter the application name in the text box on the home page. After the operation succeeds, the front-end management page is displayed.

  1. Click Task Management -> New Task (upper right corner) to create a task.

  1. After the task is created, you can see the task created on the console. If you feel that the waiting time is too long, you can directly click the Run button to run the task immediately.


  2. Go to the sample task sidebar to view the running status and online logs of the task


4.1 Basic Configuration

  • Task name: Name
  • Task description: Description
  • Task parameters: parameters that can be obtained during task processing (input parameters of each Processor’s process method)TaskContextObject’s jobParams property.
  • Timing information: Indicates the triggering mode of the task. It consists of a drop-down box and an input box
  • API -> No parameter is required, indicating that the task is triggered by OpenAPI
  • CRON -> Fill in the CRON expression (online generation site)
  • Fixed frequency -> Enter an integer in milliseconds
  • Fixed delay -> Fill in an integer in milliseconds
  • Workflow -> No parameters are required, indicating that the task is triggered by workflow

4.2 Performing the Configuration

It consists of execution type (single machine, broadcast, and MapReduce), processor type, and processor parameters. The latter two items are related to each other.

  • Built-in Java processor -> fill in the processorFully qualified class name(eg,com.github.kfcfans.oms.processors.demo.MapReduceProcessorDemo)
  • Java (Container) -> Fill inContainer ID# processor fully qualified class name(eg,18#com.github.kfcfans.oms.container.DemoProcessor)
  • SHELL -> Fill in the script to be processed (directly copy the file content) or the script download link (http://xxx)
  • PYTHON -> Complete the PYTHON script or download the link (http://xxx)

4.3 Running configuration

  • Maximum number of instances: the number of simultaneous instances of the task
  • Concurrent number of single-machine threads: the number of threads used by each Worker during the execution of this instance (MapReduce job takes effect, whatever else is filled, only the necessary number of threads will be used…)
  • Running time limit: Sets the maximum running time of a task. If a task expires, it is regarded as a failure, expressed in milliseconds. 0 indicates that the timeout period is not limited.

4.4 Retry Configuration

  • Number of Instance retries: indicates the Instance level. If the task Instance fails, the TaskTracker (the Master node of the task Instance) will be replaced. Therefore, use large Maps or MapReduce with caution.
  • Task retry count: Task level. After each sub-task fails, the ProcessorTracker (the Worker node that the Task actually executes) will be replaced. This parameter is recommended because it costs less.
  • Note: Note that the retry magnification after task retries and subtask retries are set at the same time. For example, if the task retries and subtask retries are set to 1 and both task retries fail to be executed, the actual execution times will become 4 times. You are advised to set task instance retry to 0 and the number of subtask retries based on the actual situation.

4.5 Machine Configuration

Used to indicate the state of machines that are allowed to perform tasks, avoiding those that are teetering, 0 for no restrictions.

  • Minimum NUMBER of CPU cores: A floating point number. Workers with less than this number of available CPU cores will not perform this task.
  • Minimum memory (GB) : Enter a floating point number. Workers with less available memory will not perform this task.
  • Minimum disk (GB) : Specifies a floating point number. Workers with less available disk space will not perform this task.

4.6 Cluster Configuration

  • Execution machine address: specifies the number of machines in the cluster to execute the task (a good helper of debug)192.168.1.1:27777192168 1.2:27777
  • Maximum number of machines to execute: limits the number of machines to execute

4.7 Alarm Configuration

Select the object for alarm notification after task execution failure, and input it in advance.

More examples can be found in the official documentation, workflow, MapReduce, containers and other advanced features to explore!

V. Summary and notice

This chapter describes how to get started with PowerJob and how to quickly set up a local development/test environment. In the next chapter, I will introduce the overall architecture of PowerJob to prepare for the technical analysis that follows.

I’ll see you next time

The author travels


“If you often walk by the river, you will never get wet shoes.” However, this is an extravagant hope in Chaka Salt Lake.

If you are not careful, the hidden salt caves will open up to you and engulf you in an instant

So is salt lake, so is life.

Hard to avoid misfortune, also want to smile to the beautiful tomorrow ~

Follow the public account to join the communication group (author in Java Group)