Tags: springbatch

1. The introduction

Spring Batch(1) Introduction and Application Scenarios have introduced Spring Batch is a lightweight and complete Batch processing framework. It is simple and convenient to use. It is suitable for developers who have some programming foundation (especially using Spring and SpringBoot framework). For business programming, you only need to worry about the specific business implementation, leaving the process and control of the process to Spring Batch. As the saying goes “Talk is cheap, show me the code”, let’s enter the world of Spring Batch through a simple Hello world. Through this example, You can quickly understand the process of developing a Batch and the components used in Spring Batch development, setting the stage for subsequent operations.

2. Development environment

  • The JDK: jdk1.8
  • Spring the Boot: 2.1.4. RELEASE
  • Spring Batch: 4.1.2. RELEASE
  • Development IDE: the IDEA
  • Build tool Maven: 3.3.9
  • Log component Logback :1.2.3
  • Lombok: 1.18.6

3. The helloworld development

3.1 the helloworld instructions

This HelloWorld implements a very simple function that reads a string from a data group, capitalizes the string, and prints it to the console. As shown in figure:

The entire process is a batch Job. It has only one Job Step, which is divided into three stages: read data (ItemReader), process data (ItemProcessor) and write data (ItemWriter).

3.2 Development Process

The main code developed is as follows:

In general, tasks are completed using Reader, Processor, and Writer. After the task is complete, tasks are configured using BatchConfig.

3.2.1 createSpring Bootengineering

It can be generated directly using Idea or in Spring Initializr, which is not specified here. You can also use my code examples directly. The current Spring Boot version in use is 2.1.4.release

3.2.2 Adding dependencies

  • Spring BatchDepend on in usespring-boot-starter-parentIn this case, add the following dependencies:
<! --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId> Spring-boot-starter-Batch </artifactId> </dependency>Copy the code

After the reference, two JAR packages are referenced, one is spring-Batch-Infrastructure and one is spring-Batch-core, version 4.1.2.release. The corresponding layers are the base framework layer and the core layer respectively.

  • Add memory database H2Spring BatchIs required to store the basic information of the task and the running status of the database. In this example, there is no need to operate the database logic, directly use the in-memory database H2. Add the following dependencies:
<dependency>
	<groupId>com.h2database</groupId>
	<artifactId>h2</artifactId>
</dependency>
Copy the code
  • Add test and utility class dependencies to simplify development, uselombokProcess. useSpring BootTo unit test, add the following dependencies:
<! -- Tool kit: Lombok --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.6</version> </dependency> <! <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope>
</dependency>
Copy the code

3.2.3 Develop the reading data component ItemReader

Once the dependencies are added, you can move on to business logic programming. According to Spring Batch’s Batch process, reading the data ItemReader is the first step, and in the current example, our task is to read the data from an array. ItemReader is an interface that developers implement directly. This interface defines the core method read(), which is responsible for reading available data from a given resource. The concrete implementation is as follows:

@Slf4j
public class StringReader implements ItemReader<String> {
    private String[] messages = {"aaa1"."aaa2"."aaa3"."aaa4"};
    private int count = 0;
    @Override
    public String read(a) throws UnexpectedInputException, ParseException, NonTransientResourceException {
        if(count < messages.length){
            String message = messages[count++];
            log.debug(LogConstants.LOG_TAG + "read data:"+message);
            return message;
        }else{
            log.debug(LogConstants.LOG_TAG + "read data end.");
            count = 0;
        }
        return null; }}Copy the code

Description:

  • (1) StringReader implements ItemReader;
  • (2) Messages is the data source;
  • (3) count represents the subscript of the data read. Each time the subscript increases, and null is returned after the reading to indicate the end. Also set count to 0 for the next read.
  • (4) Log output using Logback, combined with Lombok@Slf4jAnnotation, can directly use log output, simplify operations.

3.2.4 Develop the data processing component ItemProcessor

After reading the data, the returned data is streamed to the ItemProcessor for processing. Similarly, ItemProcessor is an interface that implements its own processing logic by implementing this interface. Of course, if there is no ItemProcessor, it is also possible to read the data directly to the ItemWriter process. Spring Batch uses the concept of Chunk to read data for multiple times. After the number of chunks is specified, the data is sent to the processor and writer to improve efficiency. This example is a simple implementation of ItemProcessor, which capitalizes the string. As follows:

@Slf4j
public class ConvertProcessor implements ItemProcessor<String.String> {
    @Autowired
    private ConsoleService consoleService;
    @Override
    public String process(String data) {
        String dataProcessed = consoleService.convert2UpperCase(data);
        log.debug(LogConstants.LOG_TAG + data +" process data --> " + dataProcessed);
        returndataProcessed; }}Copy the code

Description:

  • Implements the ItemProcessor interface, which has two generics, I and O, where I is the data obtained in the read phase and O is the data submitted to the write phase.
  • Use the ConsoleService service to convert data to uppercase, where the implementation uses strings directlytoUpperCase()methods

3.2.5 Develop the write data component ItemWriter

After the data is processed, the write component (ItemWriter) will write the data. ItemWriter is also an interface with a write method as its core and an array as its arguments. To implement your own logic, just implement this interface. In this example, you simply output the data to the log. As follows:

@Slf4j
public class ConsoleWriter implements ItemWriter<String> {
    @Override
    public void write(List<? extends String> list) {
        for (String msg :list) {
            log.debug(LogConstants.LOG_TAG + "write data: "+msg); }}}Copy the code

3.2.6 JobExecutionListener after the development task is complete

After the data is written to the target, the task is finished, but sometimes we also need to do some other work at the end of the task, such as clearing data, updating time, etc., which needs to be logically processed after the task is completed. Spring Batch provides listening for both the start and end of a task or step to make it easier for developers to implement listening logic. Such as through inheritance JobExecutionListenerSupport, can realize beforeJob and afterJob listening, in order to realize the task before starting and ending task after processing. In the current example, only logs about task completion are output. As follows:

@Slf4j
public class ConsoleJobEndListener extends JobExecutionListenerSupport {
    @Override
    public void afterJob(JobExecution jobExecution) {
        if(jobExecution.getStatus() == BatchStatus.COMPLETED){
            log.debug("console batch job complete!"); }}}Copy the code

3.2.7 Configuring a Complete Task

After reading, processing, writing, and listening after the task is completed, you now need to assemble them together into a completed task that can be assembled with a few simple configurations using Spring Boot. The relationships between tasks and their related components are as follows:

Create the configuration file consolebatchconfig.java as follows:

@Configuration
@EnableBatchProcessing
public class ConsoleBatchConfig {
    @Autowired
    public JobBuilderFactory jobBuilderFactory;
    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job consoleJob(Step consoleStep,JobExecutionListener consoleListener){
        String funcName = Thread.currentThread().getStackTrace()[1].getMethodName();
        return jobBuilderFactory.get(funcName).listener(consoleListener).flow(consoleStep)
                .end().build();
    }

    @Bean
    public Step consoleStep(ItemReader stringReader,ItemProcessor convertProcessor ,ItemWriter consoleWriter, CommonStepListener commonStepListener){
        String funcName = Thread.currentThread().getStackTrace()[1].getMethodName();
        return stepBuilderFactory.get(funcName).listener(commonStepListener)
                .<String,String>chunk(3).reader(stringReader).processor(convertProcessor)
                .writer(consoleWriter).build();
    }

    @Bean
    public ItemReader stringReader(a){return newStringReader(); }@Bean
    public ItemWriter consoleWriter(a){return newConsoleWriter(); }@Bean
    public ItemProcessor convertProcessor(a){return newConvertProcessor(); }@Bean
    public JobExecutionListener consoleListener(a){return new ConsoleJobEndListener();}
}
Copy the code

Description:

  • Add annotations@ the Configuration andand@EnableBatchProcessing, identified as Configure and enableSpring BatchConfiguration (can be used directlyJobBuilderFactoryandStepBuilderFactoryUsed to create jobs and steps, respectively.
  • createItemReader,ItemWriter,ItemProcessor,ListenerThe correspondingBeanFor Step and Job injection.
  • usestepBuilderFactoryCreate job Step, in which Chunk performs block-oriented processing, that is, read several times and then write to improve efficiency. Currently, three chunks are configured as one chunk.
  • usejobBuilderFactoryAdd step to create a task.
  • Note that both step and Job should have names (getMethod confirmation), the method name is used as the name of the Job and Step.

3.2.8 Test batch processing

After the above steps, the development of the Job is completed. There are two ways to test the Job: one is to write Controller and run the Job as an interface call, and the other is to write unit tests.

  • The Job succeeds. ProcedureJobLaunchertherunMethod to run the task,runThe method parameters are respectivelyJobandjobParametersIs the configured Job and Job running parameters. Each task is distinguished by the task name (jobName) and task parameters (jobParameters) as a distinction, i.e. ifjobNameandjobParametersThe same,Spring BatchThe system considers that the task is the same. If the task is successfully run, the same task will not run again. Therefore, in general, different tasks are oursjobParametersYou can use time directly as the parameter for easy distinction. generatejobParameters. The code is as follows:
JobParameters jobParameters = new JobParametersBuilder()
                .addLong("time",System.currentTimeMillis())
                .toJobParameters();
Copy the code
  • Write unit test writeConsoleJobTest, load the job and run the test as follows:
@RunWith(SpringRunner.class)
@SpringBootTest(classes = {MainBootApplication.class,ConsoleBatchConfig.class})
@Slf4j
public class ConsoleJobTest {
    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private Job consoleJob;

    public void testConsoleJob2(a) throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException {
        // Build parameters
        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("time",System.currentTimeMillis())
                .toJobParameters();
        // Execute the taskJobExecution run = jobLauncher.run(consoleJob, jobParameters); ExitStatus exitStatus = run.getExitStatus(); log.debug(exitStatus.toString()); }}Copy the code

Note: When introducing Spring Batch annotations, you need to include the Spring Batch task.

  • The following figure shows the execution result:

    As can be seen from the output, the chunk is set as 3. After reading the three pieces of data, they will be capitalized to ItemProcessor and then handed to ItemWriter for writing. After the task is executed, exitCode of the Job indicates the task execution status. If COMPLETED is COMPLETED, FAILED is displayed.

4. To summarize

After the preceding steps, the batch processing is complete. As for the state of the task, the steps of the process (read, process and write) are all handed over to Spring Batch. All the developers have to do is to compile specific read data, process data and write data according to their own business logic. Hopefully, this article has given you a clear understanding of Spring Batch’s components.