1.Spring Batch framework working principle
Before diving into the code, let’s take a look at the Spring Batch framework. It contains the following major building blocks:
Do not miss the Spring family bucket study notes.
Spring Batch framework
A Batch process consists of one Job. This entity encapsulates the entire batch process.
A Job can consist of one or more steps. In most cases, a step will read the data (through ItemReader), process the data (using ItemProcessor), and then write the data (through ItemWriter).
The JobLauncher process starts a Job.
Finally, JobRepository stores metadata about configured and executed jobs.
To demonstrate how Spring Batch works, let’s build a simple Hello World Batch job.
In this case, we read a person’s first and last name from the person.csv file. Generate a greeting from this data. Then write this greeting to the greeting. TXT file.
2. Overview of examples
We will use the following tools/frameworks:
- Spring Batch 4.1
- Spring Boot 2.1
- Maven 3.6
Our project directory structure is as follows:
Project directory structure
3. The Maven configurations
We used Maven to build and run the sample. If not, download and install Apache Maven.
Let’s use Spring Initializr to generate the Maven project. Ensure that Batch is selected as the dependency.
Click Generate Project to Generate and download the Spring Boot Project template. In the root directory of your project, you will find a pom.xml file that is the XML configuration file for your Maven project.
To avoid having to manage version compatibility for different Spring dependencies, we will inherit the default configuration from the spring-boot-starter-parent parent POM.
The resulting project includes Spring Boo Starters that manage different Spring dependencies.
Spring-boot-starter-batch Imports Spring Boot and Spring Batch dependencies.
Spring-boot-starter-test contains dependencies for testing spring boot applications. It imports libraries including JUnit, Hamcrest, and Mockito.
This also has the spring-batch-test dependency. This library contains helper classes that will help test batch jobs.
In the plugins section, you will find the Spring Boot Maven plug-in: spring-boot-Maven-plugin. It helped us build a single, runnable Uber-JAR. This is a convenient way to execute and publish code. In addition, the plug-in allows you to launch samples through Maven commands.
<? xml version="1.0" encoding="UTF-8"? ><project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.codenotfound</groupId>
<artifactId>spring-batch-hello-world</artifactId>
<version>0.0.1 - the SNAPSHOT</version>
<packaging>jar</packaging>
<name>spring-batch-hello-world</name>
<description>Spring Batch Hello World Example</description>
<url>https://codenotfound.com/spring-batch-example.html</url>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.5. RELEASE</version>
<relativePath /> <! -- lookup parent from repository -->
</parent>
<properties>
<java.version>11</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Copy the code4. Configure Spring Boot
We used Spring Boot to make a Spring Batch application “run straight”. Start by creating a SpringBatchApplication class. It contains the main() method, which launches the application using Spring Boot’s SpringApplication.run().
Note that @SpringBootApplication is a handy annotation that adds :@Configuration, @EnableAutoConfiguration, and @ComponentScan.
For more information about Spring Boot, see the Spring Boot Getting Started Guide.
By default, Spring Batch uses a database to store metadata on configured Batch jobs.
In this case, instead of using a database directly, we run Spring Batch using a memory-based Map.
Spring-boot-starter-batch starter relies on spring-boot-starter-JDBC and will attempt to instantiate the data source. Add exclude = {DataSourceAutoConfiguration. Class} add @ SpringBootApplication annotations. This prevents Spring Boot from automatically configuring the DataSource for the database connection.
package com.codenotfound;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
public class SpringBatchApplication {
public static void main(String[] args){ SpringApplication.run(SpringBatchApplication.class, args); }}Copy the code5. Create the entity model
Before you process data, you usually want to map it to entity objects.
In my example, the input data is stored in the SRC/test/resources/CSV/persons. The CSV file.
Each line in the file contains a comma-separated first and last name.
John, Doe
Jane, Doe
Copy the codeWe will map this data to the Person object. This is a simple POJO with a first and last name.
package com.codenotfound.model;
public class Person {
private String firstName;
private String lastName;
public Person() {
// default constructor
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
@Override
public String toString() {
return firstName + ""+ lastName; }}Copy the code6. Configure Spring Batch Job
Let’s start by creating a BatchConfig class that will configure Spring Batch. The @Configuration annotation at the top of the class indicates that Spring can use the class as a source for bean definitions.
We added the @enableBatchProcessing annotation, which supports all the required Spring Batch features. It also provides the basic configuration for setting up batch jobs.
There is a lot to do by adding this annotation. Here is an overview of the @enablebatchProcessing creation:
- JobRepository (bean name “JobRepository “)
- JobLauncher (bean name “JobLauncher “)
- JobRegistry (bean name “JobRegistry “)
- JobExplorer (bean name “JobExplorer “)
- PlatformTransactionManager bean name (” transactionManager “)
- JobBuilderFactory (bean name “jobBuilders”), which conveniently prevents you from having to inject a Job repository into every Job(Job)
- StepBuilderFactory (bean name “stepBuilders”) to make it easier for you to avoid injecting a job repository and transaction manager into every Step
In order for Spring Batch to use map-based JobRepository, we need to extend DefaultBatchConfigurer. Override the setDataSource() method to not set the DataSource. This results in automatic configuration using the Map-based JobRepository.
@Configuration
@EnableBatchProcessing
public class BatchConfig extends DefaultBatchConfigurer {
@Override
public void setDataSource(DataSource dataSource) {
// initialize will use a Map based JobRepository (instead of database)}}Copy the codeNow let’s go ahead and configure the Hello World Spring Batch job.
Create a HelloWorldJobConfig Configuration class and add the @Configuration annotation.
In the HelloWorldJobConfig Bean, we use JobBuilderFactory to create the job. We pass the name of the Job and the Step that needs to be run.
Note that in the helloWorlJob()Bean,Spring will automatically inject jobBuilders and stepBuilders Beans.
The HelloWorldStepBean defines the different items that our step executes. We use StepBuilderFactory to create the steps.
First, we pass in the name of the step. Using chunk(), we specify the number of items to process in each transaction. Chunk also specifies the input (Person) and output (String) types of the step. We then add ItemReader (reader), ItemProcessor (processor), and ItemWriter (writer) to the step.
We use FlatFileItemReader to read the Person CSV file. This class provides basic functionality for reading and parsing CSV files.
Have a FlatFileItemReaderBuilder implementation, it allows us to create a FlatFileItemReader. We start by specifying that the result of reading every line in the file is a Person object. We then add a name to the FlatFileItemReader using the name() method and specify the resource (in this case, the persons.csv file) that needs to be read.
In order for FlatFileItemReader to process our file, we need to specify some additional information. First, we define the data in the file to be delimited (comma as the default delimiter).
We also specify how to map each field in a row to the Person object. This is done using names(), which enables Spring Batch to map fields by matching names with setters on objects. In this article’s example, the first field of the row will be mapped using the firstName setter. To do this, we also need to specify a targetType, the Person object.
Note:
names(new String[] {"firstName"."lastName"})
Copy the codePersonItemProcessor processes the data. It converts a Person to a greeting String. We will define it in a separate class below.
Once the data is processed, we write it to a text file. We use the FlatFileItemWriter to do this.
We used to create a FlatFileItemWriter FlatFileItemWriterBuilder implementation. We add a name to Writer and specify the resource to which we want to write data (in this case, the greeting. TXT file).
The FlatFileItemWriter needs to know how to convert the generated output into a single string that can be written to a file. In this case, our output is a string, we can use PassThroughLineAggregator. This is the most basic implementation and assumes that the object is already a string.
@Configuration
public class HelloWorldJobConfig {
@Bean
public Job helloWorlJob(JobBuilderFactory jobBuilders, StepBuilderFactory stepBuilders) {
return jobBuilders.get("helloWorldJob")
.start(helloWorldStep(stepBuilders)).build();
}
@Bean
public Step helloWorldStep(StepBuilderFactory stepBuilders) {
return stepBuilders.get("helloWorldStep")
.<Person, String>chunk(10).reader(reader())
.processor(processor()).writer(writer()).build();
}
@Bean
public FlatFileItemReader<Person> reader() {
return new FlatFileItemReaderBuilder<Person>()
.name("personItemReader")
.resource(new ClassPathResource("csv/persons.csv"))
.delimited().names(new String[] {"firstName"."lastName"})
.targetType(Person.class).build();
}
@Bean
public PersonItemProcessor processor() {
return new PersonItemProcessor();
}
@Bean
public FlatFileItemWriter<String> writer() {
return new FlatFileItemWriterBuilder<String>()
.name("greetingItemWriter")
.resource(new FileSystemResource(
"target/test-outputs/greetings.txt"))
.lineAggregator(newPassThroughLineAggregator<>()).build(); }}Copy the code7. Crunch the data
In most cases, you will want to apply some data processing during batch jobs. You can do this using ItemProcessor.
In our example, we turn the Person object into a simple greeting String
To do this, we create a PersonItemProcessor that implements the ItemProcessor interface. We implement the process() method, which adds the first and last name to the string.
During debugging, we log the results.
public class PersonItemProcessor
implements ItemProcessor<Person.String> {
private static final Logger LOGGER =
LoggerFactory.getLogger(PersonItemProcessor.class);
@Override
public String process(Person person) throws Exception {
String greeting = "Hello " + person.getFirstName() + ""
+ person.getLastName() + "!";
LOGGER.info("converting '{}' into '{}'", person, greeting);
returngreeting; }}Copy the code8. Test the Spring Batch sample
To test this example, we created a basic unit test case. It runs the batch job and checks for successful completion.
We use the @runwith and @springboottest test annotations to tell JUnit to RunWith Spring’s test support and boot with SpringBoot’s support.
Spring Batch comes with a JobLauncherTestUtils utility class for testing Batch jobs.
We start by creating an internal BatchTestConfig class to add the helloWorld job to the JobLauncherTestUtils bean. The batch job is then run using the bean’s launchJob() method.
If the job is executed without any errors, the value of ExitCode is COMPLETED.
@RunWith(SpringRunner.class)
@SpringBootTest(
classes = {SpringBatchApplicationTests.BatchTestConfig.class})
public class SpringBatchApplicationTests {
@Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
@Test
public void testHelloWorldJob() throws Exception {
JobExecution jobExecution = jobLauncherTestUtils.launchJob();
assertThat(jobExecution.getExitStatus().getExitCode())
.isEqualTo("COMPLETED");
}
@Configuration
@Import({BatchConfig.class, HelloWorldJobConfig.class})
static class BatchTestConfig {
@Autowired
private Job helloWorlJob;
@Bean
JobLauncherTestUtils jobLauncherTestUtils()
throws NoSuchJobException {
JobLauncherTestUtils jobLauncherTestUtils =
new JobLauncherTestUtils();
jobLauncherTestUtils.setJob(helloWorlJob);
returnjobLauncherTestUtils; }}}Copy the codeTo trigger the above test cases, open a command prompt in the project root folder and execute the following Maven commands:
mvn test
Copy the codeThe result is a successful build and a batch job is executed in the meantime.
____ ____ _ /\\ / ___'_ __ _ _) (_ _ __ __ _ \ \ \ \ (\ ___ () |'_ | '_| | '_ \ / _` | \ \ \ \ \ \ / ___) | | _) | | | | | | | (_ | |)))) 'there comes | |. __ _ - | | | _ _ - | | | _ \ __, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.1.5. RELEASE) the 2019-05-30 19:11:12. 14588-784 the INFO [main] C.C.S pringBatchApplicationTests: Starting SpringBatchApplicationTests on DESKTOP-2RB3C1U with PID 14588 (started by Codenotfound in C:\Users\Codenotfound\repos\spring-batch\spring-batch-hello-world) 2019-05-30 19:11:12.785 INFO 14588 - [main] c.c.SpringBatchApplicationTests : No active profile set, falling back to default profiles: Default 19:11:13 2019-05-30. 305 WARN 14588 - [the main] O.S.B.C.C.A.D efaultBatchConfigurer: No datasource was provided... Using a Map -based JobRepository 19:11:13 2019-05-30. 306 WARN 14588 - [the main] O.S.B.C.C.A.D efaultBatchConfigurer: No transaction manager was provided, Using a ResourcelessTransactionManager 19:11:13 2019-05-30. 14588-328 the INFO [main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, Defaulting to synchronous executor. 19:11:13 2019-05-30. 14588-350 the INFO [main] C.C.S pringBatchApplicationTests: Started SpringBatchApplicationTests in 0.894 seconds (JVM running 1.777) for the 2019-05-30 19:11:13. 14588-732 the INFO [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=helloWorldJob]] launched with the following parameters: [459672} {the random =] 2019-05-30 19:11:13. 14588-759 the INFO [main] O.S.B atch. Core. Job. SimpleStepHandler: Executing step: [helloWorldStep] 2019-05-30 19:11:13. 812 INFO - 14588 [main] C.C.B atch. PersonItemProcessor: converting 'John Doe' into 'Hello John Doe! '14588-2019-05-30 19:11:13. 822 INFO [main] C.C.B atch. PersonItemProcessor: converting 'Jane Doe' into 'Hello Jane Doe! '14588-2019-05-30 19:11:13. 842 INFO [main] O.S.B.C.L.S upport. SimpleJobLauncher: Job: [SimpleJob: [name=helloWorldJob]] completed with the following parameters: [{random=459672}] and the following status: [COMPLETED] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: S - 1.953 com in. Codenotfound. SpringBatchApplicationTests [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, the Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- [INFO] Total time: 6.852 s [INFO] Finished at: 2019-05-30T19:11:14+02:00 [INFO] ------------------------------------------------------------------------Copy the codeYou can find the result in the target/test-output/greeting. TXT file:
Hello John Doe!
Hello Jane Doe!
Copy the codeIf you want to run the code sample above, you can get the full source code here.