tags: springbatch mongodb

1. The introduction

Spring Batch’s insta-by-instance approach has been introduced previously. If you are interested, see the following articles:

  • Data Batch artifact -Spring Batch(1) Introduction and usage scenarios
  • Quick introduction to component – Spring Batch (2) helloWorld
  • Quick use of the component – Spring Batch (3) reads file data to the database
  • Decisive Database – Spring Batch (4) Database to database
  • Convenient data read and write – Spring Batch (5) combines beetlSql for data read and write
  • Incremental Synchronization – Spring Batch (6) Dynamic parameter binding with incremental synchronization
  • Scheduling and monitoring – Spring Batch (7) Combines XXL-job to perform batch processing

In addition to data synchronization for files and relational databases, Spring Batch supports rich data types for reading components (ItemReader), processing components (ItemProcessor), and writing components (ItemWriter). MongoItemReader and MongoItemWriter are read-write components for Mongo, and users can use them directly to carry out data read-write operations of Mongodb. A common scenario is to synchronize data from a relational database (such as mysql) to mongodb. The following example is used to explain the data synchronization between mysql and mongodb. This article mainly explains the operation of Mongodb. For Spring Batch to read relational database data using BeetlSQL, please refer to the article “Convenient Data Read-write – Spring Batch (5) To read and write data with BeetlSQL”. See the Github Sample repository for sample code for this article.

2. Development environment

  • The JDK: jdk1.8
  • Spring the Boot: 2.1.4. RELEASE
  • Spring Batch: 4.1.2. RELEASE
  • Development IDE: the IDEA
  • Build tool Maven: 3.3.9
  • Log component Logback :1.2.3
  • Lombok: 1.18.6
  • MySQL: 5.6.26
  • Mongo: 4.0.10

3. Development process

3.1 Sample database and target database

The flow of this example is as follows:

The SQL directory in the sample project has the corresponding relational database script, and the mytest.sql script creates a test_USER table with the corresponding test data. Mongodb installation can see the official document, set up the corresponding storage data Collection, this example is mytest.

3.2 addmavenDependencies and ConfigurationmongodbConnection address

Because you need to use mongodb’s operations, you need to add its dependencies. As follows:

<! -- mongodb -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Copy the code

After the dependency is added, the mongodb connection address must be configured in the configuration file. If the user name and password are used, the connection address must be configured as well. As follows:

Spring. The data. The mongo. Uri = mongo: / / 192.168.222.10 / mytest # spring data. The mongo. Username = # spring. The data. The mongo. Password =Copy the code

3.3 writemongodbRead and write components of

As shown in the example, there are three components in total, including a component to read mysql database, a processing component to convert mysql database entity to mongodb, and a write component to write mongodb. The code structure is shown in the following figure:

ItemReader component and ItemProcessor component need not be discussed. Please refer to the previous article. Here we mainly talk about mongodb ItemWriter, and this writing component can only use MongoItemWriter to compile its own logic. Mongodb write operations provided by Spring Batch are introduced via MongoOperations when initializing ItemWriter. Therefore, MongoBatchConfig file adds the following code:

@Bean
public ItemWriter mongoWriter(MongoOperations mongoTemplate) {
    UserItemWriter userItemWriter = new UserItemWriter();
    userItemWriter.setTemplate(mongoTemplate);
    userItemWriter.setCollection("user");
    return userItemWriter;
}
Copy the code

MongoOperations is injected at initialization. In the custom UserItemWriter, set template and collection. If the logic is simple, don’t write custom ItemWriter, also can be used directly MongoItemWriterBuilder, constructing MongoItemWriter directly, as shown below:

return new MongoItemWriterBuilder<MongoUser>()
                .collection("user")
                .template(mongoTemplate)
                .build();
Copy the code

The above is the build of the write component. Similarly, the build of the read component of mongodb is similar, except that attention should be paid to the configuration of dynamic parameters. The following example code queries the data and returns the map, and the parameters are dynamically passed in during the build task.

@Bean
@StepScope
public MongoItemReader<Map> tweetsItemReader(MongoOperations mongoTemplate,@Value("#{jobParameters['hashTag']}") String hashtag) {
return new MongoItemReaderBuilder<Map>()
    .name("tweetsItemReader")
    .targetType(Map.class)
    .jsonQuery("{ \"entities.hashtags.text\": { $eq: ?0 }}")
    .collection("tweets_collection")
    .parameterValues(Collections.singletonList(hashtag))
    .pageSize(10)
    .sorts(Collections.singletonMap("created_at", Sort.Direction.ASC))
    .template(mongoTemplate)
    .build();
}
Copy the code

4. Execution result

Write the unit test or write the startup task in Controller to perform the data synchronization test, and the execution result is as follows:

5. To summarize

In this paper, data is synchronized from mysql to mongodb based on Spring Batch. By combining with sample code, read and write components of mongodb are written and configured. I hope that those who need to use Spring Batch for relational database and mongodb for Batch task development can help.