In a word: use dynamic data sources to operate multiple databases, flexible and concise.

1. The introduction

For the processing of multiple databases, the last article “Get SpringBoot multi-source (1) : multiple sets of source strategy” has been mentioned, there are multiple sets of data sources, dynamic data sources, parameterized change data sources and other ways, this article is the second: “dynamic data sources”. Dynamic data sources can solve the problems of inflexible processing of multiple data sources and occupying too many resources. Users can unify the operation logic based on actual service requirements, as long as the data source needs to be switched for processing. What is dynamic is that the timing of batch switching data sources can be dynamically selected and switched where necessary.

This article continues the example of the last article, with the master-slave scenario as an example, combined with the code, to explain the implementation of dynamic data source, including the principle of building dynamic data source, dynamic data source configuration, dynamic data source use, AOP annotation switch data source, etc..

This paper involved the sample code: https://github.com/mianshenglee/my-example/tree/master/multi-datasource, readers can combine together.

2. Dynamic data source process description

Spring Boot’s dynamic data source essentially stores multiple data sources in a Map. When a data source needs to be used, the Map obtains the data source for processing. In the Spring, has provided an abstract class AbstractRoutingDataSource to implement this function. Therefore, we in the implementation of dynamic data sources, just need to inherit it, to achieve their own access to the data source logic. The dynamic data source flow is as follows:

When accessing applications, users can access different data sources based on the routing logic of data sources and perform operations on corresponding data sources. The two databases in this example have a table test_user, which has the same structure. For the sake of illustration, the data in the two tables are different. The two table structures are available in the SQL directory in the sample code.

3. Dynamic data sources

3.1 Description and Data Source Configuration

3.1.1 Package Structure description

In this example, there are the following packages:

- ├ ─ an annotation// Custom annotations├ ─ aop -- -- -- -- -- -- -- -- -- -- -/ / section├ ─ config -- -- -- -- -- -- -- --// Data source configuration├ ─ constants -- -- -- -- --// Common annotations├ ─ the context -- -- -- -- -- -- --// Customize the context
├─controller ---- // Access the interface├ ─ the entity -- -- -- -- -- -- -- --/ / entity├ ─ mapper -- -- -- -- -- -- -- --// Database DAO operations├ ─ service -- -- -- -- -- -- --/ / service class└ ─ vo -- -- -- -- -- -- -- -- -- -- -- --// The view returns data
Copy the code

3.1.2 Configuring database Connection Information

The default configuration file for Spring Boot is application.properties. Since there are two database configurations, it is good practice to configure the database independently, so add the configuration file jbdc.properties to add the following custom master/slave database configurations:

# master spring.datasource.master.driver-class-name=com.mysql.cj.jdbc.Driver spring.datasource.master.jdbc-url=jdbc:mysql://localhost:3306/mytest? useSSL=false&serverTimezone=GMT%2B8&characterEncoding=UTF-8 spring.datasource.master.username=root spring.datasource.master.password=111111 # slave spring.datasource.slave.driver-class-name=com.mysql.cj.jdbc.Driver spring.datasource.slave.jdbc-url=jdbc:mysql://localhost:3306/my_test1? useSSL=false&serverTimezone=GMT%2B8&characterEncoding=UTF-8 spring.datasource.slave.username=root spring.datasource.slave.password=111111Copy the code

3.1.3 Data source Configuration

Add the DynamicDataSourceConfig file to Spring based on the connection information:

@Configuration
@PropertySource("classpath:config/jdbc.properties")
@MapperScan(basePackages = "me.mason.demo.dynamicdatasource.mapper")
public class DynamicDataSourceConfig {
    @Bean(DataSourceConstants.DS_KEY_MASTER)
    @ConfigurationProperties(prefix = "spring.datasource.master")
    public DataSource masterDataSource(a) {
        return DataSourceBuilder.create().build();
    }

    @Bean(DataSourceConstants.DS_KEY_SLAVE)
    @ConfigurationProperties(prefix = "spring.datasource.slave")
    public DataSource slaveDataSource(a) {
        returnDataSourceBuilder.create().build(); }}Copy the code

Note:

Used herePropertySourceSpecify the configuration file,ConfigurationPropertiesSpecify the data source configuration prefix

useMapperScanSpecify the package to automatically inject the corresponding Mapper class.

Write the data source constant atDataSourceConstantsIn the class

As you can see from this configuration, the SqlSessionFactory configuration has been erased from the code and you can use the SqlSessionFactory configuration automatically configured by Spring Boot.

3.2 Dynamic data source Settings

The previous configuration injected multiple data sources into Spring and then configured the dynamic data sources.

3.2.1 Dynamic Data Source Configuration

(1) Add JDBC dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
Copy the code

(2) Add dynamic data source class

public class DynamicDataSource extends AbstractRoutingDataSource {
    @Override
    protected Object determineCurrentLookupKey(a) {
        // Return the fixed master data source for the time being
        returnDataSourceConstants.DS_KEY_MASTER; }}Copy the code

Note:

Inheriting abstract classesAbstractRoutingDataSourceImplementation methods are requireddetermineCurrentLookupKeyIs the routing policy.

The dynamic routing policy is implemented next. The current policy returns the master data source directly

(3) Set dynamic data source as main data source

In the previous data source configuration file DynamicDataSourceConfig, add the following code:

@Bean
@Primary
public DataSource dynamicDataSource(a) {
    Map<Object, Object> dataSourceMap = new HashMap<>(2);
    dataSourceMap.put(DataSourceConstants.DS_KEY_MASTER, masterDataSource());
    dataSourceMap.put(DataSourceConstants.DS_KEY_SLAVE, slaveDataSource());
    // Set the dynamic data source
    DynamicDataSource dynamicDataSource = new DynamicDataSource();
    dynamicDataSource.setTargetDataSources(dataSourceMap);
    dynamicDataSource.setDefaultTargetDataSource(masterDataSource());

    return dynamicDataSource;
}
Copy the code

Use Map to save multiple data sources and set them up in a dynamic data source object.

Set the default data source to be master data source

Using annotationsPrimaryPreferentially obtained from dynamic data sources

At the same time, need in DynamicDataSourceConfig, eliminate the automatic configuration of DataSourceAutoConfiguration The dependencies of some of The beans in The application context form a cycle

@EnableAutoConfiguration(exclude = { DataSourceAutoConfiguration.class })
Copy the code

3.2.2 Dynamic Data source selection

** (1) Context of the data source key **

We fixed a data source routing policy that always returns master, which is obviously not what we want. We want to be where we want to be, and we want to switch. Therefore, there needs to be a place to dynamically obtain the data source key (we call it a context). For Web applications, where access is thread-based, ThreadLocal is appropriate, as follows:

public class DynamicDataSourceContextHolder {
    /** * Dynamic data source name context */
    private static final ThreadLocal<String> DATASOURCE_CONTEXT_KEY_HOLDER = new ThreadLocal<>();
    /** * Set/switch data source */
    public static void setContextKey(String key){
        DATASOURCE_CONTEXT_KEY_HOLDER.set(key);
    }
    /** * get the data source name */
    public static String getContextKey(a){
        String key = DATASOURCE_CONTEXT_KEY_HOLDER.get();
        return key == null? DataSourceConstants.DS_KEY_MASTER:key; }/** * Delete the current data source name */
    public static void removeContextKey(a){
        DATASOURCE_CONTEXT_KEY_HOLDER.remove();
    }
Copy the code

Storing DATASOURCE_CONTEXT_KEY_HOLDER requires a data source key

When getContextKey is null, master is returned by default

(2) Set dynamic dataDynamicDataSourceRouting strategy

The routing policy we need to achieve is that when we set the data source key to the context, we get the data source key from the context and know which data source to use. Therefore, modify the previous DynamicDataSource determineCurrentLookupKey method is as follows:

@Override
protected Object determineCurrentLookupKey(a) {
    return DynamicDataSourceContextHolder.getContextKey();
}
Copy the code

3.2.3 Use of dynamic data sources

With the above dynamic routing selection, there is no need to write the same logic code for Mapper, Entity, service, etc., as the previous multiple sets of data sources. Because the database structure is generally consistent, only one set of Entity, Mapper, service is needed. When you need to operate on different data sources, you can simply set the context. As follows:

@RestController
@RequestMapping("/user")
public class TestUserController {

    @Autowired
    private TestUserMapper testUserMapper;

    /** * query all */
    @GetMapping("/listall")
    public Object listAll(a) {
        int initSize = 2;
        Map<String, Object> result = new HashMap<>(initSize);
        // Default master query
        QueryWrapper<TestUser> queryWrapper = new QueryWrapper<>();
        List<TestUser> resultData = testUserMapper.selectAll(queryWrapper.isNotNull("name"));
        result.put(DataSourceConstants.DS_KEY_MASTER, resultData);

        // Switch data source to slave
        DynamicDataSourceContextHolder.setContextKey(DataSourceConstants.DS_KEY_SLAVE);
        List<TestUser> resultDataSlave = testUserMapper.selectList(null);
        result.put(DataSourceConstants.DS_KEY_SLAVE, resultDataSlave);
        // Restore the data source
        DynamicDataSourceContextHolder.removeContextKey();
        // Return data
        returnResponseResult.success(result); }}Copy the code

The default is to use the master data source query

Use the setContextKey of the context to switch the data source, and then use the removeContextKey to restore the data source

3.3 Use AOP to select data sources

After the above dynamic data source configuration, dynamic data source switch can be implemented, but we will find that when switching data source, setContextKey and removeContextKey operations need to be done, if there are more methods to switch, we will find a lot of repeated code, how to eliminate these repeated code, If you do not understand dynamic proxies, you can refer to my article “Java Development must Learn: Dynamic proxies”. In Spring, AOP implementations are also based on dynamic proxies. Here, we want to annotate the data source required by the function to eliminate the template code of the product when the data source is switched.

3.3.1 Define data source annotations

In the Annotation package, add the data source annotation DS, which can be written in the class or in the method definition, as follows:

@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface DS {
    /** * Data source name */
    String value(a) default DataSourceConstants.DS_KEY_MASTER;
}
Copy the code

3.3.2 Defining data source facets

Defines a data source aspect that can be used to switch data sources for methods or classes that use DS annotations.

(1) Add AOP dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>
Copy the code

(2) Define the section

@Aspect
@Component
public class DynamicDataSourceAspect {
    @Pointcut("@annotation(me.mason.demo.dynamicdatasource.annotation.DS)")
    public void dataSourcePointCut(a){}@Around("dataSourcePointCut()")
    public Object around(ProceedingJoinPoint joinPoint) throws Throwable {
        String dsKey = getDSAnnotation(joinPoint).value();
        DynamicDataSourceContextHolder.setContextKey(dsKey);
        try{
            return joinPoint.proceed();
        }finally{ DynamicDataSourceContextHolder.removeContextKey(); }}/** * Get data source annotations by class or method */
    private DS getDSAnnotation(ProceedingJoinPoint joinPoint){ Class<? > targetClass = joinPoint.getTarget().getClass(); DS dsAnnotation = targetClass.getAnnotation(DS.class);// Judge the class annotations first, and then the method annotations
        if(Objects.nonNull(dsAnnotation)){
            return dsAnnotation;
        }else{
            MethodSignature methodSignature = (MethodSignature)joinPoint.getSignature();
            returnmethodSignature.getMethod().getAnnotation(DS.class); }}}Copy the code

Annotations Pointcut useannotationSpecify the annotation

Annotations Around use the surround notification processing, and use the context for annotationsDSAfter processing, the data source is restored.

3.3.3 Use AOP to switch data sources

In the service layer, we define a TestUserService with two methods, one from master and the other from slave, using the DS annotation as follows:

/** * query User */
@DS(DataSourceConstants.DS_KEY_MASTER)
public List<TestUser> getMasterUser(a){
    QueryWrapper<TestUser> queryWrapper = new QueryWrapper<>();
    return testUserMapper.selectAll(queryWrapper.isNotNull("name"));
}

/** * query slave library User */
@DS(DataSourceConstants.DS_KEY_SLAVE)
public List<TestUser> getSlaveUser(a){ return testUserMapper.selectList(null); }
Copy the code

After this definition, the processing at the Controller layer can become:

@GetMapping("/listall")
public Object listAll(a) {
	int initSize = 2;
	Map<String, Object> result = new HashMap<>(initSize);
	// Default master data source query
	List<TestUser> masterUser = testUserService.getMasterUser();
	result.put(DataSourceConstants.DS_KEY_MASTER, masterUser);
	// Query from the slave data source
	List<TestUser> slaveUser = testUserService.getSlaveUser();
	result.put(DataSourceConstants.DS_KEY_SLAVE, slaveUser);
	// Return data
	return ResponseResult.success(result);
}
Copy the code

As you can see, the template code for database switching has been eliminated and you only need to focus on business logic processing. This is the benefit of AOP.

4. Think again

From the dynamic data sources and AOP’s choice of data sources explained above, we can see that dynamic data sources are already flexible enough to switch simply by setting the data source in the context, or by using annotations directly in a method or class. Now we are manually coded to achieve, in fact, for MyBatis Plus, it also provides a dynamic data source plug-in, interested partners can also according to its official documents for experimental use.

What else could be considered or improved for dynamic data sources?

How are transactions handled? Cross-library transactions should be avoided as much as possible in development, but if you can’t avoid them, you need to use distributed transactions.
For the current dynamic data sources, relatively fixed data sources (such as master one slave, one master many slave, etc.), that is, the number of databases has been determined at the time of coding, and it is only processed dynamically when the specific use of which one. What if the data source itself is uncertain, or you need to connect to the database based on user input? This happens more often when managing multiple databases. This situation, which I’ll cover in the next article, is called “parameterized change data sources.”

5. To summarize

This paper explains the implementation of dynamic data sources, mainly the configuration, implementation, use of dynamic data sources, and also use AOP to eliminate the template code when switching data sources, so that we focus on business code development, and finally the dynamic data sources of the expansion of thinking. I hope you can master the processing of dynamic data sources.

This article supporting examples, sample code, are interested in running examples to feel.

The resources

Spring master/slave database configuration and dynamic data source switching principle: https://www.liaoxuefeng.com/article/1182502273240832
Trade-offs between multiple and dynamic data sources: https://juejin.cn/post/6844903661655572494
Talk about Spring Boot data source loading and its simple implementation of multiple data sources: https://juejin.cn/post/6844903817260056589
Spring Boot and MyBatis realize multi-data source and dynamic data source switching: https://juejin.cn/post/6844903566717485069

The articles

SpringBoot Multiple Data Sources (1) : Multiple source policies
Java development must learn: dynamic proxy
Good books to read in 2019
Springboot + Apache HTTPS is deployed on the front and back ends
Springboot + Logback Log Output
Springboot + Logback Log output

My official account (search Mason technical records) for more technical records:

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

SpringBoot Multiple Data Sources (2) : dynamic data sources

1. The introduction

2. Dynamic data source process description

3. Dynamic data sources

3.1 Description and Data Source Configuration

3.1.1 Package Structure description

3.1.2 Configuring database Connection Information

3.1.3 Data source Configuration

3.2 Dynamic data source Settings

3.2.1 Dynamic Data Source Configuration

3.2.2 Dynamic Data source selection

3.2.3 Use of dynamic data sources

3.3 Use AOP to select data sources

3.3.1 Define data source annotations

3.3.2 Defining data source facets

3.3.3 Use AOP to switch data sources

4. Think again

5. To summarize

The resources

The articles

SpringBoot Multiple Data Sources (2) : dynamic data sources

1. The introduction

2. Dynamic data source process description

3. Dynamic data sources

3.1 Description and Data Source Configuration

3.1.1 Package Structure description

3.1.2 Configuring database Connection Information

3.1.3 Data source Configuration

3.2 Dynamic data source Settings

3.2.1 Dynamic Data Source Configuration

3.2.2 Dynamic Data source selection

3.2.3 Use of dynamic data sources

3.3 Use AOP to select data sources

3.3.1 Define data source annotations

3.3.2 Defining data source facets

3.3.3 Use AOP to switch data sources

4. Think again

5. To summarize

The resources

The articles

Related Posts

Write your own distributed stream limiting component – Ratelimiter based on Redis annotation support

When do we need soft deletions?

SpringBoot interview Killer – Automatic configuration principle