With the expansion of data scale, the distributed mode of using multi-node cluster gradually becomes a trend. In this case, how to efficiently and automatically manage cluster nodes, realize the collaborative work of different nodes, configuration consistency, state consistency, high availability, observability, etc., has become an important challenge.

The complexity of cluster management is reflected in that, on the one hand, we need to manage the status of all nodes, no matter the underlying database node, middleware or business system node, and can detect the latest configuration changes in real time, which further provides the basis for cluster regulation and scheduling.

On the other hand, the unified coordination between different nodes, the strategy of database and table and the synchronization of rules also require us to design a set of distributed coordination lock mechanism for global event notification mechanism and exclusive operation under the distributed condition. In this regard, ShardingJDBC uses Zookeeper/Etcd for configuration synchronization, status change notification, and distributed locks to control exclusive operations.

ShardingJDBC distributed governance

ShardingJDBC integrated Zookeeper/Etcd, used to achieve ShardingJDBC distributed governance, let’s first through an application to demonstrate the implementation principle.

Install the Zookeeper

Download Zookeeper from this address

Mirrors.tuna.tsinghua.edu.cn/apache/zook…

Common Operation Commands

#Start the ZK service:      
bin/zkServer.sh start
#View ZK service status: 
bin/zkServer.sh status
#Stop ZK service:    
bin/zkServer.sh stop
#Restart the ZK service:    
bin/zkServer.sh restart
#Connecting to the server
zkCli.sh -timeout 0 -r -server ip:port
Copy the code

Single machine installation

When using ZooKeeper for the first time, copy the zoo_sample. CFG file in the conf directory and rename it zoo.cfg

Modify the dataDir directory, which indicates the directory where the log files are stored (additional configuration information about zoo.cfg will be covered later)

Sharding – JDBC integrated Zookeeper

The project code demonstrated in this stage is sharding-jdbC-split-ZooKeeper, and the project structure is shown in Figure 9-1.

Add jar package dependencies

Introduce JAR package dependencies (only need to rely on the following two packages)

<dependency>
    <groupId>org.apache.shardingsphere</groupId>
    <artifactId>shardingsphere-governance-repository-zookeeper-curator</artifactId>
    <version>5.0.0 - alpha</version>
</dependency>
<dependency>
    <groupId>org.apache.shardingsphere</groupId>
    <artifactId>shardingsphere-jdbc-governance-spring-boot-starter</artifactId>
    <version>5.0.0 - alpha</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.shardingsphere</groupId>
            <artifactId>shardingsphere-test</artifactId>
        </exclusion>
    </exclusions>
</dependency>
Copy the code

Other base JAR packages (all projects are copied based on Spring Boot integration mybatis)

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>com.baomidou</groupId>
    <artifactId>mybatis-plus-boot-starter</artifactId>
    <version>Rule 3.4.3</version>
</dependency>
<dependency>
    <groupId>com.baomidou</groupId>
    <artifactId>mybatis-plus-generator</artifactId>
    <version>3.4.1 track</version>
</dependency>
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.72</version>
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.9</version>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.12</version>
</dependency>
Copy the code

Adding a Configuration File

Added basic database table configuration -application.properties

spring.shardingsphere.datasource.names=ds-0,ds-1
spring.shardingsphere.datasource.common.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.common.driver-class-name=com.mysql.jdbc.Driver

spring.shardingsphere.datasource.ds-0.username=root
spring.shardingsphere.datasource.ds-0.password=123456
spring.shardingsphere.datasource.ds-0.jdbc-url=JDBC: mysql: / / 192.168.221.128:3306 / shard01? serverTimezone=UTC&useSSL=false&useUnicode=true&characterEncoding=UTF-8

spring.shardingsphere.datasource.ds-1.username=root
spring.shardingsphere.datasource.ds-1.password=123456
spring.shardingsphere.datasource.ds-1.jdbc-url=JDBC: mysql: / / 192.168.221.128:3306 / shard02? serverTimezone=UTC&useSSL=false&useUnicode=true&characterEncoding=UTF-8

spring.shardingsphere.rules.sharding.default-database-strategy.standard.sharding-column=user_id
spring.shardingsphere.rules.sharding.default-database-strategy.standard.sharding-algorithm-name=database-inline

spring.shardingsphere.rules.sharding.tables.t_order.actual-data-nodes=ds-$->{0.. 1}.t_order_$->{0.. 1}
spring.shardingsphere.rules.sharding.tables.t_order.table-strategy.standard.sharding-column=order_id
spring.shardingsphere.rules.sharding.tables.t_order.table-strategy.standard.sharding-algorithm-name=t-order-inline

spring.shardingsphere.rules.sharding.tables.t_order.key-generate-strategy.column=order_id
spring.shardingsphere.rules.sharding.tables.t_order.key-generate-strategy.key-generator-name=snowflake

spring.shardingsphere.rules.sharding.sharding-algorithms.database-inline.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.database-inline.props.algorithm-expression=ds-$->{user_id % 2}
spring.shardingsphere.rules.sharding.sharding-algorithms.t-order-inline.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.t-order-inline.props.algorithm-expression=t_order_$->{order_id % 2}
spring.shardingsphere.rules.sharding.sharding-algorithms.t-order-item-inline.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.t-order-item-inline.props.algorithm-expression=t_order_item_$->{order_id % 2}

spring.shardingsphere.rules.sharding.key-generators.snowflake.type=SNOWFLAKE
spring.shardingsphere.rules.sharding.key-generators.snowflake.props.worker-id=123
Copy the code

Added Zookeeper configuration

# admin name (node name on ZooKeeper)
spring.shardingsphere.governance.name=sharding-jdbc-split-zookeeper
Whether local configuration overrides configuration center configuration. If it can be overridden, the local configuration prevails during each startup.
spring.shardingsphere.governance.overwrite=true

# Zookeeper/etcd
spring.shardingsphere.governance.registry-center.type=ZooKeeper
spring.shardingsphere.governance.registry-center.server-lists=192.168.221.131:2181
The default connection timeout time is 1500 milliseconds, so it is easy to time out during startup, resulting in connection failure
# retries
spring.shardingsphere.governance.registry-center.props.maxRetries=4
Retry interval
spring.shardingsphere.governance.registry-center.props.retryIntervalMilliseconds=6000
Copy the code

Start the project for testing

If the following log is displayed during the startup, zooKeeper is successfully configured. The local configuration is saved to ZooKeeper during the startup. You can modify related configurations in ZooKeeper later, and then notify related application nodes.

The 2021-07-29 21:31:25. 112916-007 the INFO [main] org. Apache. They are. They are: Client environment: Java. IO. Tmpdir = C: \ Users \ mayn \ AppData \ Local 21:31:25 \ Temp \ 2021-07-29. 112916-007 the INFO [main] org.apache.zookeeper.ZooKeeper : Client environment:java.com piler = < NA > 21:31:25 2021-07-29. 112916-007 the INFO [main] org. Apache. They are. They are: Client environment: the OS name = 10 2021-07-29 21:31:25 Windows. The 112916-007 the INFO [main] org. Apache. They are. They are: Client environment: OS. The arch = amd64 21:31:25 2021-07-29. 112916-007 the INFO [main] org. Apache. They are. They are: The Client environment: OS version = 10.0 2021-07-29 21:31:25. 112916-007 the INFO [main] org. Apache. They are. They are: Client environment: the user name = mayn 21:31:25 2021-07-29. 112916-007 the INFO [main] org. Apache. They are. They are: Client Environment :user.home=C: Users\mayn 2021-07-29 21:31:25.007 INFO 112916 -- [main] org.apache.zookeeper.ZooKeeper : Client environment:user.dir=E: Teaching - Courseware VIP Course Round 5 03 High Concurrency component 09 ShardingSphere Implements Distributed Governance based on Zookeeper \ Sharding-jdbC-Readwrite-ZooKeeper 2021-07-29 21:31:25.007 INFO 112916 -- [main] org.apache.zookeeper.ZooKeeper : Client environment: OS. Memory. Free = 482 MB 21:31:25 2021-07-29. 112916-007 the INFO [main] org. Apache. They are. They are: Client environment: OS. Memory. Max = 7264 MB 21:31:25 2021-07-29. 112916-007 the INFO [main] org. Apache. The zookeeper. The zookeeper : Client environment: OS. Memory. Total = 501 MB 21:31:25 2021-07-29. 112916-009 the INFO [main] org. Apache. The zookeeper. The zookeeper : Initiating client connection, ConnectString = 192.168.221.131:2181 60000 watcher sessionTimeout = = org. Apache. The curator. 68 e2d03e ConnectionState @ 2021-07-29 21:31:25. 112916-012 the INFO [main] org.apache.zookeeper.com mon. X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2021-07-29 21:31:25. 112916-020 the INFO [main] org. Apache. The zookeeper. ClientCnxnSocket: Jute. The value is 1048575 Bytes maxbuffer 21:31:25 2021-07-29. 112916-023 the INFO [main] org. Apache. The zookeeper. ClientCnxn: Zookeeper. Request. The timeout value is 0. Feature enabled = false 21:31:25 2021-07-29. 112916-030 the INFO [main] o.a.c.f.imps.CuratorFrameworkImpl : Default schemaCopy the code

Then access the following interface to test.

@RestController
@RequestMapping("/t-order")
public class TOrderController {

    @Autowired
    ITOrderService orderService;

    @GetMapping
    public void init(a) throws SQLException { orderService.initEnvironment(); orderService.processSuccess(); }}Copy the code

Configuration center data structure description

The data structure of the registry is as follows

Namespace: it’s spring. The shardingsphere. Governance. Name

# namespace ├ ─ ─ the users permission configuration ├ ─ ─ props # attribute configuration ├ ─ ─ schemas # Schema configuration ├ ├ ─ ─ ${schema_1} Schema name # 1 ├ ├ ├ ─ ─ a datasource # data source configuration Rule # rule configuration ├ ├ ├ ─ ─ ├ ├ ├ ─ ─ table # table structure configuration ├ ├ ─ ─ ${schema_2} # Schema name 2 ├ ├ ├ ─ ─ a datasource # data source configuration ├ ├ ├ ─ ─ rule # rule configuration ├ ├ ├─ Table # Table Structure ConfigurationCopy the code

Rules Global configuration rules

It can include the permission configuration for accessing the shardingsphere-proxy user name and password

- ! AUTHORITYusers:  - root@%:root  - [email protected]: shardingprovider:  type: NATIVE
Copy the code

Props Property Configuration

Shardingsphere-related attributes are configured

executor-size: 20sql-show: true
Copy the code

/schemas/${schemeName}/dataSources

A collection of multiple database connection pools with different database connection pool attributes (for example, DBCP, C3P0, Druid, HikariCP).

ds_0: dataSourceClassName: com.zaxxer.hikari.HikariDataSource props: url: JDBC: mysql: / / 127.0.0.1:3306 / demo_ds_0? serverTimezone=UTC&useSSL=false    password: null maxPoolSize: 50 connectionTimeoutMilliseconds: 30000 idleTimeoutMilliseconds: 60000 minPoolSize: 1 username: root maxLifetimeMilliseconds: 1800000ds_1: dataSourceClassName: com.zaxxer.hikari.HikariDataSource props: url: JDBC: mysql: / / 127.0.0.1:3306 / demo_ds_1? serverTimezone=UTC&useSSL=false    password: null maxPoolSize: 50 connectionTimeoutMilliseconds: 30000 idleTimeoutMilliseconds: 60000 minPoolSize: 1 username: root maxLifetimeMilliseconds: 1800000
Copy the code

/schemas/${schemeName}/rule

Rules can be configured for data fragmentation and read/write separation

rules:- ! SHARDING  defaultDatabaseStrategy: standard: shardingAlgorithmName: database-inline shardingColumn: user_id keyGenerators: snowflake: props: worker-id: '123'      type: SNOWFLAKE shardingAlgorithms: t-order-inline: props: algorithm-expression: t_order_$->{order_id % 2}      type: INLINE database-inline: props: algorithm-expression: ds-$->{user_id % 2}      type: INLINE t-order-item-inline: props: algorithm-expression: t_order_item_$->{order_id % 2}      type: INLINE tables: t_order: actualDataNodes: ds-$->{0.. 1}.t_order_$->{0.. 1}      keyGenerateStrategy: column: order_id keyGeneratorName: snowflake logicTable: t_order tableStrategy: standard: shardingAlgorithmName: t-order-inline shardingColumn: order_id
Copy the code

/schemas/${schemeName}/table

Table structure configuration, do not support dynamic change

configuredSchemaMetaData: tables: t_order: columns: order_id: caseSensitive: false dataType: 0 generated: true name: order_id primaryKey: true user_id: caseSensitive: false dataType: 0 generated: false name: user_id primaryKey: false address_id: caseSensitive: false dataType: 0 generated: false name: address_id primaryKey: false status: caseSensitive: false dataType: 0 generated: false name: status primaryKey: falseunconfiguredSchemaMetaDataMap: ds-0:  - t_order_complex  - t_order_interval  - t_order_item_complex
Copy the code

Dynamic effect

Table configurations cannot be dynamically changed. Other configurations that are modified on ZooKeeper are synchronized to related service nodes without restarting the application node.

For example, we modify the position of the red part shown in Figure 9-2 to place T_order_ $->{0.. 1} change to t_order_$->{0.. 4}, then four shards will be generated and the mold taking rules will be changed accordingly.

Click Save, and re-launch the interface test request without restarting the application node. You can view the result of the modification.

http://localhost:8080/swagger-ui.html

Registry node

The following node information also exists on the ZooKeeper server.

namespace   ├ ─ ─ states   ├    ├ ─ ─ proxynodes   ├    ├     ├ ─ ─ ${your_instance_ip_a} @ ${your_instance_pid_x} @ ${UUID}   ├    ├     ├ ─ ─ ${your_instance_ip_b} @ ${your_instance_pid_y} @ ${UUID}   ├    ├     ├ ─ ─...   ├    ├ ─ ─ datanodes   ├    ├     ├ ─ ─ ${schema_1}   ├    ├     ├      ├ ─ ─ ${ds_0}   ├    ├     ├      ├ ─ ─ ${ds_1}   ├    ├     ├ ─ ─ ${schema_2}   ├    ├     ├      ├ ─ ─ ${ds_0}   ├    ├     ├      ├ ─ ─ ${ds_1}   ├    ├     ├ ─ ─...
Copy the code

This is the registry node, which is used to store the server instance information of shardingSphere-Proxy middleware and the instance running status.

The run instance id consists of the IP address and PID of the run server.

The identifiers of running instances are temporary nodes, which are registered when the instance goes online and automatically cleared when the instance goes offline. The registry monitors changes to these nodes to govern access to the database by running instances, etc.

Since the registry will be covered in a later article, I will not expand on it here.

Distributed Governance Summary

The introduction of a zookeeper role assists ShardingJDBC with the following functions

Centralized configuration: With more and more run-time instances, scattered configuration is difficult to manage, and the problems caused by out-of-sync configuration are serious. The configuration is centralized in the configuration center for more efficient management.
Configuration dynamics: Distribution of configuration changes is another important capability that configuration centers can provide. It supports dynamic switching between data sources and rules.
Stores runtime dynamic/temporary state data, such as available instances of ShardingSphere, data sources that need to be disabled or fushed, etc.
Provides choreography governance capability to fuse database access and disable access from slave libraries. The governance module still has a lot of unfinished functionality (flow control, etc.).

So far, the Sharding-JDBC part of the ShardingSphere ends here, and the sharding-proxy component is left unexpanded because it implements the database level Proxy, that is, The developer does not need to configure the rules of database sub-database and sub-table in the application program, but directly takes Sharding-proxy as the database source connection. Sharding-proxy is equivalent to the Proxy of Mysql database. When the request is sent to Sharding-proxy, Sharding rules are configured on sharding-proxy and are processed according to the Sharding rules.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

ShardingJdbc implements distributed governance based on Zookeeper

ShardingJDBC distributed governance

Install the Zookeeper

Sharding – JDBC integrated Zookeeper

Add jar package dependencies

Adding a Configuration File

Added Zookeeper configuration

Start the project for testing

Configuration center data structure description

Rules Global configuration rules

Props Property Configuration

/schemas/${schemeName}/dataSources

/schemas/${schemeName}/rule

/schemas/${schemeName}/table

Dynamic effect

Registry node

Distributed Governance Summary

ShardingJdbc implements distributed governance based on Zookeeper

ShardingJDBC distributed governance

Install the Zookeeper

Sharding – JDBC integrated Zookeeper

Add jar package dependencies

Adding a Configuration File

Added Zookeeper configuration

Start the project for testing

Configuration center data structure description

Rules Global configuration rules

Props Property Configuration

/schemas/${schemeName}/dataSources

/schemas/${schemeName}/rule

/schemas/${schemeName}/table

Dynamic effect

Registry node

Distributed Governance Summary

Related Posts

MySQL transactions and locks

Spring Cloud+Nginx is that simple!!

When Redis increment encounters high concurrency, the result is…