This is an original article. Welcome any form of reprint, but please be sure to note the source cold https://lltx.github.io.
Spring Boot 2.3 features elegant downtime details
Spring Boot 2.3 features layered JARS
This is the third article in the Spring Boot V2.3 series to share the failover optimization of Spring Data Redis in V2.3.
background
In the production of Redis, we generally choose the highly available architecture of Redis Cluster to deploy, which can not only ensure data fragmentation but also realize automatic node failure transfer. The basic deployment topology is as follows:
Creating a Test Cluster
- Here, a six-node Redis Cluster test environment can be built by pig4Cloud/Redis-Cluster :4.0 image I encapsulated.
docker run --name redis-cluster -d -eCLUSTER_ANNOUNCE_IP= Host IP address \ -p 7000-7005:7000-7005 -p 17000-17005:17000-17005 Pig4cloud/Redis-cluster :4.0Copy the code- Query cluster node information
⋊>./redis-cli -h 172.17.0.111 -p 7000 -c 16:09:48 172.17.0.111:7000> Cluster Nodes 3 d882206d40935beef84ff564b538d57369e4fd9 172.17.0.111:7003 @ 17003 slave b8d24150df4a221c1045cd9a0696bd1972912d52 0 1591344590000 4 connected
b8d24150df4a221c1045cd9a0696bd1972912d52 172.17.0.111:7001@17001 master-0 1591344590513 2 Connected 5461-10922 C21167a6da7f8af31d2dd612d449cdf92ad2e7e9 172.17.0.111:7005 @ 17005 slave baa140db6e008a137708f09d4335f5207ede3 0 810 1591344591000 6 connected 810 baa140db6e008a137708f09d4335f5207ede3 172.17.0.111:7000 @ 17000 myself, master - 0 1591344590000 1 connected 0-5460-05 d2f9884d350a50ac9e38f575b57f19e864e74c 172.17.0.111:7004 @ 17004 slave b3cf24a918d96a1949f49a1d7b3a965ff9dc858c 0 1591344590011 5 connected b3cf24a918d96a1949f49a1d7b3a965ff9dc858c 172.17.0.111:7002@17002 master-0 1591344591617 3 Connected 10923-16383Copy the codeThe application layer accesses the cluster
- Spring Boot 2.2 is used here to demonstrate the default connection pool using lettuce
spring:
redis:
cluster:
nodes:
- 172.17. 0111.: 7000
- 172.17. 0111.: 7001
- 172.17. 0111.: 7002
- 172.17. 0111.: 7003
- 172.17. 0111.: 7004
- 172.17. 0111.: 7005
Copy the code- Simply use redisTemplate to manipulate the cluster
@RestController
public class DemoController {
@Autowired
private RedisTemplate redisTemplate;
@GetMapping("/add")
public String redis(a) {
redisTemplate.opsForValue().set("k1"."v1");
return "ok"; }}Copy the code- Call view log
⋊ > curl http://localhost:8080/add ok ⏎Copy the codeWe will see that operation k1 is written at node 7000
[channel= 0x5FF7AA8f, /172.17.0.156:50783 -> /172.17.0.111:7000, epID =0x8] Write () writeAndFlushcommand ClusterCommand [command=AsyncCommand [type=SET, output=StatusOutput [output=null, error='null'].commandType=io.lettuce.core.protocol.Command], redirections=0, maxRedirections=5] [channel=0x5ff7aa8f, /172.17.0.156:50783 -> /172.17.0.111:7000, epID =0x8] write()done
Copy the codeSimulate single point of failure
- Shutting down node 7000
/redis-cli -h 172.17.0.111 -p 7000 -c 172.17.0.111:7000> SHUTDOWNCopy the code- View redis cluster docker logs -f redis-cluster
We can see that at this point the cluster election is complete and failover is complete
23: S 05 Jun 08:24:49. 387# Starting a failover election for epoch 7.Jun 29: M 05 08:24:49. 388# Failover auth granted to c21167a6da7f8af31d2dd612d449cdf92ad2e7e9 for epoch 726: M 05 Jun 08:24:49. 388# Failover auth granted to c21167a6da7f8af31d2dd612d449cdf92ad2e7e9 for epoch 723: S 05 Jun 08:24:49. 389# Failover election won: I'm the new master.23: S 05 Jun 08:24:49. 389# configEpoch set to 7 after successful failover23: M 05 Jun 08:24:49. 389# Setting secondary replication ID to 5253748ecf5bd7ab3536058fba8cad62d2d5e825, valid up to offset: 1622. New replication ID is 21d6a0b199a1ba655c0279d9c78f9682477ac9a324: M 05 Jun 08:24:489 * Discarding Previously cached master State. 23:M 05 Jun 08:24:489# Cluster state changed: ok
Copy the code- The cluster status is displayed. 7005 Changed from slave to master
Application Layer Logs
- A large number of output connections to node 7000 are abnormal
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: / 172.17.0.111:7000 under Caused by: java.net.ConnectException: Connection refused io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: / 172.17.0.111:7000 under Caused by: java.net.ConnectException: Connection refused io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: / 172.17.0.111:7000 under Caused by: java.net.ConnectException: Connection refusedCopy the code- RedisTemplate will be stuck and wait for the result to return
⋊ > curl http://localhost:8080/addCopy the code- Cause analysis,
At this point, k1 is still connected to node 7000 according to the corresponding slot. Infinite retry attempts are no longer connected. The database server is not refreshing the cluster status synchronously with the Redis cluster. Remove the down node to complete failover.
Dynamic sensing of cluster topology
Topology dynamic sensing means that the client can dynamically change the node status of the client according to the changes of the Redis cluster to complete failover.
We only need to enable this feature in Spring Boot 2.3.0.
spring:
redis:
lettuce:
cluster:
refresh:
adaptive: true
Copy the code- In fact, there has always been this function in the official database, but spring data Redis has not followed it. See the user-Content-the-cluster-topology-view section for details
Compatibility with older versions
We just need to configure topology-view for our own project by referring to what is done after the adaptive switch is turned on
@Bean
public LettuceConnectionFactory redisConnectionFactory(RedisProperties redisProperties) {
RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
// https://github.com/lettuce-io/lettuce-core/wiki/Redis-Cluster#user-content-refreshing-the-cluster-topology-view
ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh()
.enableAllAdaptiveRefreshTriggers()
.refreshPeriod(Duration.ofSeconds(5))
.build();
ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
.topologyRefreshOptions(clusterTopologyRefreshOptions).build();
// https://github.com/lettuce-io/lettuce-core/wiki/ReadFrom-Settings
LettuceClientConfiguration lettuceClientConfiguration = LettuceClientConfiguration.builder()
.readFrom(ReadFrom.REPLICA_PREFERRED)
.clientOptions(clusterClientOptions).build();
return new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
}
Copy the code