High Availability Cluster part 4 - Redis, ElasticSearch, RabbitMQ cluster

Redis cluster setup

Redis cluster form

Data partitioning scheme

Client partition

Client partition scheme is represented by Redis Sharding. Redis Sharding is the Redis multi-instance Cluster method commonly used in the industry before The emergence of Redis Cluster. Java Redis client driver library Jedis, support Redis Sharding function, namely ShardedJedis and ShardedJedisPool combined with cache pool
Advantages: No third-party middleware, controllable partition logic, simple configuration, no association between nodes, easy linear expansion, strong flexibility
Disadvantages: Clients cannot dynamically add and delete service nodes. Clients need to maintain distribution logic by themselves. Clients cannot connect to each other and share services, wasting resources

Acting partition

Twemproxy and Codis are commonly used to bring partitions

redis-cluster

High availability mode

Sentinel (Sentinel mechanism) supports high availability

In front of the master-slave mechanism is introduced, but from the operational point of view, the master node there is a problem we need through the way of manual intervention from the node set is given priority to, also inform application updates the master node address, this way is cumbersome bulky, and the master node, speaking, reading and writing are limited, is there a better way to solve these two problems, The sentry mechanism is an effective solution to the first problem. The second problem relies on clustering. The function of sentry is to monitor the running status of Redis system. Its functions mainly include the following three functions:
- Monitoring: The sentry constantly checks that your Master and Slave are working properly
- Notification: Sentry can send notifications to administrators or other applications via the API when a monitored Redis has a problem
- Automatic failover: When the primary database fails, the secondary database is automatically converted to the primary database

Principle of sentry

Three scheduled tasks of Redis sentry. The Redis sentry determines that a Redis node is faulty and unreachable mainly through three scheduled monitoring tasks:
- The sentinel sends the info replication command to the master and slave nodes every 10 seconds to get the latest topology
- Sentinel_ : Hello every two seconds, sentinels send their own observations and their own node information back to the _sentinel_: Hello channel of the Redis node, and other sentinels subscribe to this channel to learn about other sentinels and their observations on the primary node
- Every second, each sentinel sends a ping command to the master, slave, and other sentinels to check the heartbeat
  
  If Job3 fails to detect the heartbeat of the node at scheduled time, it considers that the node is subjectively offline. If the node is still the master node, other sentinels will be notified to perform heartbeat detection on the master node. When the number of votes subjectively offline exceeds the number, the master node may indeed be unreachable, and the original subjective offline becomes objective offline
- Failover and Leader election
  
  If the master node is judged to be objectively offline, a sentinel node is selected to complete the subsequent failover and a leader is elected. The election algorithm Raft is used here. The sentinel leader is elected to perform failover, that is, the primary node that elects a node from among the nodes to care for

Redis-Cluster

Redis cluster documents. –
Redis’ official multi-machine deployment solution: Redis Cluster; A group of Redis Cluster is composed of multiple Redis instances. It is officially recommended that we have 6 instances, of which three are master nodes and three are slave nodes. Once a primary node fails, the Redis Cluster can elect the corresponding secondary node as the new primary node to continue external services, thus ensuring the high availability of services.
So for the client, how to know the corresponding key is to route to which node?

Redis Cluster divides all data into 16384 different slots, which can be allocated to different Redis instances according to machine performance. For Redis instances, they only store part of Redis data. Of course, the data in the slots can be migrated. The instances that do not listen can migrate data through certain protocols

slot

Functional limitations of Redis clusters; Redis cluster has some limitations in function compared with single machine, so developers need to know about it in advance and avoid it when using it. JAVA CRC16 verification algorithm
There is limited support for batch operation
- The mSET and MGET operations can be performed in batches only on keys with the same slot value. Keys mapped to different slot values are not supported because operations such as Mget and Mset may exist on multiple nodes
Limited support for key transaction operations
- Only transactions with multiple keys on the same node are supported. When multiple keys are distributed on different nodes, the transaction function cannot be used
Can be used as a minimum granularity for data partitioning
A large key-value object such as hash, list, etc. cannot be mapped to different nodes
Multiple database Spaces are not supported
- A single Redis server supports 16 databases (DB0-DB15). In cluster mode, only one database space can be used, that is, DB0
The copy structure supports only one layer
- The secondary node can only replicate the primary node, and the nested tree replication structure is not supported
Most commands are redirected, which takes a lot of time

Consistency of the hash

Consistency hash is a good way to solve the stability problem. You can arrange all storage nodes on the hash ring that is connected from end to end. After computing the hash, each key will find the adjacent storage node and store it clockwise. When a node joins or exits, only the subsequent nodes clockwise next to the node on the Hash ring are affected
Hash tilt

If there are few nodes, it is easy to tilt and load imbalance. Consistent hash algorithm, the introduction of virtual nodes, in the whole ring, a number of nodes added evenly; Such as A1, A2, B1, B2, C1, C2; Both A1 and A2 belong to node A, solving the hash skew problem

The deployment of the Cluster

Create six Redis nodes

In three-master, three-slave mode, slave is used to fragment slot data for synchronous backup

Create the directory and configuration, and start the container

for port in $(seq 7001 7006); \
do \
mkdir -p /var/mall/redis/node-${port}/conf
touch /var/mall/redis/node-${port}/conf/redis.conf
cat <<EOF>> /var/mall/redis/node-${port}/conf/redis.conf
port ${port}
cluster-enabled yes
cluster-config-file nodes.conf
cluster-announce-ip 192.168.83.133
cluster-announce-port ${port}
cluster-announce-bus-port 1${port}
appendonly yes
EOF
docker run -p ${port}:${port} -p 1${port}:1${port} --name redis-${port} \
--restart=always \
-v /var/mall/redis/node-${port}/data:/data \
-v /var/mall/redis/node-${port}/conf/redis.conf:/etc/redis/redis.conf \
-d redis redis-server /etc/redis/redis.conf; \
done
Copy the code

#Stop the six Redis containers
docker stop $(docker ps -a |grep redis-700 | awk '{print $1}')
#Delete the six Redis containers
docker rm $(docker ps -a | grep redis-700 | awk '{print $1}')
Copy the code

Build clusters using Redis

Enter the Redis container of any master node

#Into the container
docker exec -it redis-7001 /bin/bash
#If master is not specified, the cluster will be randomly specifiedRedis -cli --cluster create 192.168.83.133:7001 192.168.83.133:7002 192.168.83.133:7003 192.168.83.133:7004 192.168.83.133 192.168.83.133:7005:7006 - cluster - replicas of 1Copy the code

Enter yes to accept the configuration. The cluster is set up

Test the Redis cluster

Connect to redis client and save data (fragmented storage)

-c redis-cli -c -h 192.168.83.133 -p 7001 set Hello 1 set a aaa set bb aaCopy the code

Viewing Cluster Status
```
cluster info
Copy the code
```
Viewing Node Information
```
cluster nodes
Copy the code
```

Simulate a primary node down

Redis -cli -c -h 192.168.83.133 -p 7002 docker exec -it redis-7002 /bin/bash Redis7001 will replace the faulty 7001 host with a copy of hello (slave)Copy the code

At this point, the downed 7001 starts up again and observe the node changes (failover)
```
Docker restart redis-7001 docker restart redis-7001Copy the code
```

The ElasticSearch cluster is created

Es Cluster Principle

ElasticSearch is clustering by nature, it doesn’t have to rely on other services to discover and register components like ZooKeeper, because it has a built-in name calledZenDiscoveryThe ElasticSearch module is a self-implemented set of components for node discovery and master selection, so it is very easy for ElasticSearch to cluster, and does not require much extra configuration to install additional third-party components

A single node

A running instance of ElasticSearch is called a node, and a cluster consists of one or more nodes with the same cluster.name configuration. They share the burden of data and load, and when a node is added to the cluster or removed from the cluster, the cluster redistributes all data evenly
When a node is elected as the master node, it is responsible for managing all changes within the cluster, such as adding or deleting indexes, or adding or deleting nodes. The master node does not need to involve changes and searches at the document level, so when the cluster has only one master node, it will not become a bottleneck even if traffic increases, and any node can become the master node. Our example cluster has only one node, so it also becomes the primary node
As users, we can send requests to any node in the cluster, including the primary node; Each node knows where any document is and is able to forward our requests directly to the node that stores the document we need; No matter which child node we send the request to, it collects data from each node that contains the document we need and returns the final result to the client, which is transparently managed by ElasticSearch

Cluster health

The ElasticSearch cluster monitoring information contains many statistics. The most important one is cluster health, which is displayed as green, yellow, or red in the Status field
```
GET /_cluster/health
Copy the code
```
Status: Indicates whether the current cluster is working properly in general. Its three colors are as follows:

Green: All master and replica shards are working properly

Yellow: All master shards are running properly, but not all replica shards are running properly

Red: Master sharding is not running properly

shard

A shard is an underlying unit of work that holds only a portion of the total data; Our documents are stored and indexed into shards, but the application interacts directly with the index rather than the shard; A sharding task is an area of data
A shard can be either a master shard or a replica shard. Any document in the index belongs to a master shard, and the number of master shards determines the maximum amount of data that the index can hold
The number of master shards is determined at index creation time, but the number of replica shards can be changed at any time
Let’s create an index named Blogs in a cluster containing an empty node. Indexes are assigned 5 master shards by default, but for demonstration purposes: we will assign 3 master shards and one copy (one copy for each master shard)
```
PUT /blogs{
	"settings":{
		"number_of_shards": 3,
		"number_of_replicas": 1
	}
}
Copy the code
```
A single-node cluster with an index

In this case, if the health status of the cluster is YELLOW, all master fragments are running normally (the cluster can service all requests properly), but not all replica fragments are in the normal state. In fact, all three shards are unassigned — they are not assigned any nodes;

It doesn’t make sense to keep both original data and copies on the same node, because once we lose that node, we lose all copies on that node as well

When our cluster is up and running, there is a risk of losing data in the event of a hardware failure

The new node

When you start a second node on the same machine, only it has the same cluster.name configuration as the first node, it will automatically discover the cluster and join it. However, when starting nodes on different machines, you need to configure a list of connected unicast hosts in order to join the same cluster. For more information see best use unicast instead of multicast
Cluster with two nodes – all master and replica shards have been allocated

At this point, Cluster-Health now displays green status, which means that all 6 shards (including 3 master shards and 3 replica shards) are running normally. Our cluster is now not only up and running, but always available

Horizontal expansion – Start the third node

For three-node clusters – shards are redistributed to spread the load

One shard on Node1 and one shard on Node2 have been migrated to the new Node3 node, which now has two shards on each node instead of three. This means that the hardware resources (CPU, RAM, I/O) of each node will be shared by fewer shards and the performance of each shard will be improved

In the running cluster can dynamically adjust the number of replica fragments, we can scale the cluster on demand; Let’s increase the number of copies from the default of 1 to 2
```
PUT /blogs/_settings
{
	"number_of_replicas": 2
}
Copy the code
```
Blogs index now has nine shards: three master shards and six replica shards; This means we can scale the cluster up to nine nodes, one shard on each node; The search performance of the cluster is three times better than that of the original three nodes

Deal with failure

A cluster after a node is shut down
The node we shut down is a primary node; The cluster must have a master node to work, so the first thing that happens is to elect a new master node: Node2
We lose master shards 1 and 2 when Node1 is closed, and the index does not work properly when master shards are missing; If we were to check the health of the cluster at this point, we would see a red state: not all master shards are working properly
Fortunately, full copies of the two master shards exist on other nodes, so the new master node immediately promotes the corresponding copies on Node2 and Node3 as master shards, at which point the cluster status will be yellow. This promotion of the master shard is instantaneous, as if flipping a switch
Why is our cluster state yellow instead of green? Although we have all three master shards, two copy shards are set for each master shard, and only one copy shard exists at this time. So the cluster is not in the green state

But don’t worry too much: if we also turn off Node2, our program can still run without losing any data, because Node3 holds a copy for each shard
If we restart Node1, the cluster can fragment the missing copies for re-allocation; If Node1 still has the previous shards, it will try to reuse them and only copy the modified data files from the master shard

Problem solving

The master node

The master node is responsible for creating indexes, deleting indexes, allocating fragments, and tracking node status in the cluster. The primary node of ElasticSearch has a relatively light workload. Users’ requests can be sent to any node in the cluster, and the node is responsible for distributing and returning the results without the primary node forwarding. The primary node is elected by the candidate primary node through ZenDiscovery mechanism. If you want to become the primary node, you must become the candidate primary node first

Candidate primary node

If the ElasticSearch cluster is initialized or the primary node is down, select one of the primary nodes as the primary node and set the node configuration to node.master:true

When the load of the master node is too high, or the network problem in the cluster environment leads to the communication between other nodes and the master node, the master node does not have time to respond. In this case, some nodes think the master node is down and choose a new master node, so the whole cluster has problems. For example, there are 10 nodes in our cluster, among which 7 are candidate primary nodes and 1 candidate primary node becomes the primary node, which is normal. But if there is this what we call the master node is not timely response and lead to some of the other nodes as think the master node to choose the master node’s downtime and heavy, that’s a problem, the remaining six candidate master node may have three candidates for the master node to reelect the master node, and the last came two master node in the cluster, this situation is officially called brain crack phenomenon

Different nodes in the cluster have different choice of master, and multiple master competitions occur, leading to differences in the identification of master fragments and replicas. Some fragments in the divergence are identified as bad fragments

Data nodes

The data node is responsible for data storage and related specific operations, such as CRUD, search and aggregation. Therefore, data nodes have high requirements on the configuration of machines. First, sufficient disk space is required to store data. Second, data operations consume a lot of system CPU, Memory, and IO performance. Typically, as a cluster grows, more data nodes need to be added to improve availability; Node. data:true specifies the configuration of the data node

ElasticSearch allows a node to act as both a candidate primary node and a data node. However, due to heavy load on the data node, separate the two nodes and set a dedicated candidate primary node and a data node to prevent the primary node from responding due to the heavy load on the data node

Client node

Client nodes shop is to do neither candidate nodes doesn’t do the master node data, only responsible for the request of distribution, summary and so on, but this job, in fact, any one node can be completed, because in ElasticSearch within a cluster of node can perform any request, it will be responsible for forwarding requests to the corresponding node for processing; So adding such nodes alone is more for load balancing; Specify the configuration of this object as:

node.master:false

node.data:false

Possible causes of the split brain problem

1. Network problems: the network delay of the cluster key causes some nodes cannot access the master, and the task master node hangs, so a new master is elected, and the shards and copies on the master are marked red, and a new master shard is assigned
2. Node load: The role of the master node is both master and data. When there is a large number of visits, ES may stop responding and cause a large area of delay
3. Memory reclamation: The ES process on the data node occupies a large amount of memory, resulting in large-scale memory reclamation of the JVM, causing the ES process to lose response
Solution to brain split problem:
- Role separation: The master node is separated from the data node, limiting the role. Data nodes need to undertake the work of storage and search, so the pressure will be great. Therefore, if the node serves as both the candidate master node and the data node, then once it is selected as the master node, the working pressure of the master node will be very high, and the probability of brain split phenomenon will increase
- Reduce misjudgmentBy default, the primary node does not respond for 3 seconds. If the other nodes are down, then we can set the response time to be longer.discover.zen.ping_timeout:5
- The election triggered:discovery.zen.minimum_master_nodes:1(Default is 1), which defines the minimum number of primary nodes that are connected to each other in order to form a cluster
  - A cluster of 10 nodes with each node qualified as a master node,discovery.zen.minimum_master_nodesSet the parameter to 6
  - Normally, 10 nodes, connected to each other, can form a cluster with more than 6 nodes
  - If at some point, three of the nodes are disconnected, and the remaining seven nodes, larger than 6, continue to run the previous cluster, while the three disconnected nodes, smaller than 6, cannot form a cluster
  - This parameter is to prevent the occurrence of split brain
  - You are advised to set this parameter to (Number of candidate primary nodes / 2) + 1

The cluster structure

Take three physical machines as an example, on these three physical machines, six ES nodes, three data nodes, three master nodes (each physical machine has one data and one master node respectively), and three master nodes are set up to achieve the requirement that (n/2) +1 equals 2. After the master node fails, n equals 2. The other two master nodes assume that the master node has failed
```
Minimum_master_nodes =2 #data node.master=falsse node.data=trueCopy the code
```

Es Cluster Construction

Es cluster: three master nodes, three data nodes

Run all operations before you do them

Echo vm.max_map_count=262144>>/etc/sysctl.conf sysctl -pCopy the code

Prepare the Docker network

Docker creates containers using bridge network by default and assigns IP by itself

In actual deployment, you need to specify the IP address of the container and do not allow the container to assign IP addresses by itself. In particular, a fixed IP address is required when building a cluster. We can create our own Bridge network: Mynet, specifying the network as mynet and IP when creating the container

The network mode is Docker network LS

Create a new Bridge network
```
# create mynet docker network create --driver bridge --subnet=172.18.12.0/16 --gateway=172.18.1.1 mynet  inspect mynetCopy the code
```
Use –network=mynet — IP 172.18.12.x to specify the IP address

Create three Msater nodes

The script generates the configuration and starts the container

#discovery.zen.minimum_master_nodes:2 
#Set this parameter to ensure that the nodes in the cluster can know the other N master nodes. The official recommendation is (N/2) + 1for port in $(seq 1 3); \ do \ mkdir -p /var/mall/elasticsearch/master-${port}/config mkdir -p /var/mall/elasticsearch/master-${port}/data chmod  -R 777 //var/mall/elasticsearch/master-${port} cat<<EOF> /var/mall/elasticsearch/master-${port}/config/jvm.options -Xms128m -Xmx128m EOF cat<<EOF> /var/mall/elasticsearch/master-${port}/config/elasticsearch.yml cluster.name: Name: es-master-${port} # this node is named node.master: true # this node has the opportunity to become a master node. Host: 0.0.0.0 HTTP. Host: 0.0.0.0 HTTP. Port: 920${port} transport. 930${port} discovery.zen.ping_timeout: 10s # Set the ping timeout for automatically discovering other nodes in the cluster#Configure the initialization list of Master nodes in a cluster. You can use these nodes to automatically discover other nodes that have joined the cluster. New configuration in ES7Discovery. Seed_hosts: [172.18.12.21: "9301", "172.18.12.22:9302", "172.18.12.23:9303"] cluster. Initial_master_nodes: ["172.18.12.21"] Es7 add EOF docker run -d --name= ES-node -${port} \ -p 920${port}:920${port} -p 930${port}:930${port} --network mynet -- IP 172.18.12.2 ${port} \ - v /var/mall/elasticsearch/master-${port}/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /var/mall/elasticsearch/master-${port}/config/jvm.options:/usr/share/elasticsearch/config/jvm.options \ -v /var/mall/elasticsearch/master-${port}/data:/usr/share/elasticsearch/data \ -v The/var/mall/elasticsearch/master - ${port} / plugins: / usr/share/elasticsearch/plugins \ elasticsearch: 7.6.1. \ doneCopy the code

# stop all es container docker stop $(docker ps - a | grep es - node - * | awk '} {print $1 ') # delete all es container docker rm -f $(docker ps - a | grep es-node-*|awk '{print $1}')Copy the code

Create three Data nodes

How create script differs from Master

for port in $(seq 4 6); \ do \ mkdir -p /var/mall/elasticsearch/node-${port}/config mkdir -p /var/mall/elasticsearch/node-${port}/data chmod -R 777 //var/mall/elasticsearch/node-${port} cat<<EOF> /var/mall/elasticsearch/node-${port}/config/jvm.options -Xms128m -Xmx128m EOF cat<<EOF> /var/mall/elasticsearch/node-${port}/config/elasticsearch.yml cluster.name: Name: es-node-${port} # node.master: false # node.data: Host: 0.0.0.0 HTTP. Host: 0.0.0.0 HTTP. Port: 920${port} transport.tcp.port: 930${port} discovery.zen.ping_timeout: 10s # Set the ping timeout for automatically discovering other nodes in the cluster#Configure the initialization list of Master nodes in a cluster. You can use these nodes to automatically discover other nodes that have joined the cluster. New configuration in ES7Discovery. Seed_hosts: [172.18.12.21: "9301", "172.18.12.22:9302", "172.18.12.23:9303"] cluster. Initial_master_nodes: ["172.18.12.21"] Es7 add EOF docker run -d --name= ES-node -${port} \ -p 920${port}:920${port} -p 930${port}:930${port} --network mynet -- IP 172.18.12.2 ${port} \ - v /var/mall/elasticsearch/node-${port}/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ -v /var/mall/elasticsearch/node-${port}/config/jvm.options:/usr/share/elasticsearch/config/jvm.options \ -v /var/mall/elasticsearch/node-${port}/data:/usr/share/elasticsearch/data \ -v The/var/mall/elasticsearch node - ${port} / plugins: / usr/share/elasticsearch/plugins \ elasticsearch: 7.6.1. \ doneCopy the code

Docker logs Es-node-1 docker logs es-node-4Copy the code

Check the working status of the ES cluster

1, browser access 192.168.83.133:9201 — 9206 can be accessed normally
2. Partial commands
- /_nodes/process? Pretty: View node status
- /_cluster/stats? Pretty: View the cluster status
- /_cluster/health? Pretty: View cluster health
- /_cat/nodes: Views node information
  - The master node is es-master-3. Try to stop the es-master-3 container.
    
    docker stop es-node-3
    
    You’ll find a mechanism that triggers primary selection,es-master-1The node is elected as the primary node
    
    Start es-master-3 again, and the cluster will assume that this is a new addition and es-master-1 is still the master node
    
    docker restart es-node-3

The RabbitMQ cluster is built

The cluster form

RabbitMQ is developed by Erlang and clustering is very convenient because Erlang is a distributed language by nature but does not support load balancing
A RabbitMQ cluster contains RAM nodes, Disk nodes, and message persistence. The cluster has at least one Disk node

Normal mode (default)

In normal mode, each node in the cluster has the same queue structure, but messages exist on only one node in the cluster. For the consumer, if the message enters the Queue on node A and is pulled from node B, RabbitMQ will take the message from node A and send it to the consumer via node B
Application scenarios: This mode is suitable for situations where messages do not need persistence, such as log queues. When the queue is non-persistent and the node that created the queue breaks down, the client can reconnect to other nodes in the cluster and re-create the queue. If it is persistent, you can only wait for the faulty node to recover

Mirror mode

Different from the common mode, the message entity is automatically synchronized between the mirror nodes, instead of temporarily pulling data, which is highly available. In this mode, the mirror Queue has an election algorithm, that is, one master and n salves. Requests from producers and consumers are forwarded to the master
Application scenario: High reliability requirements, such as single order or inventory queue

Disadvantages: If there are too many mirroring queues and the message volume is large, the network bandwidth within the cluster will be consumed by such synchronous communication
- 1. A mirror cluster is also based on a common cluster, that is, a common cluster must be built before a mirror queue can be set up
- 2. If the master fails during consumption, a new master will be elected. If it is not confirmed in time, repeated consumption may occur

Setting up an MQ Cluster

Create a directory

mkdir -p /var/mall/rabbitmq/rabbitmq01
mkdir -p /var/mall/rabbitmq/rabbitmq02
mkdir -p /var/mall/rabbitmq/rabbitmq03
Copy the code

Start the MQ container

#rabbitmq01
docker run -d --hostname rabbitmq01 --name rabbitmq01 -v /var/mall/rabbitmq/rabbitmq01:/var/lib/rabbitmq  -p 15671:15672 -p 5671:5672 -e RABBITMQ_ERLANG_COOKIE='mall' rabbitmq:management

#rabbitmq02
docker run -d --hostname rabbitmq02 --name rabbitmq02 -v /var/mall/rabbitmq/rabbitmq02:/var/lib/rabbitmq  -p 15672:15672 -p 5672:5672 -e RABBITMQ_ERLANG_COOKIE='mall' --link rabbitmq01:rabbitmq01  rabbitmq:management

#rabbitmq03
docker run -d --hostname rabbitmq03 --name rabbitmq03 -v /var/mall/rabbitmq/rabbitmq03:/var/lib/rabbitmq  -p 15673:15672 -p 5673:5672 -e RABBITMQ_ERLANG_COOKIE='mall' --link rabbitmq01:rabbitmq01  --link rabbitmq02:rabbitmq02 rabbitmq:management

#--hostname sets the container hostnameRABBITMQ_ERLANG_COOKIE Node authentication. This value must be synchronized to cluster deploymentCopy the code

Adding a Node to a Cluster

Enter the rabbitmq01 container

Docker exec -it rabbitmq01 /bin/bash # reset rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app exitCopy the code

Enter the second node rabbitmq02

Docker exec -it rabbitmq02 /bin/bash Rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl join_cluster --ram rabbit@rabbitmq01 rabbitmqctl start_app exitCopy the code

Enter the second node rabbitmq03

Docker exec -it rabbitmq03 /bin/bash Rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl join_cluster --ram rabbit@rabbitmq01 rabbitmqctl start_app exitCopy the code

The default cluster is a common cluster, which may have single point of failure

Implementing a Mirror cluster

Enter any MQ container

Docker exec -it rabbitmq01 /bin/bash # Rabbitmqctl set_policy -p/ha "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'Copy the code

Rabbitmqctl set_policy -p/ha-all "^" '{"ha-mode":"all"}' The policy regular expression is "^", indicating that all matches are matchedCopy the code

Verifying a Mirror Cluster

Create a queue on the master node

Theoretical result: Rabbitmq02 and RabbitMQ03 automatically synchronize queues
Testing data synchronization

High Availability Cluster part 4 – Redis, ElasticSearch, RabbitMQ cluster