Starting from my blog: www.liutf.com/ starting link: www.liutf.com/posts/14196…

background

As the company’s internal RocketMQ messaging middleware is a single point instance, with the development of the business, more and more applications are connected to the same middleware, coupling is too strong. Also feel more and more insecure, unstable heart. Fear which day after down, all business also follow by implication impact. To ensure high availability of components, you need to build a new cluster and gradually migrate services to the cluster.

The current version used in production is: V3.2, which was still owned by Ali.

New build a set of official website with the latest version of V4.2.0, now belongs to Apache.

Cluster network architecture diagram

​

NameServer profiles

As the name suggests, NameServer must do the naming service and service governance work in the system, and the function should be similar to ZooKeeper. It is said that RocketMq used ZooKeeper in the early version, and later changed to its own NameServer.

NameServer has two main roles in RocketMQ:

  • NameServer maintains a list of addresses for brokers that are registered with NameServer when they are started and maintain the status of the brokers.

  • The NameServer maintains a Topic and Topic queue address list that the broker carries with it each time it sends a heartbeat.

NameServer is very stable. There are two reasons:

  1. Nameservers are independent of each other and do not communicate with each other. If a Single NameServer fails, other Nameservers are not affected. Even if all Nameservers fail, the service system is not affected. Stateless.
  2. NameServer does not have frequent reads and writes, so the performance overhead is very small and the stability is high.

The Broker profile

The broker is the core module that receives and processes messages. This module is used to receive messages sent by producers and consumers.

Relationship with the NameServer

  • The connection

    A single broker is in constant contact with all Nameservers

  • The heartbeat

    • Heartbeat interval: Heartbeats are sent to all Nameserver every 30 seconds (this time cannot be changed). Heartbeats contain their topic configuration information.
    • Heartbeat timeout: Nameserver scans all surviving broker connections every 10 seconds (this time cannot be changed). If a connection does not send heartbeat data within 2 minutes (the difference between the current time and the last update time exceeds 2 minutes, this time cannot be changed), it disconnects.

  • Disconnect the

  • Timing: Broker is dead; Heartbeat timeout causes Nameserver to shut down the connection

  • Action: Once the connection is disconnected, Nameserver is immediately aware of it and updates the mapping between topC and the queue, but producers and consumers are not notified

    ​

Load balancing

  • A topic is distributed across multiple brokers. A broker can configure multiple topics in a many-to-many relationship.
  • If a topic has a large volume of messages, it should be configured with several queues and distributed across as many brokers as possible to reduce the pressure on one broker.
  • The more queues there are on a broker, the more stressed it will be if the volume of topic messages is uniform.

The cluster is introduced

  1. A single Master

    This method is risky. Once the Broker restarts or breaks down, the entire service may become unavailable. Therefore, it is not recommended to use this method in an online environment.

    ​

  2. More than the Master model

    A cluster has no Slave but all masters. For example, it has two or three masters.

    ​

    Advantages:

    The configuration is simple. The maintenance does not affect applications when a single Master breaks down or restarts

    When RAID10, even if the machine is down unrecoverable, by RAID10

    The disk is very reliable and messages are not lost (asynchronous flushing loses a few messages, synchronous flushing loses none). Highest performance.

    ​

    Disadvantages:

    When a single machine is down, messages that are not consumed on that machine cannot be subscribed until the machine is restored, and message real-time is affected.

    Startup steps:

    Start the NameServer

    On machine A, start the first Master

    On machine B, start the second Master

    ​

  3. Multi-master and multi-slave mode, synchronous disk flushing

    A Slave is configured for each Master. There are multiple master-slave pairs. HA uses synchronous dual-write mode.

    ​

    Advantages:

    There is no single point of data or service. In the case of a Master failure, there is no delay in messages, and both service availability and data availability are very high.

    ​

    Disadvantages:

    Performance is slightly lower than the asynchronous replication mode, by about 10%, and slightly higher for RT sending individual messages. If the active node breaks down, the standby node cannot be automatically switched over to the host node. Automatic switchover will be supported later.

    Startup steps:

    Start the NameServer

    On machine A, start the first Master

    On machine B, start the second Master

    On machine C, start the first Slave

    On machine D, start the second Slave

    Brokers and slaves are paired by specifying the same brokerName parameter. BrokerId for Master must be 0, and BrokerId for Slave must be larger than 0. More than one Slave can be mounted under the same Master. Multiple Slaves in the same Master can be identified by specifying different Brokerids.

    ​

  4. Multiple Master and Slave mode, asynchronous disk flushing

    Each Master is configured with a Slave, and there are multiple master-slave pairs. The HA is asynchronously replicated. The Master has a short message delay of milliseconds.

    ​

    Advantages:

    Even if the disk is damaged, the message loss is very small and the real-time performance of the message is not affected because

    After the Master breaks down, consumers can still consume from the Slave. This process is transparent to the application. No human intervention is required. Performance is almost the same as multi-master mode.

    ​

    Disadvantages:

    The Master machine is down, the disk is damaged, and a few messages will be lost.

    Startup steps:

    Start the NameServer

    On machine A, start the first Master

    On machine B, start the second Master

    On machine C, start the first Slave

    On machine D, start the second Slave

The cluster structures,

In summary, I choose the multi-master multi-slave, asynchronous disk brushing scheme.

Environment to prepare

  • Host information
IP note
10.10.10.31 slave-b&namesrv
10.10.10.43 slave-a&namesrv
10.10.10.44 master-a&namesrv
10.10.10.45 master-b&namesrv

The NamesRV service is started on the four hosts, and the namesRV cluster is created.

Machines 44 and 45 are master-A and master-B respectively.

Machines 43 and 31 are slave- A and slave- B respectively.

Configuration requirements

Hardware:

12G+ memory (broker allocates 8G by default, NamesRV allocates 4G by default, adjustable)

Software:

  1. 64bit OS, Linux/Unix/Mac is recommended;
  2. 64 – bit JDK 1.8 +;
  3. Maven 3.2.x (not required)
  4. Git (optional)

The formal structures,

  1. First set up a stand-alone environment, refer to the stand-alone Quick Start official tutorial. Let each machine can run, and familiar with the basic command operation.

    ​

  2. After each single machine is set up successfully, then we start to do cluster configuration. The core of the cluster lies in the configuration files.

    ​

  3. Go to the configuration file directory /home/rocke/apache-rocketmq-conf. Here I use one of the machines, Rocket-master-B, as an example.

    ​

    catalogue

    • 2m-noslave: indicates the multi-master mode
    • 2m-2s-sync: Multiple masters and slaves in synchronous dual-write mode
    • 2m-2s-async: Indicates multiple Master and Slave asynchronous replication

    ​

    I choose multiple Master and Slave, asynchronous disk brushing mode, enter the 2m-2s-async directory to do configuration.

    [rocket@rocket-master-b conf]$ cd 2m-2s-async/
[rocket@rocket-master-b 2m-2s-async]$ ls
broker-a.properties  broker-a-s.properties  broker-b.properties  broker-b-s.properties
Copy the code

    ​

    You can see here that there are 4 files by default, so that’s what we’re focusing on.

    ​

  4. Modify configuration files

    • broker-a.properties

      # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file To You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the For the configuration, refer to the official link: http://rocketmq.apache.org/docs/rmq-deployment/ # belongs to the cluster name brokerClusterName = # rocketmq - cluster broker name, BrokerName =broker-a # 0 = Master, >0 = Slave brokerId=0 24 hour system, unit hour deleteWhen=04 # File retention time, default 72 hours. FileReservedTime =168 # Broker listening port =10911 # nameServer address, namesrvAddr=10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876 # Details:Should be configured if having multiple addresses; Default value:InetAddress for network interface BrokerIP1 storePathCommitLog = = # 10.10.10.45 commitLog storage path/home/rocket/app/rocketmq queue storage/store/commitLog # consumption path storage paths StorePathConsumerQueue = / home/rocket/app/rocketmq/store/consumequeue # commitLog default 1 g the size of each file MapedFileSizeCommitLog =1073741824 # Role of the Broker # -async_master Asynchronous replication Master # -sync_master Synchronous double write Master # -slave BrokerRole =ASYNC_MASTER # flush mode # -async_flush # SYNC_FLUSH flushType =ASYNC_FLUSHCopy the code

      ​

    • broker-a-s.properties

      # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file To You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the For the configuration, refer to the official link: http://rocketmq.apache.org/docs/rmq-deployment/ # belongs to the cluster name brokerClusterName = # rocketmq - cluster broker name, BrokerName =broker-a # 0 = Master, >0 = Slave brokerId=1 # = Master, >0 = Slave brokerId=1 # = Master 24 hour system, unit hour deleteWhen=04 # File retention time, default 72 hours. FileReservedTime =168 # Broker listening port =10911 # nameServer address, namesrvAddr=10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876 # Details:Should be configured if having multiple addresses; Default value:InetAddress for network interface BrokerIP1 storePathCommitLog = = # 10.10.10.44 commitLog storage path/home/rocket/app/rocketmq - slave/store/commitLog # consumption queue storage paths storage paths StorePathConsumerQueue = / home/rocket/app/rocketmq - slave/store/consumequeue # commitLog default 1 g the size of each file MapedFileSizeCommitLog =1073741824 # Role of the Broker # -async_master Asynchronous replication Master # -sync_master Synchronous double write Master # -slave BrokerRole =SLAVE # flush mode # -async_flush # SYNC_FLUSH flushType =ASYNC_FLUSHCopy the code

      ​

    • broker-b.properties

      # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file To You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the For the configuration, refer to the official link: http://rocketmq.apache.org/docs/rmq-deployment/ # belongs to the cluster name brokerClusterName = # rocketmq - cluster broker name, BrokerName =broker-b # 0 = Master, >0 = Slave brokerId=0 24 hour system, unit hour deleteWhen=04 # File retention time, default 72 hours. FileReservedTime =168 # Broker listening port =10921 # nameServer address, namesrvAddr=10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876 # Details:Should be configured if having multiple addresses; Default value:InetAddress for network interface BrokerIP1 storePathCommitLog = = # 10.10.10.46 commitLog storage path/home/rocket/app/rocketmq queue storage/store/commitLog # consumption path storage paths StorePathConsumerQueue = / home/rocket/app/rocketmq/store/consumequeue # commitLog default 1 g the size of each file MapedFileSizeCommitLog =1073741824 # Role of the Broker # -async_master Asynchronous replication Master # -sync_master Synchronous double write Master # -slave BrokerRole =ASYNC_MASTER # flush mode # -async_flush # SYNC_FLUSH flushType =ASYNC_FLUSHCopy the code

      ​

    • broker-b-s.properties

      # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file To You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the For the configuration, refer to the official link: http://rocketmq.apache.org/docs/rmq-deployment/ # belongs to the cluster name brokerClusterName = # rocketmq - cluster broker name, BrokerName =broker-b # 0 = Master, >0 = Slave brokerId=1 # 24 hour system, unit hour deleteWhen=04 # File retention time, default 72 hours. FileReservedTime =168 # Broker listening port =10921 # nameServer address, namesrvAddr=10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876 # Details:Should be configured if having multiple addresses; Default value:InetAddress for network interface BrokerIP1 =10.10.10.31 # commitLog store path StorePathCommitLog = / home/persiancat/rocketmq - slave - p/app/rocketmq - slave/store/commitlog # consumption queue storage paths storage paths StorePathConsumerQueue = / home/persiancat/rocketmq - slave - p/app/rocketmq - slave/store/consumequeue # commitLog default 1 g the size of each file MapedFileSizeCommitLog =1073741824 # Role of the Broker # -async_master Asynchronous replication Master # -sync_master Synchronous double write Master # -slave BrokerRole =SLAVE # flush mode # -async_flush # SYNC_FLUSH flushType =ASYNC_FLUSHCopy the code

    ​

    Key Configuration Description

    • BrokerClusterName: brokerClusterName in the same cluster must be the same
    • BrokerId: 0 indicates Master, >0 indicates Slave
    • NamesrvAddr: Configure multiple names separated by semicolons
    • BrokerIP1: brokerIP1 is automatically recognized by default, but some machines with multiple network cards will have identification errors, so it is recommended to specify this manually
    • BrokerRole: Select the role of the Broker
    • FlushDiskType: specifies the disk flushing mode

Start the

After the configuration file is configured, we start.

Go to the rocketmq root directory
cd /home/rocket/apache-rocketmq

# Background execution of the bin directory folder mqNamesrv service
nohup sh bin/mqnamesrv &

# broker-a machine to execute the broker-a. perties file to start
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a.properties -n "10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876" &

Start the broker-b. perties file on the broker-b machine
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b.properties -n "10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876" &

Start the broker-a-s. ties file on the broker-a-s machine
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a-s.properties -n "10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876" &

# broker-b-s machine to execute the broker-b-s. ties file to start
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b-s.properties -n "10.10.10.44:9876; 10.10.10.45:9876; 10.10.10.46:9876; 10.10.10.31:9876" &
Copy the code

Matters needing attention

  • Pay attention to the firewall, you can close the firewall at the beginning of building. Enable the firewall and configure the corresponding port.

RocketMQ console

  • The official download: https://github.com/apache/rocketmq-externals/tree/master/rocketmq-console