RocketMQ is a distributed open messaging middleware that implements message sending and receiving functions based on a queue model. The RocketMQ cluster consists of four modules: Namesrv, Broker, Producer, and Consumer.

  • Namesrv: Stores information about all Brokers and the mapping between topics and Brokers in the current cluster.
  • Broker: The core module of a cluster, mainly responsible for Topic message storage, consumer consumption point management (consumption progress).
  • Producer: A message Producer. Each Producer has an ID. Multiple Producer instances can share the same ID. All instances with the same ID form a producer cluster.
  • Consumer: Message Consumer, each subscriber also has an ID(number) that can be shared by multiple Consumer instances. All instances under the same ID form a consumer cluster.

Cluster Deployment Architecture

1. Start Namesrv. Namesrv listens to the port and waits for Broker, Produer and Consumer to connect to it, acting as a routing control center. 2. The Broker starts, maintains long connections to all NamesRVs, and periodically sends heartbeat packets. The heartbeat package contains the current Broker information (IP+ port, etc.) and stores all topic information. After registration, the NamesRV cluster has a mapping between topics and brokers. 3. Create a topic before sending and receiving messages. When creating a topic, specify which brokers the topic will be stored on. A Topic can also be created automatically when a message is sent. 4. When the Producer sends a message, it first establishes a long connection with one of the Namesrv clusters and obtains the brokers of the currently sent Topic from Namesrv. Then, it establishes a long connection with the corresponding brokers and sends messages directly to the brokers. 5. Consumers are similar to producers. Establish a long connection to one of the NamesRVs, obtain which brokers the current subscribed Topic exists on, and then establish a connection channel directly with the brokers to start consuming messages.

 

Module Functions and Features

Namesrv

  1. Namesrv is used to store Topic and Broker relationship information with simple function and high stability. Multiple NamesRVs do not communicate with each other. The failure of a Namesrv does not affect other NamesRVs and clusters. Even if the entire Namesrv cluster goes down, the producers, consumers, and brokers that are already working can still work, but the newly created producers, consumers, and brokers cannot work.
  2. Namesrv is not too stressful, and the main overhead is maintaining heartbeat and providing topi-broker relational data. However, it should be noted that when the Broker sends a heartbeat to Namesr, it will bring all the Topic information it is currently responsible for. If there are too many topics (ten thousand levels), the data of each Topic will be tens of meters in a heartbeat. If the network condition is poor, the network transmission will fail and the heartbeat will fail. Causes Namesrv to mistake the Broker heartbeat for failure.

 

Broker

1. High concurrency read and write services

The high level of concurrent reads and writes of the Broker depends on two things:

  • Message sequential write, all Topic data will write only one file at the same time, a file full 1G, then write a new file, real sequential write disk, making the TPS of sending messages greatly improved.
  • Messages are read randomly. RocketMQ tries to make the reads match the pagecache of the system, because when the operating system accesses the Pagecache, even if only 1K messages are accessed, the system prereads more data in advance and may hit the Pagecache in the next read, reducing IO operations.

2. Load balancing and dynamic scaling

Load balancing: Brokers store Topic information. Topics are composed of queues that are evenly distributed across multiple brokers. The sending mechanism of Producer ensures that messages are evenly distributed across all queues, so that all messages fall equally on each Broker.

Dynamic scaling (non-sequential messages) : The scaling of the Broker is reflected in two dimensions: Topic and Broker.

  • Topic dimension: If the message volume of a Topic is very large, but the water pressure of the cluster is still very low, the number of queues of this Topic can be expanded. The number of queues of this Topic is proportional to the sending and consumption speed.
  • Broker dimension: If the cluster water level is high and needs to be expanded, you can simply deploy the Broker on machines. The Broker registers with Namesrv. Producer and Consumer discover a new Broker through Namesrv, and immediately connect to the Broker to send and receive messages.

3. High availability & Reliability

High availability: The cluster is deployed in active/standby mode. The standby host synchronizes messages with the hosts in real time. If one of the hosts breaks down, the standby host provides consumption services but does not provide write services.

High reliability: All messages sent to the broker have synchronous and asynchronous flushing mechanisms. A synchronous flush returns success only when a message is written to a physical file. An asynchronous flush returns success only when the machine is down. Broker failures may occur, but machine outages are rare unless there is a sudden power outage

A single Broker makes heartbeat requests to all NamesRVs at an interval of 30 seconds. The heartbeat requests contain information about all the current topics of the Broker. Namesrv checks Broer’s heartbeat. If a Broker has not had a heartbeat for 2 minutes, it takes the Broker offline and adjusts the relationship between Topic and Broker. Namesrv does not actively notify producers and consumers that brokers are down.

 

consumers

When the consumer starts, it needs to specify the Namesrv address and establish a long connection with one of the Namesrv addresses. Consumers get the latest queue status for all topics from Nameserver every 30 seconds, meaning that if a broker goes down, it can take up to 30 seconds for clients to sense it. After the connection is established, the Broker involved in the current consuming Topic is retrieved from namesRV, the directly connected Broker.

The Consumer is connected to the Broker over a long period of time and sends heartbeat messages to the Broker every 30 seconds. The Broker checks existing consumers every 10 seconds. If a Consumer has no heartbeat for 2 minutes, it disconnects from the Consumer and sends notifications to other instances of the Consumer group to trigger load balancing for the Consumer cluster.

Firstly, the consumption mode of consumers is discussed. There are two consumption modes of consumers: cluster consumption and broadcast consumption.

  • Broadcast consumption: All queues under each consumer consumption Topic.
  • Cluster consumption: a topic can share consumption among all consumers under the same ID. Specific example: If TopicA has 6 queues and 2 consumer instances are assigned to a consumer ID, then each consumer is responsible for consuming 3 queues. If you add another consumer instance with the same consumer ID, that is, there are currently 3 consumers consuming 6 queues at the same time, then each consumer is responsible for the consumption of 2 queues.

Load balancing on the consumer side means that in the cluster consumption mode, all consumer instances with the same ID consume all queues of the Topic on average.

 

Producers (Producer)

When Producer starts, it also specifies the Namesrv address and selects one of the Namesrv clusters to establish a long-term connection. If the Namesrv is down, it is automatically connected to other NamesRVs. Until a Namesrv is available.

The producer gets the Topic and Broker mappings from Namesrv every 30 seconds and updates them into local memory. Establish a long connection to all brokers involved in your Topic, with a heartbeat every 30 seconds. The Broker also scans the currently registered Producer every 10 seconds. If a Producer does not send a heartbeat for more than 2 minutes, the connection is disconnected.

Load balancing on the producer side

When a producer sends a message, it automatically polls all available brokers. If a message is sent successfully, another broker will send it next time so that the message reaches all brokers equally.

 

It is important to note here that if a Broker goes down, it can take up to 30 seconds for producers to notice. During this time, messages are sent to the down Broker. When a message fails to be sent to a Broker, it is automatically re-sent to the Broker twice. If the message still fails to be sent, an exception is thrown. Service capture exception, send again. The client automatically polls another Broker to resend, which is transparent to the user.

 

Git Clone github.com/apache/rock… Create a configuration file conf. Properties rocketmqHome = / Users/javahongxi/lot/rocketmq/distribution namesrvAddr = 127.0.0.1:9876 mapedFileSizeCommitLog=52428800 mapedFileSizeConsumeQueue=30000 -c conf.properties In turn start NamesrvStartup BrokerStartup, Consumer and Producer

Extension github.com/apache/rock…