The introduction
Hello, everyone, I am South orange, from contact with Java to now also have almost two years, two years, from a Java have several data structures do not understand super small white, to now understand a little bit of advanced small white, learned a lot of things. The more knowledge is shared, the more valuable, I this period of time summary (including from other big guy over there to learn, quote) some of the focus in the ordinary study and interview (self think), hope to bring some help to everyone
This is a message middleware article, if you haven’t read it, you can follow it
- [Advanced Path] Message Queue — Principle and Selection (1)
- Message Queues — Principles of RabbitMQ (2)
The first thing I want to do is to post the mind map to you, because I use the free version, so there is a watermark. If you need the original version, you can add my wechat:
After we talked about RabbitMQ last time, we’ll talk about RocketMQ this time.
Apache RocketMQ is a high performance, high throughput distributed messaging middleware of Ali open source. The Alibaba team considered Kafka, but decided to develop RocketMQ on its own due to performance and high availability. RocketMQ is a messaging middleware that was born in the era of high concurrency distribution, so it is inherently high concurrency and transaction support.
The characteristics,
- Transactional messaging (sending messages and DB operations guarantee final consistency between the two parties, RabbitMQ and Kafka are not supported)
- Supports ultimate consistency across multiple systems
- Support for delayed messages (not supported in Kafka)
- Support for resending messages at specified times and intervals (not supported in Kafka)
- Support for tag filtering on the consumer side to reduce unnecessary network traffic (not supported by RabbitMQ and Kafka)
- Support for repeated consumption (not supported by Rabbitmq)
- The order of the messages is strictly guaranteed
- Hundreds of millions of messages
- Provides rich message pull patterns
- Producers, consumers, and queues can all be distributed.
2. RocketMQ deployment structure
Among other things, bring up this picture
As you can see, the architectural deployment of a RocketMQ cluster is similar to that of SpringCloud, and NameServer is similar to Nacos, Eureka, or Zookeeper. The producer is the same producer, the consumer is the consumer, and the Broker is the postman in the middle. However, they are all connected through the Name Server.
1, the NameServer
NameServer acts as a registry, similar to Zookeeper, but different from it. Each NameServer node is independent of each other. There is no information exchange between NameServer nodes. Therefore, there is no such problem as master selection or master/slave switchover. The failure of a Single NameServer does not affect other Nameservers or clusters. Even if the entire NameServer cluster goes down, the existing producers, consumers, and brokers will still work, but the new producers, consumers, and brokers will not work.
NameServer is lighter than Zookeeper. A single NameServer node stores topical-broker relationship information (including master and slave), where active is defined as having a heartbeat with a NameServer.
NameServer is not too stressful; the main overhead is to maintain the heartbeat and provide relational data to the topic-broker. However, when a Broker sends a heartbeat to a NameServer, it will carry information about all topics it is responsible for. If there are too many topics (tens of thousands of levels), the data of each Topic in a heartbeat will be tens of meters. If the network is poor, the network transmission fails and the heartbeat fails. The NameServer mistakenly thinks that the NameServer heartbeat fails.
2, the Broker
A Broker is a server that provides services. A single Broker node keeps a constant connection and heartbeat with all NameServer nodes, and periodically registers Topic information with NameServer. In addition, the underlying communication and connection are implemented based on Netty.
There are two Broker roles: master and Slave. Each master can have multiple Slave roles, but each slave can only have one master role. Different Brokerids (master 0) become a group. The synchronization mode between master and slave is divided into synchronous double write and asynchronous replication. In asynchronous replication, the performance of master and slave is 10% higher than that of synchronous double write, although there is a small delay.
- The Topic and Queue
- High concurrent read/write service
Message sequential write: All Topic data will only be written to one file at a time. When a file is full of 1G, it will be written to a new file, which makes the TPS of sending messages greatly improved.
Random message reads: RocketMQ tries to hit the system pagecache as much as possible, because when the operating system accesses the Pagecache, even if only 1K messages are accessed, the system will preread more data, and the next read is likely to hit the Pagecache, reducing I/O operations.
- Load balancing and dynamic scaling of brokers
Load balancing: Topic information is stored on a Broker. A Topic is composed of multiple queues, which are evenly distributed across multiple brokers. The sending mechanism of a Producer ensures that messages are evenly distributed across all queues, so that all messages are evenly distributed on each Broker.
Dynamic scaling capability (non-sequential messaging) : The scaling of a Broker is reflected in two dimensions: Topic and Broker.
Topic dimension: If the number of messages for a Topic is very large, but the cluster water level pressure is still very low, the number of queues for this Topic can be expanded. The number of queues for this Topic is proportional to the sending and consumption speed.
Broker dimension: If the water level of the cluster is high and needs to be expanded, add a machine to deploy brokers. After a Broker is created, a NameServer is registered. Producers and consumers discover a new Broker through the NameServer and directly connect to the Broker to send and receive messages.
- High availability and reliability
High availability (HA) : In cluster deployment, the active and standby servers work in active/standby mode. The standby server synchronizes messages from the host in real time. If one host breaks down, the standby server provides consumption services but does not provide write services.
High reliability: all messages sent to the broker are flushed synchronously and asynchronously; A synchronous flush returns success when the message is written to the physical file. An asynchronous flush only loses messages when the machine is down. Broker failure may occur, but a machine crash is rare unless there is a sudden power failure.
A single Broker keeps heartbeat requests with all Nameservers at an interval of 30 seconds. The heartbeat requests include all the topics of the current Broker. NameServer will check the heartbeat information of the Broer. If a Broker has no heartbeat within 2 minutes, it will be considered offline and adjust the relationship between Topic and Broker. However, NameServer does not proactively notify producers and consumers that brokers are down.
3, Producer
The producer obtains the Topic/Broker mapping from NameServer every 30 seconds and updates it to local memory. It establishes a long connection to all the brokers involved in the Topic and sends a heartbeat every 30 seconds. The Broker also scans the registered producers every 10 seconds. If a Producer does not send a heartbeat for more than two minutes, the Broker disconnects the Producer.
When the producer sends a message, it will automatically poll all available brokers. If a message is sent successfully, another broker will send the message next time, so that the message will reach all brokers on average.
One thing to note here is that if a Broker is down, it can take up to 30 seconds for the producer to notice. During this time messages are sent to the down Broker. When a message fails to be sent to a Broker, it is automatically resent to the Broker twice, and if it fails again, an exception is raised. If the service catches an exception, resend it. The client automatically polls another Broker to resend.
4, Consumer
A consumer needs to specify a NameServer address at startup to establish a persistent connection with one of the NameServer addresses. Consumers get the latest queue status for all topics from Nameserver every 30 seconds, which means that if a broker is down, it takes up to 30 seconds for clients to sense it. After the connection is established, the Broker involved in the current consumption Topic is fetched from the NameServer and directly connected to the Broker.
- Load balancing on the consumer side
A topic can be shared by all consumers under the same ID. Example: If TopicA has six queues and a consumer ID has two consumer instances, then each consumer is responsible for consuming three queues. If you add another instance of a consumer with the same consumer ID, that is, there are currently 3 consumers consuming 6 queues at the same time, then each consumer is responsible for consuming 2 queues. Load balancing on the consumer side means that in the cluster consumption mode, all consumer instances with the same ID consume all queues of the Topic on average.
Third, consumption model
When we use message-oriented middleware, the most commonly used function is the push and pull of messages. However, when to push messages and when to proactively pull messages still need to be considered according to different situations.
1. Push model
Advantages of the Push model: real-time (because once the server Broker receives the message, it sends it to the consumer, whether the consumer is ready or not, dead or alive, cached in the Consumer’s BlockingQueue).
Disadvantages of Push model:
- 1, messages stored in the server broker, easy to cause message accumulation. (Because the server Broker determines the consumer’s preferences when it first communicates with the consumer, it selects the Push model and sends it to your cache queue regardless.)
- 2. The server broker needs to maintain the state of each transmission and needs to retry if there is a problem.
- 3. The server broker needs to flow control according to the subscribers’ consumption capacity (RabbitMQ can set Qos when a consumer is created, and the Borker of the server can indicate the consumption capacity of the consumer in advance, so that the server can push a specified number of messages to the consumer).
2. Pull model
Advantages of the Pull model:
- 1, saved in the consumer end, easy to get messages.
- 2. Transmission fails and no retry is required.
- 3. The consumer can decide whether to Pull according to its own consumption ability.
Disadvantages of the Pull model:
- The real-time performance of the default short polling depends on the pull interval. The larger the interval, the lower the real-time performance. The long polling is the same as push. (refers to when there is no message for a long time, the consumer to implement the interval time to the server to rotate the message process)
3. Scenario cases
First, when the rate of the Producer is greater than that of the Consumer
There are several possibilities for this scenario. The first is that the Producer is more efficient than the Consumer (for example, the Consumer may have complex business logic for processing messages, or involve DISK or network I/O operations). The other is when the Consumer breaks down, which leads to a short period of time unable to consume or consumption is not free.
The Pull solution to this problem is simple. Since the Consumer is proactively pulling the data from the server, it simply needs to reduce the frequency of access.
Second, emphasize the real-time situation of the message
In the Push mode, once the message arrives, the server can immediately Push it to the server, which is obviously very good in real-time.
When using the Pull method, in order not to cause pressure on the server (especially when the amount of data is insufficient, continuous polling is meaningless), it is necessary to control the polling interval, but this will inevitably bring some impact on the real-time performance. (Pull will not Pull frequently, set certain interval)
Third, consumers take the initiative to obtain information for a long time
The long polling method of Pull, because the initiative is in the consumer side, the consumer can not accurately decide when to Pull the latest news. If you get the message once, you can continue to Pull. If you don’t get the message, you need to wait for a period of time and then Pull again.
The more mature approach in the industry is to start short, then grow exponentially and wait.
Anyhow is end for a long time no news consumption, consumption training in rotation intervals if it is too long, can let the part of the message delay time consumption in the interval training in rotation, if training in rotation time too short, frequent requests in consumption service Broker, the Broker to response to the consumer request (thread overhead, etc.) caused by the burden of server Broker.
You can set the consumer to wait on the connection if a pull attempt fails, instead of returning directly. If the server receives a new message, it will pull up the connection and return the latest message.
Some or all consumers are not online
In the Pull model, the server no longer cares about the state of the Consumer, but takes a “I’ll serve you as you come” approach, with no guarantees (and cleanup timeouts) as to whether the Consumer can consume the data in a timely manner.
4. Timed messages
Like RabbitMQ, RokcetMQ comes with timed messages.
Timed messages are messages sent to the Broker that are not immediately consumed by the Consumer, but can be consumed at a specific point in time or wait for a specific time.
But to support any temporal precision, message ordering must be done at the Broker level, and if persistence is involved, message ordering inevitably incurs significant performance overhead.
RocketMQ supports timed messages, but does not support any time precision. It supports specific levels, such as timed 5s, 10s, 1m, etc. There are 18 levels of message precision in RocketMQ.
5. Sequential messages
To preserve distributed transaction messages, it is necessary to ensure the order of the messages. RocketMQ ensures that message consumers consume messages in the order in which they were sent. Sequential messages can be divided into global order and local order. It is generally recommended to use local order, that is, the producer sends a certain type of messages to the same queue in order to achieve this.
Produce sends messages to the same queue when it sends them, and the consumer registers the message listener as MessageListenerOrderly, which ensures that the consumer has only one thread to consume the message.
Note: Messages are sent to the same queue, not the same topic. By default, a topic has 4 queues
Partial sequential messages can also be implemented by implementing the on-column selector method of sending messages. For example, if a database is synchronized using MQ, you only need to ensure that the data in each table is synchronized. Parse the binlog, taking the table name as an argument to the pair column selector, to ensure that every table’s data goes to the same pair column, thus ensuring that the table’s data is consumed in order.
6. Retrace your spending
Retrospective consumption is when a Consumer has successfully consumed a message and needs to re-consume it due to business requirements. To support this function, the Broker needs to keep the message after it has been delivered to the Consumer. The re-consumption is generally based on the time dimension. For example, due to the Consumer system failure, the data one hour ago needs to be re-consumed after recovery.
RocketMQ supports consumption back in time, down to the millisecond, backwards and forwards.
conclusion
In the context of RocketMQ and RabbitMQ, the distributed transaction solution is also summarized. Message-oriented middleware is an essential tool in a distributed framework, so I’ll focus on the different distributed transaction solutions. The RocketMQ cluster is not covered in this chapter, because the cluster setup is not too different. There is more to be discovered.
At the same time, if you need a mind map, you can contact me, after all, the more knowledge is shared, the more fragrant!
