In the financial industry, if the user subscribes to the SMS notification service of balance change, the user will receive the SMS notification when the balance changes, but the order of receiving the SMS must be the same as the order of changing the balance of the user’s bank account.

This scenario is a typical sequential consumption scenario. In the distributed architecture system, account balance service and SMS sending are two different micro-services, which are usually uncoupled based on MQ. The sequence diagram is shown as follows:

With MQ introduced, how do you guarantee order?

1. Theoretical basis

RocketMQ provides partition-based (queue level) sequential consumption to ensure that messages in a queue are delivered sequentially. The RocketMQ sequential consumption mechanism can implement the above scenarios as follows:

As shown in the figure above, messages from the same account are sent to the same partition during the message sending phase by hashing and modulating the account key.

RocketMQ implements sequential consumption on the consumer side as shown below:

The RocketMQ sequential consumer implementation relies on three keys:

After being allocated to a queue by the queue load algorithm consumer, it needs to apply for the queue’s lock from the Broker before pulling messages.

  • However, when consumers consume messages in the same queue after they pull messages from the same queue, the message queue is locked to ensure that the messages in the queue are executed in sequence.

  • A lock is placed on the processing queue during consumption to ensure that data is not re-consumed due to rebalancing during consumption.

From the above mechanism, sequential consumption also minimizes repeated message consumption while strictly implementing sequential semantics.

2. Code level implementation

The message sender can adopt a customized load algorithm to realize the load balancing mechanism of the queue, and its code implementation is shown in the following figure:

Warm tip: If a customized message sending load algorithm is used, the retry mechanism inside RocketMQ message sending will be invalid. Please call the upper layer of this method again and retry.

The code on the consumer side is even simpler, simply selecting the sequential consumption listener when creating the consumer, as shown below:

Note: Sequential consumer retries are not 16, but integer. MAX_VALUE, so please note that business class exceptions must be handled in the consumer listener. If the business rule is not met, infinite retries are not meaningful.

3, advanced

Ideal is beautiful, reality is skinny.

3.1 Impact of Zone Capacity Expansion or Reduction on sequential consumers

Consumption in implementation in RocketMQ order top priority is to the same account data sent to the same queue, but due to the expansion and shrinkage capacity queue, because in the process of the message is sent due to the change in the queue, the queue algorithm of load will lead to the same account information may be distributed in multiple queue at the same time, the resulting from order to execute concurrently, Cause the order to be out of order, which is never allowed in the financial industry. ,

For human operations such as capacity expansion and downsizing, we have completed the consumption of all messages in the waiting queue, which can be avoided by downtime maintenance. But what if the queue is reduced due to the abnormal operation of the Broker itself?

To realize a customized queue load algorithm, it is necessary to pass in the total number of queues of a queue. If the number of queues is not correct in the load balancing process, the message will be temporarily stored in the database and the information of these failed queues will be stored in RedIS. When sending new messages, if the load queue calculated is the failed queue, And the current queue information has been restored to the current initial value, it determines whether there are messages waiting to be sent to the database, if so, it continues to send messages to the database, and starts a thread to send messages from the database to MQ, so that subsequent messages will continue to enter MQ

Tips: If you have any questions about this piece, you can send me a private message to exchange and learn together.

3.2 Performance Problems

Threads in the order of the RocketMQ consumption model, all the messages in a partition must be executed sequentially, its performance is relatively low, the its granularity is too thick, because in the actual scene, usually only need to order the same account, different accounts, even in a partition, can also be executed in parallel, approximate solution:

For messages in a message consuming queue, we have a thread group that selects threads by key, and the messages in the thread are executed in sequence.

Well, this article is introduced here, one key three (follow, like, leave a message) is the biggest encouragement to me.

To master one or two Java mainstream middleware is a necessary skill to knock on BAT and other big factories. It gives you a learning route of Java middleware and helps you realize the transformation of the workplace.

Java advanced ladder, growth path and learning materials to help break through the middleware field

Walk into the author

Here’s some advice from a 10-year IT veteran for new employees

“I” was flattered by Ali Baba

How can programmers increase influence

How to read source code efficiently

Another way FOR me to get involved in the RocketMQ open source community