A list,

MQ, full name for Message Queue- Message Queue, is an application to application communication, one end of the Queue to continuously publish information, the other end of the Queue to read messages, publishers do not care who is reading the Message, the Message readers do not care who is publishing messages, each do not interfere with each other.

The most popular message queues in the market are: RabbitMQ, RocketMQ, Kafka and ActiveMQ

Second, the advantages of MQ

(1) decoupling

With message MQ, you only need to keep the message format unchanged, and you don’t need to care about the relationship between publishers and consumers, which don’t need to be related to each other

(2) asynchronous

In some return result operations that do not require immediate (synchronous), asynchrony is achieved through message queues.

(3) the peak clipping

In the case of a large number of requests (seckilling scenario), message queues are used for buffering to reduce peak traffic and prevent the system from being overwhelmed by peak traffic in a short period of time.

Scenario: In the peak traffic, for example, the database can only withstand 2000 concurrent traffic, you can use MQ to control 2000 to the database

(4) Log processing

Logs are stored in message queues for processing logs, such as Kafka.

3. Disadvantages of MQ

  • The availability of the system decreases

Before the introduction of MQ, the system only needed to maintain the interface consistency between the production end and the consumer end. Now, after the introduction of MQ, the system needs to pay attention to the stability of the production end, MQ, and the consumer end, which increases the burden on the system and increases the system operation and maintenance cost.

  • System complexity increases

With the introduction of MQ, there are more issues to consider, such as how to ensure that messages are consistent and that consumption is not re-consumed,

  • Consistency problem

System A returns A success message after sending the message. However, if the BCD system fails to write to the system library, data inconsistency will occur.

4. Common problems

(1) How to ensure no repeated consumption of messages? How is idempotency guaranteed using message queues

Idempotent: The result of one or more requests for the same operation is the same without side effects caused by multiple clicks

Cause of the problem

Let’s start by looking at the causes of message duplication. There are three roles for the use of MQ: producer, MQ, and consumer, and message duplication occurs:

  • Producers: Producers may push duplicate data to MQ, either because the Controller interface commits twice, or because of the retry mechanism
  • MQ: Suppose there is a fluctuation in the network. When a consumer consumes a message and sends an ACK, MQ suddenly hangs up before it can accept it. As a result, MQ thinks that the consumer has not consumed the message, and then pushes the message again after replying, resulting in repeated consumption.
  • Consumer: The consumer receives the message and is about to send an ACK to MQ. Suddenly, the consumer hangs up and cannot send an ACK. MQ assumes that the consumer has not consumed the message, and when the consumer restarts, MQ pushes the message again.

The solution

In normal circumstances, the producer is the customer, and it is difficult to avoid repeated user clicks. MQ allows multiple identical messages, but consumers are not allowed to consume two identical data, so idempotency is generally implemented on the consumer side:

  • Status judgment: consumers record consumption messages in Redis, and then go to Redis to judge whether there is such data when they consume again. If there is such data, it indicates that they have consumed and discard it directly
  • Service judgment: After data is consumed, it needs to be inserted into the database. The unique constraint of the database is used to prevent repeated consumption. Before inserting the database, query whether the data exists, and discard the message directly. This method is relatively simple and crude to solve the problem

(2) Information loss

(3) The transmission sequence of messages

solution

When the production end advertises a message, the ID of the previous message is recorded in the message body each time the method advertises a message. After receiving the message, the consumer performs the following operations

  • First, check whether the last message has not been consumed according to the last Id. If it does not exist (the Id is removed after consumption), it will proceed normally. If it works normally
  • If yes, the system checks whether the id is consumed in the database. If yes, the operation is normal
  • If it is not consumed, sleep for a certain period of time (for example, 30ms), and then check again. If it is consumed, the operation is normal
  • If it has not been consumed, an exception is thrown

(4) How to solve the problem of millions of messages backlog

There are two types of processing based on the importance of the message

  • If a message can be discarded, then simply discard it
  • In general, messages cannot be discarded, so a strategy needs to be considered. We can redeploy the original consumer side as the production side, redeploy MQ for one day, and then add the consumer side, thus forming another production-message-consumption line