MQ stands for message queue. Message queues are a first-in, first-out data structure.

Why use MQ?

MQ is used in the following three scenarios:

  • The application of decoupling

The more coupled the system, the less fault tolerant it is. Taking e-commerce applications as an example, after users create orders, if they call the inventory system, logistics system and payment system coupled with each other, any subsystem fails or becomes temporarily unavailable due to upgrading or other reasons, the ordering operation will be abnormal and the user experience will be affected.

Using message queues to decouple, the system becomes more coupled. For example, when the logistics system breaks down, it takes several minutes to repair. During this time, the data to be processed by the logistics system is cached in the message queue, and the user’s order operation is completed normally. When the logistics system replies, the order messages in the message queue can be processed, and the terminal system cannot perceive that the logistics system has broken down for several minutes.

  • Traffic peak clipping

An application system may be overwhelmed by a sudden surge in system traffic. Message queues can cache a large number of requests and spread them over a long period of time, which greatly improves system stability and user experience.

Generally, in order to ensure the stability of the system, if the system load exceeds the threshold, the user request will be blocked, which will affect the user experience. However, if the message queue is used to cache the request and notify the user that the order has been completed after the system finishes processing, the ordering experience will not be better.

  • Data distribution

Message queues allow data to flow more across multiple systems. The producer of the data doesn’t need to care who uses it, just sends it to the message queue, where the consumer gets it directly.

Advantages and disadvantages of MQ:

Advantages: decoupling, peak clipping, data distribution.

Disadvantages:

  • The system availability decreases

The more external dependencies a system introduces, the worse its stability will be. Once MQ is down, services are affected. How can MQ be highly available?

  • System complexity enhancement

The addition of MQ has greatly increased the complexity of systems where synchronous remote calls between systems are now made asynchronously through MQ. How do I ensure that messages are not re-consumed? How to handle message loss? What about ensuring sequential message delivery?

  • Consistency problem

After system A processes services, system A sends messages to system B, C, and D using MQ. If system B and C process the messages successfully, system D fails to process the messages. How to ensure consistency of message data processing?

Comparison of MQ products

Common MQ products include Kafka, ActiveMQ, RabbitMQ, and RocketMQ.