Let’s take a look at the common usage scenarios of message queues. There are many scenarios, but there are three core ones: decoupling, asynchronous, and peaking.

The decoupling

Look at this scene. System A sends data to the BCD three systems through interface calls. What if system E also wants this data? What if system C doesn’t need it anymore? The person in charge of A system almost crashed……



In this scenario, system A is heavily coupled with A variety of other messy systems, and system A produces A critical piece of data that many systems need to send to system A. System A should always consider BCDE. What should I do if the four systems fail? Do you want to resend it? Do you want to save it? The hair is white!

If MQ is used, system A generates A piece of data and sends it to MQ, which system needs the data to consume in MQ. If the new system needs data, it can be consumed directly from MQ. If the data is no longer needed by a system, simply cancel the consumption of the MQ message. In this way, system A does not need to consider who to send data to, does not need to maintain the code, does not need to consider whether the call succeeds, failed timeout, and so on.

Summary: With A model of MQ, Pub/Sub publishing and subscribing, system A is completely decoupled from the rest of the system.

Interview tip: You need to consider whether there is a similar scenario in the system you are responsible for, that is, a system or a module, calls to multiple systems or modules, calls to each other is very complex, maintenance is very troublesome. However, this call does not need to be called directly synchronously to the interface, and it is also possible to use MQ to decouple it asynchronously. You need to consider whether you can use MQ to decouple the system in your project. Show this in your resume, using MQ for decoupling.

asynchronous

In another scenario, system A receives A request, and needs to write data to its own local library and BCD libraries. The local library needs to be written in 3ms, while the BCD system needs to write data to 300ms, 450ms, and 200ms respectively. The total delay of the final request is 3 + 300 + 450 + 200 = 953ms, which is close to 1s. The user feels that something is going on. The user initiates a request through the browser and waits for a second, which is almost unacceptable.



General Internet enterprises, for the user’s direct operation, the general requirement is that each request must be completed within 200 ms, almost no perception of the user.

If MQ is used, then system A continuously sends 3 messages to the MQ queue. If it takes 5ms, the total time from system A receiving A request to returning A response to the user is 3 + 5 = 8ms. For the user, it actually feels like clicking A button and returning directly after 8ms, cool! The website was done so well, so fast!

Peak clipping

From 0:00 to 12:00 every day, system A is calm, with only 50 concurrent requests per second. Results From 12:00 to 13:00, the number of concurrent requests per second suddenly increased to 5K +. However, the system is directly based on MySQL, and a large number of requests flood into MySQL, executing about 5K SQL queries per second against MySQL.

For MySQL, the number of requests per second is 2k. For MySQL, the number of requests per second is 5K. MySQL will crash and users will no longer be able to use the system.

But once the peak is over, in the afternoon, there is a low peak, maybe there are only 1 million users operating on the site at the same time, the number of requests per second may be 50 requests, almost no pressure on the whole system.



If using MQ, 5K requests per second are written to MQ, system A can handle at most 2k requests per second, because MySQL can handle at most 2k requests per second. System A slowly pulls requests from MQ, pulling 2k requests per second, do not exceed the maximum number of requests it can handle per second, so that system A will never fail even at peak times. With 5K requests coming in for MQ and 2k going out for MQ every second, the result can be hundreds of thousands or even millions of requests stuck in MQ during the midday peak (an hour).

This brief peak backlog is ok because after the peak, 50 requests per second are coming into MQ, but system A is still processing at A rate of 2k requests per second. Therefore, as soon as the peak is over, system A will quickly solve the backlog of messages.

What are the advantages and disadvantages of message queues

Advantages have been said above, is in special scenarios have its corresponding benefits, decoupling, asynchronous, peak cutting.

The disadvantages are as follows:

- Reduced system availability The more external dependencies the system introduces, the more likely it is to fail. Originally you are A system call BCD three system interface is good, people ABCD four system is good, no what problem, you add MQ to come in, in case MQ hang how to do, MQ hang, the whole system crash, you are not finished? How to ensure high availability of message queues. - System complexity increases

How can you ensure that there is no repeated consumption of messages by simply adding MQ? How do you handle message loss? How do you ensure that messages are delivered sequentially? Big head, lots of problems, lots of pain.

- Consistency problem A system finished processing directly returned success, people think you this request was successful; But the question is, what if the BCD three system, BD two system write library succeeds, but the C system write library fails? Your numbers are inconsistent.

So the message queue is actually a very complex architecture, and you introduce it with a lot of benefits, but you have to do all sorts of additional technical solutions and architectures to get around the disadvantages, and when you do that, you find that, gosh, the complexity of the system has increased by an order of magnitude, maybe 10 times as much. But when the chips are down, use it. Use it.

What are the advantages and disadvantages of Kafka, ActiveMQ, RabbitMQ and RocketMQ?

In conclusion, after all kinds of comparison, the following suggestions are made:

General business system to introduce MQ, everyone used ActiveMQ at the earliest, but now we do not use much, not through the large-scale throughput scenario verification, the community is not very active, so we still forget it, I personally do not recommend using this;

Then they started using RabbitMQ, but it was true that Erlang prevented a large number of Java engineers from working on it. It was almost out of control for the company, but it was open source, stable support, and high activity.

But now more and more companies are using RocketMQ, which is great because it’s made by Alibaba, but there is a risk that the community will suddenly disappear (RocketMQ has been donated to Apache, but GitHub activity is actually not very high). RocketMQ is recommended, otherwise go back to RabbitMQ, because there is an active open source community and it will never go bad.

So for small and medium sized companies, where the technical strength is relatively moderate and the technical challenge is not particularly high, RabbitMQ is a good choice. For larger companies with strong infrastructure development, RocketMQ is a good choice.

If it is real-time computing in the field of big data, log collection and other scenarios, using Kafka is the industry standard, absolutely no problem, community activity is very high, absolutely not yellow, let alone almost the world in this field of the de facto specifications.