Initial message queue & RocketMQ

Make writing a habit together! This is my first day to participate in the “Gold Digging Day New Plan · April More text challenge”, click to see the details of the activity.

To learn RocketMQ, we need to understand what it does and why we use it, not just for the sake of learning. After all, time is precious.

What is messaging middleware

Message queue, also known as MQ (Message Queue), is now the core means of message flow in enterprise internal systems. It has the following characteristics:

• Low coupling
• Reliable delivery of messages
• The news spread far and wide
• Flow can be controlled (peak cutting and valley filling)
• Data final consistency

I will cover all of the above, but also take you through an architectural understanding of message-oriented middleware, how messaging is used in enterprise systems, and how MQ works and is used in real-world scenarios.

Why use MQ

Peak peel

Using the MQ scene has a lot of, such as take electricity market for example, the scene is known to all, when double tenth of a year, taobao’s backend server will be carrying huge pressure, if all the traffic that instant hit business server, may be instant service goes down, but if we are the users from the first news piled up, The specific service server obtains messages from the message queue according to its actual situation and then performs corresponding operations to avoid the problem caused by excessive instantaneous pressure. This is peak load cutting and valley filling.

The service of decoupling

When you click the order button in Taobao, in fact, the background is doing a lot of work:

• First check your basic information, then check the inventory of goods and make deductions, then calculate the discount information, then pull the payment method, notify the seller of shipment after successful payment, etc., each step of the work is not small, if we put all the above process in one piece of code, It is very likely that a small problem in the previous link will lead to the failure of the whole link, and the user experience will be poor. This is the traditional design of strong coupling.
• For this, we can use message queue to decouple, send a message after the execution of a step, the downstream link reads the corresponding message and then performs further operations, changing the synchronous operation into asynchronous operation, so that some of the tasks are executed at the same time.

As shown in the figure below, A, B, C and D are the processes for users to initiate payment, while E is the process for notifting sellers of delivery. We can completely decouple E process and let B, C and D proceed simultaneously to speed up the execution of the whole link:

How should I choose the technology?

Currently, there are four main types of MQ:

• ActiveMQ
• RabbitMQ
• Kafka
• RocketMQ

Top to bottom technology from old to new. Specific how to do the technology selection you can refer to the following table:

• Kafka is recommended for big data applications
• RabbitMQ is recommended for small and medium sized companies, as the software is not as data-intensive and has more functionality than Kafka. The disadvantage is that it is developed in The Erlang language, which prevents large Java programmers from improving it
• Large companies should choose before Kafka and RocketMQ because they have enough data, enough money to build a clustered environment, and even enough people to customize message queues

In addition, there are several ways to think about what kind of message queue a company is suited for:

1. performance
2. Degree of functional completeness
3. Development of language
4. Has any company used it? What is the effect?
5. Community activity and documentation detail
6. Employee learning cost
7. Whether cluster and distributed are supported

Be cautious with message-oriented middleware

The benefits of message-oriented middleware have been described above, but that doesn’t mean that all of our projects need to “mindlessly” introduce message queues. After all, too much is too much.

Once we decided to introduce MQ, the first question we had to ask was would it pose a risk to the entire project? Think about it from the following perspectives:

1. System complexity: The introduction of MQ into a project inevitably adds to the complexity of the system, such as how to achieve final alignment? How to ensure that messages are not re-consumed? How to ensure reliable transmission of messages? And so on and so on, whether we can accept that kind of complexity increase.

2. Maintenance cost: With the increase of system complexity, additional cost will be needed for the design of architecture and technical scheme. Do we accept additional human cost?

But in fact, the current domestic Internet companies are often rough and rapid development mode, rapid product iteration has become a basic requirement of the enterprise, so the project often adopts the micro-service architecture, so the introduction of MQ has become inevitable.

The purpose of this section is to show you how to design MQ in detail

RocketMQ application scenario Example

This section will give you a few classic examples of how to use MQ.

Asynchronous processing

Suppose there is a user login requirement, because the security level of our application is set relatively high, so the product requires that all registered users must be authenticated by mobile phone + email before registration is successful.

If the process is designed in a serial way:

The user initiates a login request, and after receiving the request, the background sends an authentication email, then an authentication SMS, and finally feedback to the user for a response. Assume that a total of 150ms is required.

If MQ and asynchronous processing are introduced:

If message queue + asynchronous processing is used, the response time will become 50ms. The analysis is as follows:

1. The user initiates a login request
2. Upon receiving the request, the login module writes a new message to MQ, which takes almost no time, and then directly returns the response to the user
3. The background system asynchronously reads messages from the MQ server and executes their respective logic. After receiving the MESSAGE, users complete the operation

So overall, the number of login requests processed in 1s goes from 7 synchronous to 20 asynchronous, which is about a threefold improvement in performance!

The application of decoupling

Returning to the example of the previous part of e-commerce, after users place an order, the order system needs to inform the inventory system to reduce the inventory. The intuitive idea is to directly call the inventory system in the order system and perform subsequent operations after the inventory deduction is completed:

Such a call method has an obvious disadvantage, that is, once the inventory system is suspended, the order system will always return failure, users can not place orders normally, the two systems are strongly coupled together, we can adopt the following approach:

The order system writes a message after the user places an order and then returns directly; The inventory system reads this message consumption and subtracts the inventory of the corresponding item.

In this way, even if the inventory system is down, users will not be affected to place an order, because the order system ends after the message is written, decoupling the two.

Log processing

Log collection The system collects log data and writes it to MQ. The log processing system is responsible for processing log data in MQ.

Interprocess communication

Message queues are generally based on efficient communication frameworks, such asNetty, so it can be used to realize interprocess communication:

Distributed transactions guarantee ultimate consistency

User A transfers money to user B, using MQ to ensure final consistency of the system: