​

Message queue MQ is an application – to – application communication method. It is essentially a first-in, first-out data structure.

1. MQ application Scenarios

1, the decoupling

  • Apply decoupling to solve the problem

    In the following systems, if the functions of B, C and D change, system A should also change; or when system E is added, the call of A should also be added. In this way, system A and B, C, D and E are coupled together, and it is troublesome to add or modify functions.

​

After adding MQ components:

If system A wants to call system B, C and D, it only needs to send the request data of A to MQ, which is then consumed by the BCD system area. After adding MQ, if the functions of BCD system need to be added and modified, it has little relevance with system A.

​

2, asynchronous

  • Problems solved by applying asynchrony:

The system does not asynchronously send the previous request to system A, and system A needs to call three systems B,C and D for serial processing, so the processing time is the sum of the processing time of all systems. For example, the order of goods needs to be processed by payment, inventory and order systems

​

After the introduction of MQ, system A only needs to send the request data to MQ when initiating the order request, and then directly return the response to the user. Finally, the order message is consumed by the BCD system, so that the processing time felt by the user is the processing time of system A.

​

3, front

Traffic problem solved:

Suppose that the normal traffic of a system is only 10,000 TPS, and the peak traffic is 100,000 TPS during activities. Considering the cost, the enterprise will not design the system according to the requirement of 100,000 TPS, but sometimes there is a need to support 100,000 TPS, how to do? The solution is to add an MQ. During the activity, when there are 100,000 requests, the request data will be put into MQ, and system A will take out the data in batches for processing. If the system’s processing capacity is 20,000 per second, it will only take 5 seconds to process all the data. This way, you don’t have to deal with 100,000 TPS at the same time, causing the system or data to crash under the load to the system processing.

​

Second, MQ problems and solutions

1. System complexity increases and availability decreases

If you add one more component to the system, you need to consider whether MQ itself will fail. If MQ itself fails, the entire system will become unavailable. Each additional component adds to the complexity of the system.

2. Data consistency

How to ensure the consistency of data when one of the services fails

Solution: Use distributed transactions

3. Common MQ problem solving methods

  • How to improve reliability?

Solution: Set up an MQ mirroring cluster

  • How to ensure that messages are not lost?

Solution: After each message is sent to MQ, MQ processes the data processing of the notification sender. For example, the message producer will resend the message if it does not receive a reply within a specified time after sending the message.

  • How do I ensure that messages are not consumed repeatedly?

Solution: Add a unique ID to the message, and each time the consumer pulls the message from MQ, it checks to see if it already exists, and if it does, it discards it.

  • How to ensure the order of message delivery?

For example, an order generates three messages, namely the creation of order, payment, inventory deduction, consumption needs to be in order to consume

Solution: Let three messages in an order be sent to the same queue, and block the queue for each message retrieved at consumption until the lock is released after the first message is processed

3. MQ is not applicable

User requests that require the return of all data dependent on system processing cannot be processed by MQ, such as user registration.

MQ test concerns

  • Message Producer testing

    1. Whether the message is correctly pushed to the expected queue

    2. Whether the message is pushed to the correct topic

    3. Whether the same message is received after repeated sending

    4, the number of messages sent exceeds the total length of the queue is the data processing method

    5. How is the queue of messages pushed to each topic distributed

  • Message consumer testing

    1. Whether consumer consumption patterns are consistent with expectations

    2. Whether consumers consume data from the expected queue

    3. Consumption queue policies for topics set by consumers

    4. Whether the message is deleted in time after consumption (see whether repeated consumption is allowed)

    5. The number of consumed messages cannot catch up with the speed of message production

    6. Pull and push consumption type testing

  • Message persistence testing

    1. Different MQS persist messages in different ways

  • Reliability test

    1. The Broker closes properly

    2. Is the consumption of subscriptions and broadcast messages consistent with expectations

    3. Any node in the cluster is faulty

Five, commonly used MQ product features

features ActiveMQ RabbitMQ RocketMQ Kafka
Single machine throughput The throughput is an order of magnitude lower than RocketMQ and Kafka The throughput is an order of magnitude lower than RocketMQ and Kafka At class 100,000, RocketMQ is also a MQ that can support high throughput This is the greatest advantage of Kafka, which is its throughput. Real-time data calculation and log collection for common configuration and data systems
timeliness Ms level The subtlety level, which is a major feature of RabbitMQ, is the lowest latency Ms level The delay is within the MS level
availability High availability based on master-slave architecture High, based on master – slave architecture to achieve high availability Very tall, distributed architecture Very high, Kafka is distributed, multiple copies of the same data, a few machines down, no loss of data, no unavailability
Message reliability There is a low probability of losing data No message loss After parameter optimization configuration, 0 can be lost After parameter optimization, 0 is lost
The core features The MQ domain is extremely functional Based on Erlang development, so the concurrency ability is strong, the performance is very good, the delay is very low MQ function is more complete, or distributed, good scalability It has simple functions, mainly supports simple MQ function, and is widely used in real-time computing and log collection in the field of big data. It is the standard in real-time.
It is very mature and powerful, and has been applied to a large number of companies and projects in the industry. But the probability of occasional message loss, and now the community and domestic applications are less and less, the official community to activemQ5.x maintenance is less and less, and indeed is mainly based on decoupling and asynchronous to use, less used in large-scale throughput scenarios Erlang language development, performance and its good, low latency. And the open source version, the management interface is very good, some domestic Internet companies in recent years to use RabbitMQ is more, especially suitable for small and medium-sized companies obvious shortcomings, that is, the throughput will be low, this is because it does the implementation mechanism comparison, because the use of Erlang development, Not many companies currently use it for development. So customizing for the source code community is very difficult, so the company’s control is very weak, and it depends on the open source community to maintain it. Interface simple and easy to use, after all, large-scale application in ali, ali platform security, day processing messages on billions of mass throughput can be done, the performance is very good also, the distributed extended very convenient also, community can maintenance, reliability and availability are OK, you can also support a massive number of topic, MQ support complex business scenarios. It provides very few core functions, but it provides extremely high throughput, MS-level latency, extremely high availability and reliability, and is distributed to scale arbitrarily. At the same time, It is better for Kafka to support a small number of topics to ensure high throughput.

Welcome to pay attention to my subscription number, will regularly share some software testing related articles, questions are also welcome to discuss learning!

​

​