Why message queues

Why message queues are used as middleware in a project, and what their role is, is nothing more than their core characteristics. As a kind of middleware for message processing, message queue has three main functions: decoupling asynchronous xiao Feng

The decoupling

When a system or module calls multiple systems or modules, calling each other is complicated and difficult to maintain. At this time, we can decouple multiple systems or modules called into multiple services and asynchronously decouple them through message queue linksCopy the code

asynchronous

Through the decoupling above, we know that the invocation of system or module is linked by message queue. When we call, we just send messages to the message queue between us, without waiting for feedback from the calling system or module. This time is through the asynchronous way, as soon as possible to complete the entire business process.Copy the code

XiaoFeng

When many requests come in, the system cannot process them. At this time, it cannot wait for the completion of one request to process another one, which will take a long time to give feedback to the user. At this time, we can put the request information into the message queue, let the following message slowly to process the large number of messages, so as to achieve the role of Xiao Feng.Copy the code

Disadvantages of introducing message queues

We all know that space and time are often in conflict in programs, and middleware has its drawbacks as well. In a word, ‘what suits you is the best’.

The system availability decreases

The introduction of message queue will lead to the increase of the system's dependence on it. When message queue is unavailable, our system may not be available.Copy the code

Increased system complexity

The introduction of message queue, we need to consider the order of message consumption, loss and other issues.Copy the code

Consistency problem

After the message queue is introduced, we know that the producer only puts the message into the message queue, and we don't know what happens to the consumer after that, whether it succeeds or fails? At this point we need to consider data consistencyCopy the code

Message queue comparison

Characteristics of the ActiveMQ RabbitMQ RocketMQ Kafka
Single machine throughput The throughput is an order of magnitude lower than RocketMQ and Kafka The throughput is an order of magnitude lower than RocketMQ and Kafka At level 100,000, RocketMQ is also an MQ that can support high throughput 100,000 levels, which is the biggest advantage of Kafka, is high throughput.
The impact of number of topics on throughput Topics can be in the hundreds, thousands of levels, with a small drop in throughput and this is one of the great advantages of RocketMQ, being able to support a large number of topics on the same machine Throughput drops dramatically from dozens to hundreds of topics, so Kafka tries to keep the number of topics low on the same machine. If you want to support large-scale topics, you need to add more machine resources
timeliness Ms level Microsecond, which is a feature of RabbitMQ, with the lowest latency Ms level The delay is within ms class
availability High, based on master-slave architecture to achieve high availability High, based on master-slave architecture to achieve high availability Very high, distributed architecture Very high. Kafka is distributed, with multiple copies of a single piece of data, and a few machines down, without data loss and unavailability
Message reliability There is a low probability of data loss After optimized parameter configuration, 0 can be lost After optimized parameter configuration, messages can be 0 lost
Function support The MQ realm is extremely functional Based on Erlang development, so the concurrency is very strong, performance is extremely good, very low latency MQ function is more complete, or distributed, good scalability It supports simple MQ functions. Real-time computing and log collection are used on a large scale in the field of big data and are the de facto standard
Summary of advantages and Disadvantages Very mature, powerful, used in a large number of companies and projects in the industry, occasionally with a low probability of lost messages. Besides, there are fewer and fewer community and domestic applications. The official community maintains less and less ActiveMQ 5.x, and only releases a version in several months. And it is true that it is primarily based on decoupling and asynchronism, and is less likely to be used in large-scale throughput scenarios Erlang language development, extremely good performance, very low delay; Throughput to ten thousand levels, MQ is quite complete and the open source management interface is very good, it is very easy to use Interface is simple and easy to use, and, after all, in large-scale application of ali, with ali brand protection, with tens of billions of news, can do mass throughput performance is very good also, the distributed extended very convenient also, community maintenance can also, reliability and availability are ok, you can also support a massive number of topic, support MQ complex business scenarios, The community activity is relatively average, but it’s ok, the documentation is relatively simple, and the interface piece doesn’t follow the standard JMS specification. Some systems need to change a lot of code to migrate Kafka’s characteristics are obvious: it only provides a small number of core functions, but provides high throughput, ms class latency, high availability and reliability, and distributed can be arbitrarily scalable while Kafka is best to support a small number of topics, to ensure its high throughput. The only disadvantage of Kafka is the possibility of repeated consumption of messages, which has a minimal impact on data accuracy. This is negligible in the big data world and log collection. This feature is natural for real-time big data computation and log collection

conclusion

Most of the projects I have contacted are ActiveMQ, which is not very sensitive to messages and is simply deployed with middleware. The project is mainly about simple message sending and receiving, and ActiveMQ is used. The other message queues have been played by themselves, and each has its own strengths and weaknesses. You can use the above table to select the message queue middleware that suits you.Copy the code