With a multi-dimensional comparison of the five major distributed MQ message queues, mom no longer had to worry about my technology selection

This article is quoted from the public account “DBAplus community” “Distributed message queue differentiation summary, too complete!” Article content, original link: mp.weixin.qq.com/s/-cv9Z0_KPJ8sJ95v43XapQ.

1, the introduction

2. Relevant materials

Official website address:

Kafka：kafka.apache.org/
RabbitMQ：www.rabbitmq.com/
ZeroMQ：zeromq.org/
RocketMQ：rocketmq.apache.org/
ActiveMQ：activemq.apache.org/

Related articles:

IM development basics tutorial (5) : Easy to understand, correctly understand and use MQ message queue
Selection of MQ Messaging middleware for IM systems: Kafka or RabbitMQ?

3. Dimension 1: Data documents

1) Kafka: Medium amount of data. There are Kafka’s own books and some online sources.
2) RabbitMQ: Lots of data. There are some good books and lots of information online.
3) ZeroMQ: Less data. There are few books dedicated to ZeroMQ, and the web is full of code implementation and brief introduction.
4) RocketMQ: Less data. There are currently two books devoted to RocketMQ; The online literature is spotty, and official documentation is brief but light on technical details.
5) ActiveMQ: The amount of data is large. There are no books specifically written about ActiveMQ, and there are many online materials.

Dimension 2: Development language

1) Kafka: Scala
2) RabbitMQ: Erlang
3) ZeroMQ: C language
4) RocketMQ: Java
5) ActiveMQ: Java

Dimension 3: Supported protocols

1) Kafka: A self-defined set of… (Based on TCP)
2) RabbitMQ: AMQP
3) ZeroMQ: TCP and UDP
4) RocketMQ: A self-defined set…
5) ActiveMQ: OpenWire, STOMP, REST, XMPP, AMQP

Dimension 4: Message storage

1) Kafka:

2) the RabbitMQ:

3) ZeroMQ:

4) RocketMQ:

5) ActiveMQ:

Dimension 5: Message transactions

1) Kafka: Supported
2) RabbitMQ: Supported. The client sets the channel to transaction mode, and the transaction will commit only if the message is received by RabbitMQ, otherwise it will be rolled back if an exception is caught. Using transactions degrades performance
3) ZeroMQ: Not supported
4) RocketMQ: Supported
5) ActiveMQ: support

8. Dimension 6: Load balancing

8.1 Kafka

8.2 the RabbitMQ

Client balancing algorithm:

A. Polling: Returns the connection address of the next server in sequence.
B. Weighted polling method: assign higher weight to machines with high configuration and low load, so that they can handle more requests; The machine with low configuration and high load is assigned a lower weight to reduce its system load.
C. Random method: Randomly select the connection address of a server.
D. Weighted random method: select the connection address randomly according to the probability.
E. Source address hash: a value calculated by the hash function, which is used to modulus the size of the server list.
F. Minimum number of connections: Dynamically selects the connection address of the server with the least number of current connections.

8.3 ZeroMQ

8.4 RocketMQ

8.5 ActiveMQ

9. Dimension 7: Cluster mode

1) Kafka:

2) the RabbitMQ:

3) ZeroMQ:

4) RocketMQ:

5) ActiveMQ:

10. Dimension 8: Management interface

1) Kafka: Average
2) RabbitMQ: Ok
3) ZeroMQ: None
4) RocketMQ: There is a management background, but it is not built into the project. You need to start a separate instance of the management background
5) ActiveMQ: General

11. Dimension 9: Availability

1) Kafka: Very tall (distributed)
2) RabbitMQ: high (master/slave)
3) ZeroMQ: High
4) RocketMQ: Very high (distributed)
5) ActiveMQ: high (master slave)

Dimension 10: Message duplication

Kafka: supports at least once, at most once
2) RabbitMQ: supports at least once and at most once
3) ZeroMQ: There is only retransmission mechanism, but there is no persistence. Retransmission is useless even if the message is lost. At least once/at most once/exactly only once
4) RocketMQ: supports at least once
5) ActiveMQ: supports at least once

13. Dimension 11: Throughput TPS

1) Kafka: maximum. Kafka sends messages and consumes messages in batches. The sender merges small messages and sends them in batches to the Broker, and the consumer takes out batches of messages at a time.
2) RabbitMQ: Large
3) ZeroMQ: Maximum
4) RocketMQ: large. The RocketMQ receiver can consume messages in bulk and can configure the number of messages consumed at a time, but the sender does not send messages in bulk.
5) ActiveMQ: relatively large

Dimension 12: Subscription form and message distribution

14.1 Kafka

1) Send:

2) Receiving:

14.2 the RabbitMQ

1) Send:

You declare a queue, and that queue will be created or has been created, and a queue is the basic storage unit.
It is up to the Exchange and key to decide which queue the message is stored in.
Direct > sends to the queue that exactly matches the bindingKey.
Topic > Routing keys are strings containing “.” and are sent to queues corresponding to fuzzy matching bingkeys containing “*” and “# “.
Fanout > is sent to all queues bound to Exchange regardless of key
Headers > has nothing to do with key. Messages are sent to this queue if the HEADERS attribute (a key-value pair) and the bound key-value pair match exactly. This method has low performance and is generally not used

2) Receiving:

14.3 ZeroMQ

14.4 RocketMQ

1) Send:

2) Receiving:

14.5 ActiveMQ

1) Send:

Point-to-point mode: specify a queue that will be created or has already been created.
Publish/subscribe: specify a topic that will be created or has already been created.

2) Receiving:

Point-to-point mode: For queues that have been created, the consumer specifies which queue to receive messages from.
Publish/subscribe: For topics that have been created, the consumer specifies which topic to subscribe to.

Dimension 13: Sequential messages

Kafka:

The RabbitMQ:

ZeroMQ:

RocketMQ:

ActiveMQ:

Dimension 14: message acknowledgement

16.1 Kafka

1) Sender confirmation mechanism:

Ack =0, regardless of whether the message was successfully written to the partition
Ack =1, the message is successfully written to the leader partition, return success
Ack =all, return success if the message is successfully written to all partitions.

2) Recipient confirmation mechanism:

Offsets are automatically or manually submitted. Earlier versions of Kafka are submitted to Zookeeper, which puts a lot of pressure on Zookeeper. Newer versions of Kafka are submitted to The Kafka server, which is no longer dependent on the Zookeeper group and provides stable cluster performance.

16.2 the RabbitMQ

1) Sender confirmation mechanism, after the message is delivered to all matching queues, return success. If the message and queue are persistent, success is returned after writing to disk. Batch confirmation and asynchronous confirmation are supported.
2) Receiver confirmation mechanism: Set autoAck to false, which requires explicit confirmation; set autoAck to true, which automatically confirms.

16.3 ZeroMQ

16.4 RocketMQ

16.5 ActiveMQ

Dimension 15: Message backtracking

1) Kafka: supports backtracking of the specified partition offset position
2) RabbitMQ: Not supported
3) ZeroMQ: Not supported
4) RocketMQ: Supports backtracking at a specified point in time
5) ActiveMQ: not supported

Dimension 16: message retry

18.1 Kafka

18.2 the RabbitMQ

18.3 ZeroMQ

18.4 RocketMQ

1) The send method on the sending end supports internal retry. The retry logic is as follows:

Retry a maximum of 3 times;
If sending fails, it is forwarded to the next broker.
The total time of this method does not exceed the value set by sendMsgTimeout. The default value is 10 seconds.

2) Receiving end:

18.5 ActiveMQ

19. Dimension 17: Concurrency

19.1 Kafka

19.2 the RabbitMQ

19.3 ZeroMQ

19.4 RocketMQ

Modify the consumption parallelism method:

Under the same ConsumerGroup, the parallelism is increased by increasing the number of Consumer instances. Consumer instances that exceed the number of subscription queues are invalid.
Increase the consumption of a single Consumer in parallel threads by modifying the parameters consumeThreadMin and consumeThreadMax

19.5 ActiveMQ

Appendix: More architectural design articles

[1] Articles on IM Architecture design:

Brief talk about IM system architecture design

Briefly describe the pitfalls of mobile IM development: architectural design, communication protocols, and clients

Sharing of a set of mobile TERMINAL IM Architecture Design practice of massive Online Users (with detailed pictures and texts)

A set of original distributed instant messaging (IM) system theoretical architecture scheme

From zero to excellence: The evolution of the technical architecture of JINGdong customer service INSTANT messaging system

Architecture selection for mogujie INSTANT messaging /IM server development

Tencent QQ140 million online users technical challenges and architecture evolution road PPT

Design practice of time-sequence-based cold and hot classification architecture for massive data in wechat background

Speech by technical Director of wechat on Architecture: The Way of wechat — Avenue to Simplicity

How to interpret “wechat Technical Director on Architecture: The Way of wechat — The Road to Simplicity”

Fast Fission: Witness the evolution of wechat’s powerful background architecture from 0 to 1 (I)

17 years of practice: Technical methodology of Tencent’s massive products

How to ensure the efficiency and real time of pushing large-scale group messages in MOBILE IM?

Discussion on synchronization and storage scheme of chat messages in modern IM system

How to design a large number of image file server storage architecture?

Quick understanding of server database read/write separation principles and practical suggestions

Understand cookies, sessions, and tokens in HTTP short connections

WhatsApp technology Practice Sharing: Technology myth created by 32 engineering team

Technical challenges and practice summary behind 100 billion visits of wechat circle of Friends

Behind king of Glory’s 200 million users: product positioning, technical architecture, network scheme, etc

Selection of MQ messaging middleware for IM systems: Kafka or RabbitMQ?

Tencent senior architect dry goods summary: one article to understand the design of large distributed system aspects

Taking microblog application scenarios as an example, this paper summarizes the architectural design steps of massive social system

Quickly understand the principles of high-performance HTTP server load balancing

Bullet SMS behind the bright: netease yunxin chief architect to share 100 million IM platform technology practice

Zhihu technology sharing: From single machine to 20 million QPS concurrent Redis high performance cache practice road

IM development basics tutorial (five) : easy to understand, correctly understand and use MQ message queues

Wechat Technology Sharing: Practice of generating serial number of wechat’s Massive IM Chat Messages (Algorithm Principle)

Wechat Technology Sharing: Practice of generating serial numbers of wechat’s massive IM Chat Messages (Disaster Recovery Solution)

Beginner: Zero-based understanding of evolution history, technical principles, and best practices of large-scale distributed architecture

A set of high availability, easy to scale, high concurrency IM group chat, single chat architecture design practice

Ali Technology Sharing: In-depth insight into the 10-year history of Ali database technology solutions

Ali technology sharing: The arduous growth path of OceanBase, a financial database developed by Ali

Social software red envelope technology decryption (a) : comprehensive decryption QQ red envelope technology scheme – architecture, technical implementation, etc

Social software red envelope technology decryption (II) : decrypt wechat shake a red envelope from 0 to 1 technology evolution

Social software red envelope technology decryption (three) : wechat shake a red envelope rain behind the technical details

Decryption of social software Red envelope technology (IV) : How does wechat red envelope system cope with high concurrency

Decryption of social software Red envelope technology (5) : How does wechat red envelope system achieve high availability

Decryption of social software Red envelope technology (VI) : Evolution practice of storage layer architecture of wechat red envelope system

Social software red envelope technology decryption (seven) : Alipay red envelope mass high concurrency technology practice

Social software red envelope technology decryption (eight) : full decryption micro blog red envelope technology scheme

(9) : Talk about the functional logic, disaster recovery, operation and maintenance, architecture, etc

Instant Messaging beginner: What is Nginx? Can it implement IM load balancing?

Instant Messaging Novice: A quick understanding of RPC technology – basic concepts, principles, and uses

With a multi-dimensional comparison of the five major distributed MQ message queues, mom no longer had to worry about my technology selection

More of the same…

[2] More architectural Design related articles:

Tencent senior architect dry goods summary: one article to understand the design of large distributed system aspects

Quickly understand the principles of high-performance HTTP server load balancing

Bullet SMS behind the bright: netease yunxin chief architect to share 100 million IM platform technology practice

Zhihu technology sharing: From single machine to 20 million QPS concurrent Redis high performance cache practice road

Beginner: Zero-based understanding of evolution history, technical principles, and best practices of large-scale distributed architecture

Ali Technology Sharing: In-depth insight into the 10-year history of Ali database technology solutions

Ali technology sharing: The arduous growth path of OceanBase, a financial database developed by Ali

Dada O2O backend architecture evolution practice: The efforts behind high concurrent requests from 0 to 4000

A good backend architect must know: The most comprehensive optimization solution for MySQL large tables

Xiaomi Technology sharing: Decrypting the evolution and practice of the ten-million-high concurrency architecture of Xiaomi Snap System

Read the distributed architecture under the load balancing technology: classification, principle, algorithm, common solutions, etc

Easy to understand: How to design a database architecture that can support millions of concurrent transactions?

With a multi-dimensional comparison of the five major distributed MQ message queues, mom no longer had to worry about my technology selection

More of the same…