1. Definition of queues
Wikipedia defines it this way:
Queues are Linear lists of first-in-first-out (FIFO) lists. In the specific application commonly used linked list or array to achieve. Queues only allow insertion at the back end (called rear) and deletion at the front (called front).
There are several key points in this definition. The first is first-in, first-out (FIFO). The implicit requirement is to ensure that the messages are written to the queue in the same order that they are read from the queue. However, the queue does not have a “read” operation, which is to unqueue, or “delete” the message from the queue.
2. Message model
Early message queues were designed according to the data structure of “queues”. Taking a look at this diagram, Producer sending messages is the action of joining the queue, while Consumer receiving messages is the action of deleting from the queue. The container of messages stored on the server is naturally called a queue.
This is one of the original message models:Queuing models.
If multiple producers send messages to the same queue, the messages that can be consumed in the queue are the sum of all the messages produced by these producers. The order of messages is the natural order in which these producers send messages. If multiple consumers receive messages in the same queue, the relationship between these consumers is actually competitive, each consumer can only receive a part of the message in the queue, that is, any message can only be received by one of the consumers.
If a message data needs to be distributed to multiple consumers, each consumer is required to receive the full message. For example, for an order data, risk control systems, analysis systems, payment systems, and so on all need to receive messages. At this point, a single queue is not sufficient, and a possible solution is to create a separate queue for each consumer and let producers send multiple copies. Copying the same message data to multiple queues wastes resources, and more importantly, producers must know how many consumers there are. Sending a single message for each consumer actually defeats the design of message queue decoupling.
To solve this problem, another message model has evolved:Publish – subscribe model(PublishSubscribe Pattern).
For a long time in the history of messaging, the queue pattern and the publish-subscribe pattern co-existed, and some message queues supported both message models, such as ActiveMQ. If we compare the two models carefully, producers are publishers, consumers are subscribers, and queues are topics, there is no essential difference. The biggest difference is whether a single piece of message data can be consumed more than once. In fact, if there is only one subscriber in this publish-subscribe model, it is essentially the same as the queue model. That is, the publish-subscribe model is functionally compatible with the queue model.
Most modern message queue products use this publish-subscribe model, with some exceptions.
2. Message models for different queues
2.1 RabbitMQ message model
The exception is RabbitMQ, which is one of the few products that still sticks with the queue model. So how does it solve the problem of multiple consumers?In RabbitMQ, the Exchange sits between the producer and the queue. The producer does not care which queue the message is sent to. The producer sends the message to the Exchange, and the policy configured on the Exchange determines which queue the message is sent to.
2.2 RocketMQ’s message model
The message model used by RocketMQ is the standard publish-subscribe model, and in RocketMQ’s glossary, producers, consumers, and topics are the same concepts as in the publish-subscribe model described above.
However, there is also the concept of Queue in RocketMQ, and queues are a very important concept in RocketMQ. What is the role of queues in RocketMQ? This starts with the consumption mechanism of message queues.
Almost all message queuing products use a very naive “request-acknowledge” mechanism to ensure that messages are not lost during delivery due to network or server failures. The specific approach is also very simple. On the production side, the producer sends the message to the server, or Broker, which sends an acknowledgement to the producer after receiving the message and writing it to a topic or queue.
If the producer receives no acknowledgement or a failed response from the server, it resends the message. On the consumption side, consumers to buy the messages are received and completed its business logic (for example, to save the data in the database), will send the service side consumption successful validation, the service side only after receiving the consumer to confirm, to think that a message has been successfully consumption, otherwise it will be to send this message to consumers, until receive the corresponding consumption successful validation.
This validation mechanism does a good job of ensuring the reliability of the messaging process, but its introduction presents a small problem on the consumer side. What’s the problem? To ensure the orderliness of messages, the next message cannot be consumed until a message is successfully consumed. Otherwise, message emptiness occurs, violating the orderliness principle.
In other words, each theme can only have at most one consumer instance consuming at any time, so it is impossible to improve the overall consumption performance of the consumer end by horizontally expanding the number of consumers. To solve this problem, RocketMQ adds the concept of queues under topics.
Each topic contains multiple queues through which multi-instance parallel production and consumption can be achieved. Note that RocketMQ guarantees message ordering only on queues, not strictly ordered messages at the topic level.
In RocketMQ, the concept of a subscriber is represented by a Consumer Group. Each Consumer group has a complete message in the consumption topic, and consumption progress among different Consumer groups is not affected by each other. That is to say, a message once consumed by Consumer Group1 will also be consumed by Consumer Group2.
A consumer group contains multiple consumers, and the consumers in the same group are competing consumers. Each consumer is responsible for part of the messages in the consumer group. If a message is consumed by consumer Consumer1, no other consumers in the same group will receive the message.
During Topic consumption, since messages need to be consumed multiple times by different groups, they are not immediately deleted. This requires RocketMQ to maintain a Consumer Offset on each queue for each Consumer group, where all previous messages have been consumed. None of the subsequent messages are consumed, and for each successful message consumed, the consumption position is increased by one. This consumption location is a very important concept. When we use message queues, most of the lost messages are caused by improper consumption location.
These are the key concepts in RocketMQ’s message model. To help you understand, I drew the following picture:
2.3 Kafka’s message model
Taking a look at Kafka, another popular message queue, Kafka’s message model is exactly the same as RocketMQ, and all of the RocketMQ concepts AND validation mechanisms that I’ve just described apply to Kafka. The only difference is that Kafka uses a different name for a queue. Kafka uses the same name as a Partition.