There are many kinds of message queues, such as Kafaka, rocketMq, rabbitMq, and all of them have similar models, including concepts such as Queue, Topic, and Partition. In this article, we will explain the meaning of these concepts in the message queue. See: Geek Time – Message Queue Expert

Queuing models

The picture above is the earliest message model and the most intuitive message model. When I first understood the message model, this is how I understood it. Consumers send messages to queues and producers consume messages from queues. If there are multiple consumers, each consumer is a competing relationship in this model, and each message can only be consumed by one consumer.

Many scenarios require a single message to be consumed by multiple consumers (messages from upstream systems, multiple downstream systems), hence the publish-subscribe model

Publish and subscribe model

  • The sender of the message is called the Publisher
  • The receiver of the message is called Subscriber
  • The container in which the messages are stored on the server is called a topic

Publishers send messages to topics, and subscribers need to “subscribe to topics” before receiving messages. “Subscribe” here is both an action and a logical copy of the topic at consumption, with each subscription allowing the subscriber to receive all messages for the topic.

The model of the RockteMQ

RocketMQ uses the publish-subscribe model, which is shown below

You can see the concept of a queue in rocketMq. Message queues have a “request-acknowledgement” mechanism to ensure that messages are not lost. (On the production side, the producer sends the message to the Broker, which receives the message and writes it to a topic or queue. If the producer receives no acknowledgement or a failed response from the server, it resends the message. On the consumption side, consumers to buy the messages are received and completed its business logic (for example, to save the data in the database), will send the service side consumption successful validation, the service side only after receiving the consumer to confirm, to think that a message has been successfully consumption, otherwise it will be to send this message to consumers, until receive the corresponding consumption successful validation)

There are also some problems with this mechanism: in order to ensure that the message is ordered, the next message cannot be consumed until one message is successfully consumed, otherwise the message will be empty, which violates the principle of order. In other words, each theme can only have at most one consumer instance consuming at any time, so it is impossible to improve the overall consumption performance of the consumer end by horizontally expanding the number of consumers. To solve this problem, RocketMQ adds the concept of queues under topics

Producers will send messages to all queues (you can think of it as a database and table mode, certain data will be inserted into a table according to a certain rule). When consuming, it can be consumed from multiple queues, which is equivalent to parallel consumption, and the parallelism is the number of queues

However, this does not guarantee ordering of message consumption, which can be ensured by using sequential topics or by ensuring that messages are stored on a queue or partition through the production side

Each topic contains multiple queues through which multi-instance parallel production and consumption can be achieved. Note that RocketMQ guarantees message ordering only on queues, not strictly ordered messages at the topic level

During Topic consumption, since messages need to be consumed multiple times by different groups, they are not immediately deleted. This requires RocketMQ to maintain a Consumer Offset on each queue for each Consumer group, where all previous messages have been consumed. None of the subsequent messages are consumed, and for each successful message consumed, the consumption position is increased by one. This consumption location is a very important concept. When we use message queues, most of the lost messages are caused by improper consumption location.

Kafka’s message model

The message model for Kafka is exactly the same as RocketMQ, and all of the RocketMQ concepts I’ve just described, and the validation mechanism for production and consumption, apply exactly to Kafka. The only difference is that Kafka uses a different name for a queue. Kafka uses the same name as a Partition.

The original address

Cbaj. Gitee. IO/blog / 2020/0…