It can be seen that the wind direction of the technology circle has been changing, the heat of big data and cloud has been gradually fading, and now AI and IoT are hot. These hot concepts will eventually be transformed from papers and POWERPOINT presentations into systems that can actually solve problems. Otherwise, they are just castles in the air. So what’s constant? (Some digressions)

What is the difference between topics and queues?

The original message queue is a strict queue

  • Consumers are actually competing with each other, and each consumer can only receive a portion of the messages in the queue

If a message data needs to be distributed to multiple consumers, each consumer is required to receive the full message. For example, for an order data, risk control systems, analysis systems, payment systems, and so on all need to receive messages. At this point, a single queue is not sufficient, and a possible solution is to create a separate queue for each consumer and have producers send multiple copies (bad practice).

To solve this problem, another message model has evolved: the publish-subscribe Pattern.

In the publish-subscribe model, the sender of the message is called Publisher, the receiver of the message is called Subscriber, and the container where the message is stored on the server is called Topic. Publishers send messages to topics, and subscribers need to “subscribe to topics” before receiving messages. “Subscribe” here is both an action and a logical copy of the topic at consumption, with each subscription allowing the subscriber to receive all messages for the topic.

Most modern message queue products use this publish-subscribe model

The message model for RabbitMQ

It is one of the few products that still adheres to the queue model.

In RabbitMQ, the Exchange sits between the producer and the queue. The producer does not care which queue the message is sent to. The producer sends the message to the Exchange, and the policy configured on the Exchange determines which queue the message is sent to.

If the same message needs to be consumed by multiple consumers, Exchange needs to be configured to send the message to multiple queues. Each queue holds a complete piece of message data and can provide consumption service for a single consumer. This is also a way of realizing the functionality of the new publish-subscribe model, where a single piece of message data can be consumed multiple times by multiple subscribers.

The message model for RocketMQ

The messaging model used by RocketMQ is the standard publish-subscribe model

The validation mechanism does a good job of ensuring reliability during message delivery, but its introduction presents a small problem on the consumer side. To ensure the orderliness of messages, the next message cannot be consumed until a message is successfully consumed. Otherwise, message emptiness occurs, violating the orderliness principle.

In other words, each theme can only have at most one consumer instance consuming at any time, so it is impossible to improve the overall consumption performance of the consumer end by horizontally expanding the number of consumers. To solve this problem, RocketMQ adds the concept of queues under topics.

  • Each topic contains multiple queues through which multi-instance parallel production and consumption can be achieved

  • RocketMQ ensures that messages are ordered only on queues, not strictly ordered at the topic level (ordered in the same queue, out of order between queues)

In RocketMQ, the concept of a subscriber is represented by a Consumer Group. Each Consumer group has a complete message in the consumption topic, and consumption progress among different Consumer groups is not affected by each other. That is to say, a message once consumed by Consumer Group1 will also be consumed by Consumer Group2.

A consumer group contains multiple consumers, and the consumers in the same group are competing consumers. Each consumer is responsible for part of the messages in the consumer group. If a message is consumed by consumer Consumer1, no other consumers in the same group will receive the message.

During Topic consumption, since messages need to be consumed multiple times by different groups, they are not immediately deleted. This requires RocketMQ to maintain a Consumer Offset on each queue for each Consumer group, where all previous messages have been consumed. None of the subsequent messages are consumed, and for each successful message consumed, the consumption position is increased by one. This consumption location is a very important concept. When we use message queues, most of the lost messages are caused by improper consumption location.

Kafka’s message model

Kafka’s message model is identical to RocketMQ’s.

The only difference is that in Kafka, the concept of a queue is called a Partition.

conclusion

  • Topic: Publish-subscribe

  • Queue: First in, first out

Business models are not implementation-level models. For example, MySQL and Hbase also support SQL databases. In their service models, the units storing data are “tables”. However, at the implementation level, no database stores data in the form of two-dimensional tables. HBase uses the KV structure for storage. Similarly, business models like Kafka and RocketMQ are basically the same, not that their implementations are the same; in fact, the implementations of the two message queues are completely different.

Phase to recommend

  • What are pessimistic locks and optimistic locks in MySQL?
  • Into the Black Box: How is SQL executed in the database?
  • Hash algorithm principle analysis
  • Consistent hashing design idea
  • Explains Redis cache penetration, cache breakdown, and cache avalanche with solutions
  • Faced with massive data, how can we search faster?