Blog.csdn.net/chunlongyu/…

When talking about message-oriented middleware, we usually talk about one feature: the sequential consumption of messages. The problem seems simple: Producer sends messages 1, 2, 3… Consumer press 1, 2, 3… Sequential consumption.

But the reality is that neither RocketMQ nor Kafka guarantees strict ordered consumption of messages by default!

This feature seems simple enough, but why aren’t they guaranteed by default?

How difficult it is to “consume strictly sequentially”

Here are three ways to look at how difficult or impossible strict sequential consumption is for a message-oriented middleware.

The sender

The sender cannot send messages asynchronously. If sending messages asynchronously fails, the order of messages cannot be guaranteed.

Let’s say you say 1,2,3 in a row. After a while, returns 1 failure, 2, 3 success. You send the 1 again, and then the order is out of order.

Storage end

For the storage end, there are the following problems to ensure the order of messages: (1) Messages cannot be partitioned. So one topic can only have one queue. In Kafka, it is called partition; In RocketMQ, it’s called queue. If you have multiple queues, messages from the same topic will be distributed across multiple partitions, with no guarantee of order.

(2) Even if there is only one queue, there will be a second problem. Can I switch to another machine after this machine hangs up? That’s the high availability problem.

Let’s say your current machine is down and there are messages left on it. Switch to another machine at this point, availability is guaranteed. But the message sequence is out of order.

To ensure that, on the one hand, synchronous replication, not asynchronous replication; On the other hand, before cutting the machine, all messages on the dead machine must be consumed without residue. Obviously, this one is hard!!

The receiving end

For the receiver, parallel consumption is not allowed, that is, multi-threading or multiple clients consuming the same queue is not allowed.

conclusion

From the above analysis, you can see how difficult it is to keep messages in strict order!

The problem of sender and receiver is solved by limiting asynchronous sending and parallel consumption. But for the storage side, after the machine hangs up, the problem of switching is very difficult to solve.

If you switch, the message might get messy; If you don’t switch, it’s temporarily unavailable. There’s a tradeoff between these two.

Does the business need global order?

From the above analysis, it can be seen that it is very difficult to ensure the strict order of messages within a topic, or the conditions are very demanding.

So what to do? Must we use all our strength, all our means, to keep the message strictly in order?

Here’s another way to think about it: from a business perspective. As in the following blog post: www.jianshu.com/p/453c6e7ff…

In practice: (1) there are a large number of businesses that do not care about the order; (2) Queue disorder does not mean message disorder.

Clause (2) means: we do not guarantee global order of queues, but local order of messages.

For example: ensure that messages from the same order ID are in order!

Here’s a look at how Kafka and RocketMQ treat this problem:

In Kafka, you can specify topic, partition, and key parameters when sending a message. Partiton and Key are optional.

If you specify a partition, then all messages are sent to the same partition, which is ordered. And on the consumer side, Kafka guarantees that a partition can only be consumed by one consumer.

Or if you specify a key (such as an order ID), all messages with the same key will be sent to the same partition. It’s also ordered.

RocketMQ: RocketMQ takes this one step further with Kafka. We specify (topic, key), but do not specify which queue to send to. In other words, it does not want the business side to insist on a strict global order.

Key point: This opening up actually involves a bigger problem. This is the big difference between RocketMQ and Kafka in the underlying storage. I talked about this in the sequel to Setting Things Right.

This problem will be further examined later in the source code analysis sequence.

So much for message order.

\