In order to achieve the message order, both Producer and Consumer should be considered.
First, when a Producer produces messages, there must be order.
Then, when Consumer consumption, also in order, not disorderly.
Producer and orderly
In a normal messaging system such as RabbitMQ, the queue structure is simple and the Producer sends messages to the queue. The messages that enter the queue must be ordered.
Kafka is special because one of its topics (the concept of queues) is actually divided into partitions.
When the Producer sends messages, the messages are distributed among different partitions.
The Producer sends messages in order, but once they enter the Kafka Topic, the messages don’t necessarily go to the same Partition, so the order is definitely out of order.
If you want messages to be globally ordered within a Topic, you can only set one Partition, which becomes a RabbitMq-like structure.
However, this structure is not consistent with Kafka’s design philosophy, and a Topic with only one Partition loses scalability.
Kafka also supports a local ordering approach, where messages of a particular type are placed into the same Partition, thereby ensuring the order of the group of messages.
When a message is sent, the Partition Key is specified. Kafka hashes the Partition Key and determines which Partition to add.
Therefore, messages with the same Partition Key must be together.
For example, the user ID is used as the key, so that the messages of the same user must be together, which ensures the order of the messages in this group.
Consumer and orderly
The messages in MQ are ordered, so consumers naturally receive them in order.
However, if you use more than one Consumer, you can get out of order.
For example, a RabbitMQ Queue with three consumers will receive messages in order, but their processing speed is different, so the result is likely to be out of order.
If you want to do this in strict order, you can only use one Consumer.
If local ordering is possible, split a previous queue into multiple queues, like Partition keys in Kafka, where the same group of data is placed in the same queue.
In Kafka, a Partition can only correspond to one Consumer, but if a Consumer uses multiple threads, it will have the same effect as multiple consumers.
This will require further refinement of the grouping of messages.
Create a memory queue for each thread. When a Consumer receives a message, it places all messages in the same memory queue for processing by the same thread.
To summarize, messages need to be ordered both producers and consumers.
RabbitMQ has a simple queue structure and messages sent by Producer are ordered. In Kafka, however, a Topic has multiple partitions. If global order is required, only one Partition can be used.
If local ordering is acceptable, you can set the Partition Key for messages so that all messages with the same Hash result will be on the same Partition.
When consuming Consumer, you need to be aware of multiple consumers, such as multiple consuming threads.
The grouping can be refined after the Consumer receives the message, and messages from the same group are sent to the same Consumer thread for processing.
Recommended reading
OAuth2 diagram
Easy to understand the core concepts of Kubernetes
Architecture technology trends that developers must understand: Service Mesh