Messages with the same key value are written to the same partition (the messages in the partition are ordered). The messages in one partition are consumed by only one consumer.
If a consumer is consumed by multiple threads, pull messages need to be written to different memory queues by key value, messages with the same key value are written to the same memory queue (messages in the memory queue are ordered), and then one thread consumes one memory queue.
1, the rabbitMq
Problem analysis:
As shown in the figure, datA1 and datA2 are sequential, datA1 must be executed before datA2; These two data are consumed by different consumers, maybe datA2 is executed first, datA1 is executed later, so the original order is out of order.
Solution:
As shown in the figure, multiple queues are created in MQ, and the data of the same rule (hash unique identifiers) are placed in the queue in order. The consumers only fetch data from one queue in order. Or there is only one queue corresponding to one consumer, which is queued internally by the memory queue and then distributed to different workers at the bottom for processing.
2, kafka
Problem analysis:
As shown here, in Kafka,You assign a key to the data, and the data goes to the same partition, where the data is ordered. From here there is nothing wrong, the data inserted into the database is in order.
However, we may use multithreading on the consumer side because of the slow processing speed of a single thread. In order to speed up the processing time and throughput, thread is used for processing. When threads are added to the consumer side, an out-of-order situation occurs.
Figure, is the use of multi-threading, data order inconsistent situation.
After using multithreading, how to solve the data order problem?
As shown in the figure, memory queue is used at the consumer side. The data in the queue is distributed using hash, and there is a queue for each thread to ensure the order of data.
3, rocketMq
As shown in the figure, the producer moulded the orderId and put the data of the same mould into the Messagequeue. The consumers consume the same Messagequeue. As long as the consumers consume in order, the data can be consumed in order.
4, activeMq
As shown in the figure, activeMq contains the messageGroups attribute, which can specify JMSXGroupID, and consumers will consume the specified JMSXGroupID. This ensures the sequence and solves the problem of load balancing.
end