- The problem
Let’s say we build a topic with three partitions. The producer can specify a key when writing. For example, if we specify an order ID as the key, the data related to the order must be distributed to the same partition, and the data in the partition must be in order. When consumers retrieve data from a partition, there must also be an order. Up to this point, the order is ok, there’s no confusion. Then, we might have multiple threads in the consumer to process messages concurrently. Because if the consumer is single-threaded consuming processing, and the processing is time-consuming, such as processing a message takes tens of ms, then only dozens of messages can be processed per second, which is too low throughput. If multiple threads are running concurrently, the order may be out of order.
- The solution
One topic, one partition, one consumer, internal single-thread consumption, single-thread throughput is too low to use this. Write N memory queues, all data with the same key to the same queue; Then, for N threads, each thread consumes a queue to ensure orderliness.
