A few trivia points about MQ (iii) How to ensure that messages are not consumed twice

1. The idempotence

Idempotent (idempotence) is a mathematical and computer concept commonly found in abstract algebra. The characteristic of an idempotent operation in programming is that any number of executions have the same effect as a single execution. An idempotent function, or idempotent method, is a function that can be executed repeatedly with the same parameters and achieve the same results. These functions do not affect system state, nor do they have to worry about system changes caused by repeated execution. For example, the “setTrue()” function is an idempotent function that gives the same result no matter how many times it is executed. More complex operational idempotent guarantees are implemented using unique transaction numbers (serial numbers).

In simple terms, idempotence is a piece of data or a request that is repeated to you many times, and you have to make sure that the corresponding piece of data doesn’t change, you can’t go wrong.

2. Repeated consumption scenario

(1) First, rabbitMQ, RocketMQ, and Kafka are all prone to repeated message consumption. This problem is usually not guaranteed by MQ, but by consumers themselves. Kafka has a concept called offset. Every message in kafka has an offset representing its serial number. After consuming the data, the consumer submits the offset of the message once in a while. The next time kafka restarts, the consumer will continue to consume at the offset from the last consumption.

However, there are exceptions to this rule. If a consumer consumes data and the offset hangs before it can send the message it has consumed, it will receive duplicate data after restarting.

3. Ensure idempotency (repeated consumption)

To keep the message idempotent, this is handled in conjunction with the type of business. Here are a few ideas for reference:

(1) A set can be maintained in memory. As long as a message is obtained from the message queue, the first query is whether the message is in the set. If the message is consumed, it is discarded directly. If not, it is added to the set after consumption.

(2), how to write database, can take a unique key to the database query, if there is no write, if there is a direct update or discard messages.

(3) If it is written redis that there is no problem, each time is set, natural idempotent.

(4) Let the producer send a message, each message add a global unique ID, and then consume, save the ID in redis. Go to Redis first to check whether there is any consumption, there is no consumption.

(5), the database operation can set up a unique key to prevent the insertion of repeated data, so that the insertion will only report an error and will not insert repeated data.

Previous article how to Ensure high Availability of Message Queues Next article How to Prevent Data Queue Data Loss