1. The principle of mq

No more data, no less data, no more means no more messages; No less, which means no data loss. If MQ is delivering very core messages that support the core business, then this scenario must not lose data.

2. Data loss scenario

Lost data is generally divided into three types, one is mq lost the message, one is consumed when the message is lost. Rabbitmq and Kafka for data loss scenarios,

A: The producer loses data. When sending data to RabbitMQ, the producer may lose data during transmission due to network problems.

B: RabbitMQ loses data. If rabbitMQ is not persistent, data will be lost once rabbitMQ restarts. You must enable persistence to persist messages to disk, so that if RabbitMQ is down, the data stored will be read automatically after recovery, and generally data will not be lost. Unless extremely rare, RabbitMQ dies before persisting, which can result in some data loss.

C: Consumer lost data. If a consumer application has not processed a purchase, such as a process that has been suspended or restarted, rabbitMQ will assume that it has been consumed and the data is lost.

3. How to solve it

A: The producer lost the message

If the message is not received successfully, the producer will receive an exception, which will roll back the transaction and try to resend the transaction. If a message is received, the transaction can be committed.

Note: transactions in AMQP protocol only refer to the transaction mechanism in which a producer sends a message to a broker, and do not include the process on the consumer side.

channel.txSelect(); // start things try{// send messages}catch(Exection e){channel.txrollback (); // Rollback // recommit} Disadvantages: When rabbitMQ transactions are started, they will block synchronously, and producers will block to see if they can be successfully sent, which can lead to throughput degradation.

② : Confirm mode can be enabled. After the producer has enabled confirm mode, a unique ID is assigned to each message written to RabbitMQ, and rabbitMQ will send you an ACK message telling you that the message is OK to send. If RabbitMQ fails to process the message, it will call back to an NACK interface telling you that the message failed and you can retry. And you can use this mechanism to know that you maintain the ID of each message in memory, and if you don’t receive a callback for that message after a certain amount of time, you can resend it.

// Open confirm channel.confirm(); Public void ack(String messageId){} Public void nack(String messageId){Copy the code

The two different

Transactions are synchronous, you commit an item and it blocks, but confirmations are asynchronous, you send a message and the next message can be sent, and RabbitMQ will call back to tell if it was successful. Generally in the producer part to avoid loss, is to use the confirm mechanism.

B: RabbitMQ loses the data itself

Set message persistence to disk. There are two steps to set persistence: 1. Create a queue and set it to persist. This ensures that rabbitMQ will persist the metadata in the queue, but not the data in it. ② Set deliveryMode to 2 to persist messages and RabbitMQ will persist messages to disk. You have to turn both on at the same time.

And persistence can be combined with the production confirm mechanism, so that the producer ACK will not be notified until the message has been persisted to disk, so that even if rabbitMQ dies before persistence, the data will be lost and the producer will resend the message without receiving the ACK callback.

C: Consumers have lost their data

Rabbitmq has both manual and automatic ack mechanisms to deal with consumer data loss: if you use rabbitMQ’s automatic ack mechanism, first turn off RabbitMQ’s automatic ACK and use manual ack, calling the code manually each time you are sure to process the message. This avoids the need to ack messages before they have been processed.

The ACK mechanism, however, can lead to repeated consumption in exceptional cases: when a consumer breaks the connection but does not hang, the broker knows that it has not received an ACK yet, and the message is re-queued, resulting in repeated consumption of data.

The application layer solves the problem of repetition:

  1. Special Map storage: Used to store the execution status of each message (distinguished by msGID). After successful execution, the Map is updated. When another message is repeatedly consumed, the Map data is read to determine the execution status corresponding to the MSGID.
  2. Business logic judgment: after the message is executed, the entity state will be changed, and the entity state will be determined whether it is updated. If it is updated, repeated consumption will not be carried out.

Summary: AMQP provides at-least-once delivery. In exceptional cases, messages will be repeatedly consumed, in which case the business is idempotent (repeated message processing).