What are the scenarios for message loss?

To analyze a RabbitMQ message loss scenario, take a look at the process of sending a message from producer to consumer.

From the diagram, we can see the whole process of the message from production to consumption:

① The producer sends a message to the switch.

② The switch routes messages to queues.

③ Consumers consume messages from the queue.

Each of the above three steps can result in message loss.

1. The message is lost during message sending

When a producer sends a message to the switch, the message may fail to reach the switch due to network fluctuation, resulting in message loss.

Solution: Sender confirmation mechanism, whereby RabbitMQ sends an acknowledgement message to the producer after the message has correctly arrived at the switch

Code example: Use Spring Boot to integrate RabbitMQ as an example

Add the following configuration:

# on the sender to confirm the mechanism spring. The rabbitmq. Publisher - confirms = trueCopy the code

Create a Controller interface to test message sending

public class RabbitMQController {
    @Autowired
    private RabbitTemplate rabbitTemplate;

    @PostMapping("/sendMessage")
    public void sendMessage(a){		
        // Set the acknowledgement callback method
        rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
            if(ack){
                log.info("Message sent successfully, message ID: {}", correlationData.getId());
            }else {
                log.info("Message failed to be sent, message ID: {}, failure cause: {}", correlationData.getId(), cause); }}); String uuid = UUID.randomUUID().toString(); log.info("Send message ID: {}", uuid);

        rabbitTemplate.convertAndSend( "test-routing-key", (Object) "this is a message".newCorrelationData(uuid)); }}Copy the code

Here we do not specify the switch to which the message is sent, but the DC switch to which RabbitMQ defaults to amq.direct.

Using the Postman test interface:

You can see that the callback method is invoked and the message reaches the switch successfully.

Send failure test:

We send the message by specifying a non-existent switch, simulating a failure to send

rabbitTemplate.convertAndSend("not-exists-exchange", "test-routing-key", (Object) "this is a message", new CorrelationData(uuid));
Copy the code

The RabbitMQ server cannot find the corresponding switch and returns an acknowledgement failure (ACK = false).

Tips: So far we have not created a queue. The tests we performed above are simply a producer sending a message to the switch. As long as the message reaches the switch successfully, it will return a success confirmation regardless of whether there is a queue to receive the message.

2. The message is lost during routing

When a message arrives at the switch, there is no corresponding queue bound to the switch and the message is lost.

Add configuration:

# when the message matches the Queue and fails, Will pass the callback returnCallback method returns a message spring. The rabbitmq. Publisher - returns = true spring. The rabbitmq. Template. Mandatory = trueCopy the code

Set the callback method when no matching queue is found:

rabbitTemplate.setReturnCallback((message, replyCode, replyText,
                                          exchange, routingKey) -> {
            log.info("return message: {}".new String(message.getBody()));
        });
Copy the code

Send a message:

public void sendMessage(){ rabbitTemplate.setConfirmCallback((correlationData, ack, Cause) -> {if(ack){log.info(" Message sent successfully, message ID: {}", correlationdata.getid ()); }else {log.info(" Failed to send message, message ID: {}, failed cause: {}", correlationdata.getid (), cause); }}); RabbitTemplate. SetReturnCallback ((message, replyCode replyText, exchange, routingKey) - > {the info (the "message routing failure: {}", new String(message.getBody())); }); String uuid = UUID.randomUUID().toString(); Log.info (" send message, ID: {}", uuid); rabbitTemplate.convertAndSend( "test-routing-key", (Object) "this is a message", new CorrelationData(uuid)); }Copy the code

At this point we have not created the bind queue, and the switch cannot post messages to the queue

As you can see, the message successfully arrived at the switch, the acknowledgement method was called back, and the message was successfully acknowledged. On the other hand, the ReturnCallback method is also called back because no binding queue can be found, indicating a message routing failure.

3. Messages are lost during message consumption

By default, after receiving a message from the queue, the consumer will directly confirm receipt, that is, automatic receipt. After receiving the message, the message will be removed from the queue. Messages can be lost if an outage or a program exception occurs while the consumer is processing received messages.

Consumer add configuration:

Manual confirmation # open consumers spring. The rabbitmq. Listener. Simple. Acknowledge - mode = manualCopy the code

Create a “test-queue” queue on the RabbitMQ console and bind it to the “test-exchange” switch with a “test-routing-key”.

Consumer Code:

@Service
@Slf4j
@RabbitListener(queues = "test-queue")
public class RabbitConsumer {
    /** * specifies the queue to consume */
    @RabbitHandler
    public void consume(String msg, Message message, Channel channel) throws IOException {
        log.info("Received message: {}", msg);
        // Use division by 0 runtime exception to simulate program exception
        int i = 2 / 0;
        // Confirm manually after the processing is complete
        log.info("Processed, send confirmation.");
        channel.basicAck(message.getMessageProperties().getDeliveryTag(), false); }}Copy the code

Here we simulate program exceptions with a division by 0 runtime exception

Start the producer and consumer, postman calls the interface that sends the message to send the message, and the consumer throws a runtime exception to view the message in the queue through the console as follows:

You can see that the message is still in the queue and has not been deleted.

We remove the division by 0 code so that the message can be properly acknowledged, restart the consumer, and see the message being re-consumed

Looking at the queue again, it shows that the message has been deleted

4. The message is lost because the message is not persisted

RabbitMQ provides a persistence mechanism for switches, queues, and messages. Ensure that data is not lost when RabbitMQ is down.

4.1 Persistence of switches and queues

How can we create switches and queues that are not configured for persistence and will be lost when RabbitMQ goes down and messages will not be delivered correctly?

In Spring Boot RabbitMQ integration, we can use configuration classes to create switches and queues, as shown below:

Both constructors pass in only names and are persistent by default.

4.2 Message persistence

Messages sent using rabbitTemplate in Spring Boot are persistent by default because their default setting delivery_model = 2.

Tips:

  • Since messages are attached to queues, message persistence must first be set up to persist queues.
  • Because it takes time for messages to be written from memory to disk, if an outage occurs during this time, messages that are not written to disk will be lost.

Persistent summary:

  1. Both queues and messages can persist after the RabbitMQ service is restarted.
  2. If the RabbitMQ service is restarted, the queue will exist and messages in it will be lost.
  3. Set only the persistence of messages, and when the RabbitMQ service restarts, the messages are lost as the queue disappears.