Welcome to my personal public account: Architecture Notes of Huishania (ID: Shishan100)

8:30am Monday to Friday! High-quality technical articles sent on time!

directory

(1) Previous information

(2) Review of ACK mechanism

(3) Implementation principle of ACK mechanism: Delivery Tag

(4) How does RabbitMQ sense when a storage service instance is down

(5) Retransmission of messages when storage service processing fails

(6) Phase summary

1. Previous tips

In the last article, how to Guarantee the data loss of the whole link of the message-oriented middleware (1), we introduced the problems left by the technical solution of the data loss of the message-oriented middleware.

One of the biggest problems is that messages sent by producers can get lost.

Messages can be lost in the middle of a network trip due to a network failure, or they can be delivered to MQ’s memory due to a sudden MQ failure.

RabbitMQ actually provides some mechanism for the loss of producer delivery data.

One heavyweight mechanism, for example, is transactional messaging. Using a transaction-like mechanism to deliver messages to MQ can guarantee that messages are not lost, but performance is so poor that it can be hundreds of times worse when tested.

So instead of using this heavy-weight mechanic, you’re going to use the lightweight Confirm mechanic.

However, this article cannot directly explain the confirmation mechanism for producers to ensure message loss, because this mechanism is actually implemented by the ack mechanism similar to the consumer.

So, to understand confirm, we need to start with this article and take a closer look at the underlying mechanism behind the consumer manual ACK mechanism to ensure that messages are not lost.

2. Review of ACK mechanism

In fact, the manual ACK mechanism is very simple. The consumer must ensure that he has processed a message before manually sending an ACK to MQ, which will delete the message after receiving the ACK.

If the consumer is down before sending an ACK, MQ will sense his outage and repost the message to other consumer instances.

This ensures that data will not be lost if the consumer instance goes down.

Once again, if you are not familiar with the ack mechanism, you can go back to the previous article: Heart piercing! How can I ensure 100% data loss when online services are down? . Then in this article, we will continue to explore the implementation of ack mechanism in depth.

3. Implementation principle of ACK mechanism: Delivery Tag

If you code a consumer service to start consuming data from RabbitMQ, the consumer service instance will register itself with RabbitMQ.

So RabbitMQ actually knows which consumer service instances exist.

Let’s take a look at the following figure to get an intuitive feeling:

RabbitMQ will then use its own internal “basic.delivery” method to deliver the message to the storage service for its consumption.

When the message is delivered, an important thing is attached to the message delivery, which you can think of as a unique identifier for the message delivery.

This so-called unique identifier is somewhat similar to an ID, such as the unique ID of a message delivered to a repository service instance. With this unique ID, we can locate a message delivery.

So the Delivery Tag mechanism is easy to ignore. In fact, it is the core foundation of many of the mechanisms we will talk about later.

Another important concept to emphasize here is that every consumer accessing a message from RabbitMQ is done through the concept of a channel.

Looking back at the consumer code snippet below, we must first establish a connection to RabbitMQ deployed on the specified machine and then retrieve a channel through this connection.

And if you remember, all of our storage operations, such as message consumption, ack, etc., are based on this channel, which is a bit like a handle to RabbitMQ, for example:

When I wrote this article about manual ack, I heard a lot of questions about why I would execute a try Finally manual ACK if there was an exception. Actually, it’s very simple. Just add catch.

Okay, let’s move on. You can probably think of a channel as a conduit for data transmission. For each channel, a “delivery tag” uniquely identifies a message delivery, which is roughly a growing number.

Take a look at the following picture, I believe it will be well understood:

If manual ack is used, the warehouse service will actually send an ACK message to the RabbitMQ server with its own delivery tag each time it consumes a message.

The ack code contains a delivery tag.

channel.basicAck(
        delivery.getEnvelope().getDeliveryTag(), 
        false);
Copy the code

RabbitMQ can then uniquely locate a delivery based on which channel and which delivery tag.

That message can then be deleted and marked as processed.

It is important to note that the Delivery tag uniquely identifies message delivery within a channel.

Therefore, when you ack a message, it must be through the same channel that receives the message.

Take a look at the picture below and get a feel for it.

Another important point here is that we can set a parameter and then send ack messages to RabbitMQ in batches to improve overall performance and throughput.

For example, set the second parameter to true in the following line of code.

channel.basicAck(
            delivery.getEnvelope().getDeliveryTag(), 
            true);
Copy the code

This should give you a little more insight into the underlying ack mechanism. At least you know what delivery Tag is, which is the underlying mechanism for implementing ACK.

Then, let’s briefly review the differences between automatic and manual ACK.

It’s actually quite simple to default to automatic ACK. RabbitMQ sends a message to the storage service and immediately marks it as deleted because it does not care whether the storage service received or processed it.

So in this case, the performance is good, but the data is easily lost.

A manual ACK will not be sent to RabbitMQ until the warehouse service has finished dispatching the goods, at which point RabbitMQ will consider the message complete and mark it as deleted.

If the storage service goes down before sending an ACK, RabbitMQ will resend the message to another storage instance to ensure data is not lost.

4. How does RabbitMQ sense when a storage instance is down

Some students have raised this question before, but in fact, to figure out this problem, it is not necessary to explore the bottom layer, as long as their own general thinking and speculation.

If your warehouse service instance receives a message but does not have time to dispatch a shipment or send an ACK, it is down.

If we think about it, RabbitMQ has previously been registered with the repository instance, so there must be some sort of connection between them.

Once a repository instance goes down, RabbitMQ will inevitably sense its outage and send any unack messages to the other repository instances.

So this is a question that we can talk about in depth when we have a chance. Here, we can actually establish this understanding first.

Let’s go back to the architecture diagram below:

5. Resend the message when the storage service fails to process

First, let’s take a look at the following code:

What if a repository instance fails to process a message and enters a catch block? Is it still a direct ACK message?

Of course not, if you are still ACK, that will cause the message to be deleted, but the actual dispatch is not completed.

In that case, isn’t the data still missing? Therefore, the logical approach is to use nACK operations.

To notify RabbitMQ of its failure to process the message, and then ask RabbitMQ to re-deliver the message to another repository instance in an attempt to complete the dispatch.

We simply add the following code to the catch block:

channel.basicNack(
            delivery.getEnvelope().getDeliveryTag(), 
            true);
Copy the code

Note that the second argument is true, which means RabbitMQ should redeliver the message to another repository instance because it failed.

Setting it to false will cause RabbitMQ to know you failed but delete the message anyway, which is not correct.

Again, let’s take a picture to get a feel for it:

6. Phase summary

This article takes a closer look at previous ACK mechanisms, including the underlying Delivery Tag mechanism and the retransmission of messages when message processing fails.

Through the implementation of such mechanisms as ACK mechanism and message retransmission, data will not be lost in the case of sudden breakdown of a consumer service or message processing failure.

End

If there is any harvest, please help to forward, your encouragement is the biggest power of the author, thank you!

A large wave of micro services, distributed, high concurrency, high availability of original series of articles is on the way

Please scan the qr code below and keep following:

Architecture Notes for Hugesia (ID: Shishan100)

More than ten years of EXPERIENCE in BAT architecture

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Internet Interview: How to Ensure that the Full link data of message-oriented middleware is 100% Not lost (2)

1. Previous tips

2. Review of ACK mechanism

3. Implementation principle of ACK mechanism: Delivery Tag

4. How does RabbitMQ sense when a storage instance is down

5. Resend the message when the storage service fails to process

6. Phase summary

End

Internet Interview: How to Ensure that the Full link data of message-oriented middleware is 100% Not lost (2)

1. Previous tips

2. Review of ACK mechanism

3. Implementation principle of ACK mechanism: Delivery Tag

4. How does RabbitMQ sense when a storage instance is down

5. Resend the message when the storage service fails to process

6. Phase summary

End

Related Posts

Python – Data extraction for crawlers

Git merge Internet companies use Git Rebase

SpringBoot customization error handling