Message-oriented middleware is widely used for peak load cutting, system decoupling, and asynchronous processing. Asynchronous processing may be the most commonly used scenario. For example, the current technical blog sites adopt a point system. After a user publishes an article, they can obtain the points they want.

We can put the user ID, need to increase the integral encapsulated into a message delivery to the messaging system, asynchronous processing and integral operation, because it is happened between different servers, the message may delivery failure, processing, and other issues, leading to the user to add points of failure, and one possibility is that news repeat delivery, so the user can repeat integral, Whatever happens, it’s an abnormal situation.

In order to avoid the above two situations, we need to ensure that messages are not lost and messages are consumed only once. This article will talk about how to avoid these two situations from the general level of messaging system, leaving behind the specific message middleware.

1. Ensure that messages are not lost

In the link from production to consumption, messages may be lost in the following three places:

• Failed to deliver a message from the producer to the message queue.
• Message in message queue, persistence failed.
• An exception occurred in the process of the message being consumed by consumers.

1.1 Failed to deliver message during message production

Message producers and message systems are generally deployed independently on different servers. Communication between two servers must be completed through the network. The network is unstable and jitter may occur, and data may be lost. Network jitter occurs in the following two situations.

Scenario 1: Network jitter occurs when a message is transferred to the messaging system and data is lost. Scenario 2: The message has reached the message system, but the network jitter occurs when the message system returns the message to the producer server. At this time, the data may not be really lost, but the producer thinks that the data is lost.

If a message is lost during message production, a redelivery mechanism can be adopted. When the program detects a network anomaly, the message will be delivered to the message system again. However, redelivery in scenario 2 May cause data duplication. How to solve this problem will be mentioned later.

1.2 Failed to persist in message queues

Messaging systems can persist messages, usually to local disk, but there are also a few messaging middleware that support persisting data to a database, and the performance of messaging systems may suffer.

If you are familiar with Redis persistence, you will notice that Redis does not persist data to local disk as soon as new data is added. Instead, Redis writes data to the operating system’s Page Cache first. Flushing Page Cache data to disk reduces random I/O to disk, which is known to be time consuming, and improves system performance, including in messaging middleware, for persistence.

In extreme cases, data in the Page Cache can be lost, such as sudden power outages or machine restarts. To solve the problem of data loss in Page Cache, cluster deployment can be used to prevent data loss.

1.3 Messages are lost during consumption

Messages are also lost during consumption, and are much more likely to be lost during consumption than in the first two cases. A message consumption process is roughly divided into three steps: the consumer pulls the message, the consumer processes the message, and the message system updates the consumption progress.

The first step in pull messages may occur when the network jitter is unusual, the second step in the process the message when some business exceptions can occur, and lead to process did not go out, if an exception occurs in the first step, the second step, notification message system update consumption progress, so the news of the failure will never be dealt with, nature is lost, In fact, our business is not finished.

Want to avoid the condition of the missing messages in the consumer, can in the message receiving and processing completed update consumption progress, but in extreme cases, there will be a problem of repeated consumption of information, such as a news after processing is completed, the consumer goes down, then no progress update consumption, consumer after the restart, the message will be to consumption.

2. How to ensure that messages are consumed only once

The message system itself cannot guarantee that the message is consumed only once, because the consumption itself may be repeated, the downstream system may start to pull the repetition, the repetition caused by the retry failure, and the repetition caused by the compensation logic may create repeated messages. To ensure that the message is consumed only once can be achieved by using idempotency.

An idempotent is a mathematical concept that results in the same result when the same operation is performed many times and once.

From the concept of idempotent, even if the message is executed multiple times, the system will not be affected, so how to ensure idempotent when using a message system? Because both producers and consumers are likely to produce duplicate messages, idempotency should be guaranteed at both ends.

Guarantee producers such as power, in the production of news, a global ID generated by snowflakes algorithm to the message, the message in the system maintenance message has ID mapping relation, if already exists in the mapping table is the same ID, the discard this message, although the message is delivered twice, but is actually save a, avoids the problem of repeated information.

Producer idempotency is related to the chosen message-oriented middleware, because most of the time the messaging system doesn’t need to be implemented ourselves, so idempotency is less manageable, and consumer idempotency is where we developers focus our control.

Idempotent operations can be done on the consumer side from both the common layer and the business layer, depending on our business requirements.

At the general level, the global unique ID generated by good news generation is used. After the message is processed successfully, the global ID is stored in the data. Before processing the next message, the global ID is queried in the database to see whether the global ID exists.

Using this globally unique ID, message idempotency is implemented with the following pseudocode:

boolean isIDExisted = selectByID(ID); // Check whether the ID exists
if(isIDExisted) {
  return; // If it exists, it returns
} else {
  process(message); // If the message does not exist, process the message
  saveID(ID);   / / store ID
}
Copy the code

However, in extreme cases, this approach can still cause problems. If the consumer crashes and restarts after processing the message before saving it to the database, it will retrieve the message again and execute the query that the message has not been consumed. Database transactions can be introduced to solve this problem, but will degrade system performance. If message re-consumption is not strictly required, it is a good idea to use this generic scheme without introducing transactions, which is also extremely unlikely.

At the business level, we have more options, such as optimistic locking, pessimistic locking, memory deduplication (github.com/RoaringBitm…

For example, if we want to add credits to a user, we can use the messaging system to asynchronously notify the user because the credits operation does not need to be in the main business. To use optimistic locks, we need to add a version number field to the score table. In addition, the version number of this account is first queried when the message is produced and sent to the message system together with the message.

After receiving the message and version number, the consumer carries the version number when executing the SQL for the update integral operation, similar to:

update score set score = score + 20.version=version+1 where userId=1 and version=1;
Copy the code

After the message is consumed successfully, version becomes 2, so if a duplicate message with version=1 is pulled again by the consumer, the SQL statement will not execute successfully, thus ensuring the idempotency of the message.

To ensure that the message is consumed only once, we need to focus on the consumer segment and use idempotency to ensure that the message is consumed once.

Today, standing on the general level of message-oriented middleware, I talked about how to ensure that data is not lost and only consumed once. I hope today’s article is helpful to your study or work. If you think the article is valuable, you are welcome to like it, thank you.

The last

At present, many big names on the Internet have message-oriented middleware related articles, if there is the same, please forgive. The original is not easy, the code word is not easy, but also hope you support. If there are mistakes in the article, please also put forward, thank you.