Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.
You used to be on the receiving end, receiving messages. I remember that when I was making payment, I received all kinds of news from the third-party payment company, such as payment success, payment failure, refund success and refund failure.
Some companies do a good job in the accuracy and timeliness of the message push, while others are very poor, especially several European companies. If we fail to consume, the message will be blocked in the queue head, resulting in the whole consumption queue stuck, unable to consume the following message, causing all kinds of ridicule.
Heaven good reincarnation, heaven rao who, finally when it is my turn to send. Although this project is not very difficult, it is the first project that I send, so take note of it.
why
Originally provided a single pull function, consumers can be timed to pull a single.
However, there are several problems with timing pull orders. One is timeliness. If the consumer pulls data every N minutes, the maximum delay is N minutes, and the timeliness is not good. Second, the order information pulled may be incomplete. For incomplete orders, consumers need to obtain them regularly, which increases the complexity.
Push single solution can solve the above problems, first of all, only complete information to push single, followed by complete information immediately push single. This not only ensures timeliness, but also reduces docking complexity.
Project research
I haven’t done the push list function before, so I thought about it carefully. I need to research three aspects: what content to push, how to push, and how to deal with push failure.
Push content
How do you decide what to push to consumers?
Through communication with the consumer, consulting the business side of the company and investigating competing products, determine the content to be pushed.
However, there is still something that is not taken into account. When the single mode is combined with the receiver, some recipients want the batch push function.
At present, it is designed as a single push, and the batch push function will not be implemented based on the current situation. Because of the timeliness problem, the message will be pushed to the receiver as soon as possible, and the receiver will transfer it as soon as possible after receiving it. Bulk push reduces overall timeliness and makes implementation more complex.
But this example shows that there are gaps in the research. In the future, more research can be done. Some needs can be chosen not to do, but to know the demand.
How to push
How do you push the data out?
Of course there has to be a platform! Because push needs to consider many details, such as data security, exception handling, interface management, etc., fortunately, some departments have made push management platform. Two schemes are provided, synchronous scheme and asynchronous scheme.
Synchronization scheme
In synchronous mode, the message sender directly invokes the receiver interface to immediately perceive the result
The asynchronous solution
In asynchronous mode, the receiver pushes the message to the message queue, and the receiver consumes it on demand
The difference between
The differences between synchronous and asynchronous modes are as follows
-
Synchronous mode can directly know the processing results of the receiver; The asynchronous mode cannot know the result of the processing
-
In synchronous mode, the sender ensures the success of message push. The asynchronous mode is guaranteed by the receiver
In asynchronous mode, only messages of the same type can be pushed to the same recipient. However, merchant orders belong to different recipients. In asynchronous mode, information may be leaked. In addition, the asynchronous mode does not know the message received, so the synchronous scheme is selected.
Exception handling
The push process is simple, consuming MQ messages and pushing order information to the recipient. We should never trust the network and the receiver, there are always various problems, when the message consumption failure how to deal with?
The hope is that if something happens when a Consumer consumes a message packet, the message packet will not be discarded directly, but can continue consuming it after a period of time without blocking. In short, retry.
rabbitmq
Rabbitmq can send packets back to the end of the main queue in two different ways:
Dead-letter queue
-
Basic_consume sets the ACK mode
-
To declare a dead letter queue, set x-message-TTL =30000, x-dead-letter-routing-key= the name of the main queue
-
Declare the main queue and set x-dead-letter-routing-key= the name of the dead-letter queue
-
The main queue column is bound to Exchange via a RoutingKey
If the consuming logic is abnormal, the consuming script calls basic_reject() and the message packet is Requeue to the RabbitMQ dead-letter queue. After a timeout of 30 seconds, the message packet enters the end of the main queue again.
To deliver
By chanel. BasicAck (tagId, false) and chanel. BasicPublish (message. GetMessageProperties () getReceivedExchange (), message.getMessageProperties().getReceivedRoutingKey(), true, MessageProperties.PERSISTENT_TEXT_PLAIN, body.getbytes()); To put the message back at the end of the queue, one function will no longer send unacknowledged messages back to the queue, and the other will be used to repost messages.
rocketmq
Rocketmq, now used by companies, can also do non-blocking retries based on my understanding of RocketMQ, because there is a retry queue:
Retry queue
After a consumption failure, the message is put into the Retry queue of RocketMQ.
-
For example, a consumer belongs to a message group named AAAConsumerGroup
-
The name of the RETRY queue is %RETRY%AAAConsumerGroup
-
The message in the retry queue will be sent to the consumer again after a period of time, and if it still fails to execute properly, it will be put into the retry queue again
-
By default, 16 retries fail, and the message goes from the retry queue to the dead-letter queue
Since the company has made some changes to RocketMQ, I asked my classmate to confirm the retry mechanism. I asked a lot of students, everyone said it was obstructive retry, the message is blocked at the head of the queue until the retry succeeds or the retry limit is reached. I had no choice but to find the corresponding R & D students for confirmation.
It turns out they designed two configurations, message ordered and message unordered. In the ordered case, a failed consumption is blocked at the queue head until the retry succeeds. In the unordered case, the system enters the retry queue. The system retries based on the configured retry interval and number of retries without blocking. This solves the retry problem.
This is of concern because obstructive retries can affect functionality by preventing subsequent messages from being pushed.
It is also a good idea to set a maximum retry limit even for non-obstructive retries. If there are too many exceptions, the message producer may become overwhelmed and crash. You need to understand that the responsibility for multiple delivery failures lies with the recipient.
monitoring
After the project goes online, it needs to be monitored; otherwise, the running status cannot be sensed. The data team was so helpful that I could quickly sort out the report and check the number of successful orders, the number of failed orders, the complete time of order information and the details of failed orders in real time.
Through these data, hidden problems can be found quickly, but also can analyze the situation of each receiver, some of the receiver is really a long story.
The development of
After the project goes online, we can determine the time when the order information is complete, and push the data to relevant parties after the order information is complete.
In the future, the pull order function will be optimized to ensure that the pulled orders are complete orders. At the same time, the logic for judging the integrity of information in the system will be changed, and the judgment module will be simplified and converged to prepare for the future system update.
data
-
Message queue RocketMQ version fails to consume messages and re-consumes them
-
Mechanism for handling failed message consumption in message queues
-
Rabbitmq messages return to the queue
-
Rabbitmq retry mechanism
-
Message confirmation mechanism for RabbitMQ
-
The team uses RabbitMQ in several scenarios
-
www.rabbitmq.com/documentati…
-
Rabbitmq dead letter queue
-
RabbitMQ dead letter queue + TTL introduction
-
Abnormal RocketMQ dead-letter queue | consumers how to deal with?
-
RocketMQ failed to consume a message
The last
If you like my article, you can follow my public account (Programmer Malatang)
My personal blog is shidawuhen.github. IO /
Review of previous articles:
-
Design patterns
-
recruitment
-
thinking
-
storage
-
The algorithm series
-
Reading notes
-
Small tools
-
architecture
-
network
-
The Go