Hello everyone, I’m Xiao CAI. A man who wants to be a man who talks architecture! If you also want to be the person I want to be, or point a concern to do a companion, let small dishes no longer lonely!

This article focuses on the common problems with RabbitMQ

Refer to it if necessary

If it is helpful, do not forget the Sunday

Wechat public number has been opened, xiao CAI Liang, did not pay attention to the students remember to pay attention to oh!

  • Message reliability Problem: How do I ensure that sent messages are consumed at least once?
  • Delayed message problem: How to implement delayed message delivery?
  • Message accumulation problem: How to solve the problem of millions of messages piled up and unable to timely consume?

We have already explained how to solve the problem of message loss, that is, to ensure the reliability of the message, so the other two problems are also important, this article will talk about the solution of the other two problems ~!

Message loss solution: “the RabbitMQ | message loss is the way it is

Delayed messages

Delaying the message literally means delaying the receipt of a message, so how can you delay the arrival of a message? This is what we need to think about, and there are two concepts in RabbitMQ that we need to understand before we look at delay queues

  • Dead-letter exchange
  • TTL

1) Dead-letter switch

A “dead letter” is the abandonment of a message that has died. Under what circumstances can an ordinary message become a “dead letter”? The following three conditions must be met:

  1. Consumer usebasic.reject 或 basic.nackDeclare a consumption failure and set the Requeue parameter of the message tofalse
  2. The message is an expired message that has not been consumed
  3. When the queue of messages to be delivered fills up, the earliest messages become dead letters

The dead letter switch is where the dead letter belongs.

If a queue is configured with dead-letter-exchange and a switch is specified, the dead letter from the queue will be delivered to the switch, which is called dead-letter exchange (DLX).

Steps: When a producer sends a message to a queue (siml. queue), if a consumer consumes a message from the queue (reject), and the queue is bound to a dead-letter switch (DL-queue), Dead-letter messages will then be sent to the dead-letter queue.

From the normal queue -> dead-letter queue process, we must declare two key pieces of information

  • The name of the dead-letter switch
  • Dead-letter Routing key bound to a dead-letter queue on a switch

These two pieces of information are also the basic configuration for delivering messages.

Let’s briefly simulate the scenario for condition 1

1. First declare a dead letter switch and a dead letter queue

We use simple annotations to generate it directly

On the RabbitMQ console interface, you can see that it has been successfully generated

2. Declare normal use of switches and queues

At this point we can create a working switch and queue and specify the dead letter switch

You can also view the creation status from the console

We can determine whether there is a declared dead letter switch by the DLX and DLK flags of the queue

3. Simulated rejection

Then we now simulate the client rejecting the message in code

1) Message sending

2) Message reception

View the console and the result is as follows:

2021-11-06 23:56:52.095  INFO 2112 --- [ntContainer#0-1Switch |] C.L.M.C.L istener. SpringRabbitListener: normal business receives the message: [hello!]2021-11-06 23:56:52.118  INFO 2112 --- [ntContainer#1-1] C.L.M.C.L istener. SpringRabbitListener: | dead-letter switch receives the message: helloCopy the code

This shows that our dead-letter switch has been successfully functioning

2) the TTL

We have successfully seen the use of dead letter switches, but this does not seem To have much To do with the delay queue we started with, let alone the TTL(time-to-live) which is used To handle delayed messages!

In the concept of TTL, if a message in a queue is not consumed after the TTL ends, the message will automatically become dead letter, and there are two kinds of TTL timeout cases:

  1. The queue where the message resides has a lifetime set
  2. The message itself sets the lifetime

We also perform the simulation scenario of condition 2 above

1. Declare dead letter switches and dead letter queues (completed above)
2. Declare a delay queue and specify a dead-letter switch

Again, the console looks at the creation result, and we see not only DLX and DLK flags, but also a TTL, indicating that the queue is deferred

3. Simulate consumption timeout

We send a message to the delay queue, and no consumer consumes it. We wait 1 minute to see if we can get into the dead letter queue

We have sent a message to the delay queue and a minute later we have also successfully discovered on the console that this message has entered the dead letter switch

2021-11-07 00:01:30.854  INFO 32752 --- [ntContainer#1-1] C.L.M.C.L istener. SpringRabbitListener: | dead-letter switch receives the message: the test TTL - the messageCopy the code

If both the message and the queue have a timeout, the shortest TTL is used. The message timeout is as follows:

As shown in the figure above, we can use the Message class to transmit Message information and set the timeout time. We set the timeout time to 5000 ms. After sending the Message successfully, the console also successfully prints the Message consumed by the dead letter switch after 5000 ms:

2021-11-07 00:03:09.048  INFO 39996 --- [ntContainer#1-1] C.L.M.C.L istener. SpringRabbitListener: | dead-letter switch receives the message:this is a ttl message
Copy the code

3) Delay queue

We used the dead letter switch to implement the delay queue indirectly, but RabbitMQ doesn’t have to be so troublesome, RabbitMQ already packages the plugin for us, we just need to download and install it

RabbitMQ plug-in download address

When we enter the address, we can find many plug-ins. Search the keyword delay to find the plug-ins we need for download

RabbitMQ plugins: RabbitMQ plugins: RabbitMQ plugins: RabbitMQ plugins: RabbitMQ plugins: RabbitMQ plugins: RabbitMQ plugins

docker run -itd --name rabbitmq -v plugins:/plugins -p 15672:15672 -p 5672:5672 rabbitmq:management
Copy the code

So I just upload the plugin to the plugins directory in the container

Then enter the container and run the following command to open the plug-in

rabbitmq-plugins enable rabbitmq_delayed_message_exchange
Copy the code

And we can see that there is an option for type type when we create the switch in the console

This step will indicate that RabbitMQ delay queuing is enabled

Then we can use DelayExchange, first we need to understand the code to create a delay switch:

Method 1

Way 2

We can send messages when we have everything ready

When sending a message, the message header must contain the X-delay parameter, which specifies the delay time

After this configuration, we can see on the console that delay.queue receives the corresponding message after 10 seconds and is then consumed by the corresponding consumer

3) summary

From dead letter switches to TTL to delay queues, we’ve looked at how to implement delayed messages step by step. Then we’ll conclude with a little bit:

Question 1: What kind of news becomes a dead letter?
  1. The message is rejected by the consumer or returned to NACK
  2. The message timed out and was not consumed in time
  3. The message queue is full
Problem 2: The way the message timed out
  1. Set the TTL attribute for the queue
  2. Set the TTL attribute for the message
Question 3: How do I use delay queues
  1. Download and enable the RabbitMQ delay queue plug-in
  2. Declare a switch and set the delayed property to true
  3. When sending a message, add the X-delay header with the value of timeout
Problem 4: Use scenario of delay queues
  1. Delay sending SMS notification
  2. Automatic order cancellation
  3. Inventory automatically rolls back

Lazy queues

With delay queues out of the way, let’s move on to lazy queues

Before we talk about lazy queues, let’s throw out a question

How does RabbitMQ solve the message heap problem

When does message accumulation problem occur?

  1. When producers are producing much faster than consumers are consuming
  2. When the customer is down, it does not restart in time

So how to solve this problem? The general idea is as follows:

  1. After the consumer machine restarts, add more consumers for processing
  2. Create a thread pool inside the consumer processing logic to improve the processing speed by using multiple threads
  3. Expand the capacity of queues and increase the upper limit of stacking

These methods are theoretically fine, but they are not elegant or even flexible. Instead, we can use the lazy queue type that comes with RabbitMQ

What is an lazy queue? Let’s take a look at some of the features of lazy queues:

  • When received, the message is stored directly to disk instead of memory
  • The message is read from disk and loaded into memory before it is consumed by the consumer
  • It supports the storage of millions of messages

The bottom line is to take advantage of disk buffering. The downside of this mechanism is that messages are less time-sensitive and performance is limited by disk I/O. With that in mind, how can we create lazy queues

Method 1

Way 2

Methods 3

This is to change a running queue to an inert queue based on command line modifications directly

rabbitmqctl set_policy Lazy "^lazy-queue$" '{"queue-mode":"lazy"}' --apply-to queues  
Copy the code

The meanings of several command parameters are as follows:

  • Rabbitmqctl: command line tool
  • Set_policy: Adds a policy
  • Lazy: indicates the policy name, which can be customized
  • ^lazy-queue$: Matches the queue name with a regular expression
  • ‘{“queue-mode”:”lazy”}’ : sets the queue to the lazy mode
  • — Apply-to queues: Policies that act on queues

While this lazy queuing approach has the disadvantage of reducing message timeliness, it is not unacceptable in some scenarios, and its advantages are equally obvious:

  • Disk-based storage, high message upper limit
  • No intermittent page-out, stable performance

So far we have covered the common problems with RabbitMQ, which we might not encounter in a normal development scenario, but we still need to be aware of them before they happen!

Don’t talk, don’t be lazy, and xiao CAI do a blowing bull X do architecture of the program ape ~ point a concern to do a companion, let xiao CAI no longer lonely. See you later!

Today you work harder, tomorrow you will be able to say less words!

I am xiao CAI, a man who grows stronger with you. 💋

Wechat public number has been opened, xiao CAI Liang, did not pay attention to the students remember to pay attention to oh!