RocketMQ message sending common errors and solutions

Message sending times out. The client logs are as follows:



When a client sends a message timed out, the first suspect is usually the RocketMQ server, if Broker performance is jitter and unable to handle the current volume.

So how do we find out if RocketMQ currently has a performance bottleneck?

First we run the following command to check the distribution of RocketMQ message write time:

cd /${USER.HOME}/logs/rocketmqlogs/
grep -n 'PAGECACHERT' store.log | more
Copy the code

The following output is displayed:



RocketMQ will be able to see if there is a detailed performance bottleneck in RocketMQ message writing based on the distribution of message delivery time per minute prior to printing. The range is as follows:

• [<=0ms] is less than 0ms, which is subtle.
• [0 to 10ms] indicates the number of milliseconds smaller than 10ms.
• [10 to 50ms] Smaller than 10ms
• At 50ms

According to the author’s experience, if there are more than 20 intervals of 100-200ms and above, it means that the Broker does have a certain bottleneck. If there are only a few, it means that this is the jitter of memory or pagecache, which is not a problem.

Timeouts usually have little to do with the Broker’s ability to process them. Another example is RocketMQ Broker’s fast failure mechanism, where the Broker queues messages when it receives a request from a client and executes them sequentially. If a message queue waits more than 200ms and starts a quick failure, return the [TIMEOUT_CLEAN_QUEUE]broker busy to the client, which is covered in more detail in Part 3 of this article.

When encountering network timeouts on the RocketMQ client, it is often a good idea to consider whether some of the application’s own garbage collection is causing message delivery timeouts due to GC pauses. This is something I have encountered in my stress test environment, but not in production.

Network timeouts are commonly encountered in RocketMQ, usually related to network jitter, but since I am not particularly good at networking, I cannot find direct evidence, but I can find indirect evidence, such as connecting kafka and RocketMQ clusters in one application. Discover that all brokers connected to the RocketMQ cluster and connections to the Kafka cluster timed out at the same time as the timeout occurred.

But if there is a network timeout, we have to deal with it. Is there a solution?

Our lowest expectations of message middleware is high and low latency, from the above message is sent time distribution can also see RocketMQ indeed conforms to our expectations, most of the requests are within the subtle level, therefore I give solution, reduce the timeout message is sent, increase retries, and increase the rapid failure of the maximum waiting time. Specific measures are as follows:

• To increase the duration of a quick Broker failure, 1000 is recommended. Add the following configuration to the Broker configuration file:

  maxWaitTimeMillsInQueue=1000
  Copy the code

  The main reason is that in the current version of RocketMQ, a quick fail error is SYSTEM_BUSY and does not trigger a retry. Increase this value appropriately to avoid triggering this mechanism as much as possible. For details, see Part 3 of this article, which focuses on SYSTEM_BUSY and Broker_BUSY.

• If RocketMQ is a client version 4.3.0 or later (excluding 4.3.0) set timeout to 500ms for sending messages and set the number of retries to 6 (this can be adjusted to be greater than 3). The philosophy behind this is to timeout as soon as possible and retry. Since network jitter in the LAN is found to be instantaneous, it can be recovered in the next retry. Besides, RocketMQ has a fault avoidance mechanism, and tries to select a different Broker when retrying. The related code is as follows:

  DefaultMQProducer producer = new DefaultMQProducer("dw_test_producer_group");
  producer.setNamesrvAddr("127.0.0.1:9876");
  producer.setRetryTimesWhenSendFailed(5);// Synchronous sending mode: Retry times
  producer.setRetryTimesWhenSendAsyncFailed(5);// Asynchronous sending mode: Retry times
  producer.start();
  producer.send(msg,500);// Message sending timeout
  Copy the code

• If RocketMQ client version is 4.3.0 or later

  If the client version is 4.3.0 or later, the message sending timeout period is set to the total timeout period of all retries. Therefore, you cannot directly set the RocketMQ sending API timeout period. Instead, you need to wrap the RocketMQ SENDING API, and retry is performed on the outer receiver.

  public static SendResult send(DefaultMQProducer producer, Message msg, int 
                                retryCount) {
      Throwable e = null;
      for(int i =0; i < retryCount; i ++ ) {
          try {
              return producer.send(msg,500); // set the timeout period to 500ms with an internal retry mechanism
          } catch(Throwable e2) { e = e2; }}throw new RuntimeException("Message sending exception",e);
   }
  Copy the code

Let’s start with a graph to illustrate when these errors are thrown during the lifetime of the message.



According to the above five types of error logs, the original triggered can be classified into the following three types.

• Pagecache is under high pressure

  The following three types of errors fall into this category

  • [REJECTREQUEST]system busy
  • [PC_SYNCHRONIZED]broker busy
  • [PCBUSY_CLEAN_QUEUE]broker busy

  Determine whether pagecache is busy based on writing messages, in the memory of the time to add a message lock, the default judgment criteria is lock time is more than 1s, it is considered that pagecache pressure, to the client to throw the relevant error log.

• In RocketMQ, messages are sent from a thread pool of only one thread. A bounded queue, with a default length of 1W, is maintained internally. If the number of squeezes in the current queue exceeds 1W, the thread pool reject policy is executed. Throw the [too many Requests and System Thread pool busy] error.

• The Broker side fails rapidly

  By default, RocketMQ has fast failure enabled on the Broker. If the pagecache is not busy (locked for more than 1s), RocketMQ will not queue up any requests for 200ms. System BUSY is returned directly to the client, but since the RocketMQ client currently does not retry this error, extra processing is required to resolve this type of problem.

Since the client will not retry a TIMEOUT_CLEAN_QUEUE error, it is recommended to add a quick failure criterion to the broker’s configuration file:

# The default value is 200, indicating 200ms
waitTimeMillsInSendQueue=1000
Copy the code

Broker Busy: Broker busy: Broker busy

RocketMQ sends System BUSY and Broker Busy Cause analysis and solution

RocketMQ broker busy

This article is based on RocketMQ In Action and Progression,This column introduces RocketMQ using scenarios, what problems are encountered, how to solve them, and why. The design of the column focuses on the actual combat, aiming to enable a RocketMQ beginner to quickly “beat the monster” through the study of this column, the combination of theory and practice, to become a leader in the field.