RocketMQ source version 4.9.1

An overview of the

The overall architecture

Description of roles:

  • NameServer, which provides routing services
  • A Producer is responsible for sending messages
  • The Broker, a message queue, stores messages and provides associated API operations
  • Consumer is responsible for Consumer messages

Production-consumption model

Message sending mode: synchronous, asynchronous, unidirectional

Message types: normal message (including delayed message), sequential message (global and local order), transaction message

This article through synchronous sending ordinary message Demo, to understand the main process of message sending.

producers

See below a demo of a producer sending a message

I mainly did several things:

  • Initialize a DefaultMQProducer object
  • Set the address of NameServer
  • Start producer
  • Send a message

Start the

During startup, the main scheduled tasks are:

  • NameServer address update task (NameServer scenarios are not displayed), which supports dynamic NameServer address update
  • The local routing information update task periodically pulls the latest routing information from NameServer to the local cache for update
  • Broker heartbeat detection task that periodically sends heartbeat packets to a cluster of Brokers and clears brokers that have gone offline

Note: Since both producer and consumer communicate with the Broker service at the bottom through the MQClientInstance class, the message pull service and the heavy load balancing service are both consumer-oriented services, they do not actually take effect even if the producer starts them.

Message is sent

There are two core steps in message sending:

  • Route information retrieval to get the service routes of available brokers based on the Topic of the message
  • Load balancing mechanism, that is, selecting the appropriate message queue

Routing for

The steps are as follows:

  • Retrieves routing information for the specified Topic from the local cache, and returns the result if obtained
  • Retrieves routing information for a given Topic from NameServer and updates the local cache if it does
  • Get routing information for the default Topic from NameServer, update the local cache if it does, and return the result

Load balancing

The goal is to evenly send messages to message queues across brokers.

There are two load balancing policies:

  • The default load balancing policy, which selects message queues by polling, maintains a queue subscript counter at the thread level
  • A load balancing strategy for failure delay, in which message queues are selected to avoid failing brokers for a certain period of time

Failure delay mechanism

Normal load balancing strategies have logic to avoid failed brokers, but they only work in the retry scenario of a single message being sent.

The implementation logic is as follows:

  • Maintains a queue subscript counter at the thread level
  • Each time the queue is selected, the counter value will be + 1, and then modulo with the total number of available queues, and finally calculate the subscript of the target queue
  • When the message is resent, the BrokerName for the last failed Broker sent is passed in, and if the target queue is found to belong to the same Broker name as the last failed Broker sent, proceed to step 1,2 and re-select the queue (circumventing the failed Broker in this way)

If you want to avoid failing brokers during multiple messages, you can enable the fault delay mechanism by configuring sendLatencyFaultEnable.

The overall process is as follows:

Emphasis:

  • After a message fails to be sent, the corresponding Broker is isolated for a period of time based on the delay in sending the message. This is called the failure isolation period
  • The Broker is considered unavailable by the producer during fault isolation, that is, it will be shunned the next time a message is sent.

The isolation time is based on the last message sending delay. The longer the delay, the longer the isolation time. The following is the relationship between latency and isolation times defined by RocketMQ.

conclusion

This article mainly contains the following contents:

  • Producer initiation process
  • Main process for sending producer messages
  • Process for obtaining routing information
  • Producer load balancing policy, and mainly introduces the fault delay mechanism

By reading this article, you can get an overview of the RocketMQ message delivery process, which implements scalable, highly available features through load balancing policies and failure delay mechanisms.