RocketMQ features and terminology

RocketMQ basic features

1. Subscribe and publish

Publishing a message is when a producer sends a message to a topic; Subscribing to messages is when a consumer follows messages with certain tags in a topic.

2. Message order

Message ordering refers to the ability to consume a class of messages in the order in which they are sent. For example, an order produces three messages: order creation, order payment, and order completion. Consumption must be in this order in order to make sense, but at the same time orders can be consumed in parallel. RocketMQ ensures strict message ordering.

3. Message filtering

RocketMQ consumers can filter messages based on tags, as well as support custom attribute filtering. Message filtering is currently implemented at the Broker side, with the advantage of reducing the network transmission of unwanted messages to consumers, but the disadvantage of increasing the burden on the Broker and relatively complex implementation.

4. Message reliability

RocketMQ supports high reliability of messages. There are several conditions that affect message reliability:

  • 1. Broker closes abnormally
  • 2. The Broker crashed abnormally
  • 3, the operating system Crash
  • 4, the machine power, but can immediately restore the power supply situation
  • 5. The machine cannot be turned on (CPU, motherboard, memory and other key equipment may be damaged)
  • 6. The disk is damaged

Matters needing attention

In the first four cases, where hardware resources can be recovered immediately, RocketMQ can ensure that messages are not lost, or that a small amount of data is lost (depending on whether the flush is synchronous or asynchronous).

The last two cases are single points of failure and cannot be recovered; once they occur, all messages at that single point are lost.

RocketMQ ensures 99% of messages are not lost through asynchronous replication in both cases, but a very small number of messages can still be lost. Synchronous double write technology can completely avoid single points, synchronous double write is bound to affect performance, suitable for high message reliability requirements, such as money-related applications.

Note: RocketMQ supports synchronous double-write starting with version 3.0.

5. At least once

At least Once means that each message must be delivered Once. The Consumer first pulls the message to the local and then returns an ACK to the server after the consumption is complete. RocketMQ supports this feature well because it does not ack the message if there is no consumption.

6. Backtracking

Backdating consumption is when a Consumer has successfully consumed a message. To support this function, the message needs to be retained after the Broker has delivered a successful message to the Consumer because of business requirements for re-consumption. Reconsumption is usually done in time. For example, if the Consumer system fails and needs to be reconsumed one hour ago, the Broker should provide a mechanism to reverse the consumption progress in time. RocketMQ supports backtracking of consumption in time, down to the millisecond.

7. Transaction messages

The RocketMQ Transactional Message means that applying a local transaction and sending a Message operations can be defined into a global transaction that either succeeds or fails at the same time.

RocketMQ’s transaction messages provide distributed transaction functionality similar to X/Open XA, through which the ultimate consistency of distributed transactions can be achieved.

8. Timing messages

Timed messages (deferred queues) are messages that are sent to the broker but are not consumed immediately and wait for a specific time to be delivered to the real Topic.

The default value of the broker is “1s 5s 10s 30s 1m 2m 3m 4M 5M 6m 7m 8M9m 10m 20m 30m 1H 2H”. The default value of the broker is “1s 5s 10s 30S 1M 2m 3m 4m 5M 6m 7m 8M9m”.

MessageDelayLevel is a property of the broker and does not belong to a topic.

To send messages, set the delayLevel: msg.setdelaylevel (level). Level has the following three situations:

  • If level == 0, the message is non-delayed

  • 1<=level<=maxLevel, the message is delayed for a specific time, for example, level==1, delay 1

  • If slevel > maxLevel, level== maxLevel, for example, level==20, the delay is 2h

The scheduled messages are temporarily stored in a topic named SCHEDULE_TOPIC_XXXX and stored in a queue according to delayTimeLevel. QueueId = delayTimeLevel -1, that is, a queue only stores messages with the same delay. Ensure that messages with the same delivery delay can be consumed sequentially. The broker consumes SCHEDULE_TOPIC_XXXX schedulingly, writing the message to the actual topic.

Note that timed messages are counted on the first write and scheduled writes to real topics, so the number of sent messages and TPS will be higher.

9. Message retry

When a Consumer fails to consume a message, a retry mechanism is provided to make the message consume again.

Consumer Consumer message failure can be considered in the following situations:

  • 1. Due to the reasons of the message itself, such as deserialization failure, message data itself cannot be processed (such as phone charge recharge, the mobile phone number of the current message is cancelled and cannot be recharged), etc. This error usually requires that this message be skipped and consumed before another message is consumed. 99% of failed messages are unsuccessful even if they are consumed immediately, so it is best to provide a timed retry mechanism that retries after 10 seconds.

  • 2. The dependent downstream application service is unavailable, such as DB connection is unavailable, and the external system network is unreachable. With this error, even if the current failed message is skipped, consuming other messages will also result in an error. In this case, it is recommended to apply a sleep 30s before consuming the next message to reduce the stress on the Broker to retry the message.

10. Message recasting

When a producer sends a message:

  • If the synchronization fails, the message will be recast

  • Asynchronous messages have retry

  • Oneway has no guarantees.

Message redelivery ensures that messages are delivered as successfully as possible without loss, but it can cause message duplication, which is an inevitable problem in RocketMQ. In general, message repetition does not occur. When a large number of messages and network jitter occur, message repetition is a high probability event. In addition, producer active retransmissions and consumer load changes can also lead to duplicate messages.

How do I set up message recast

  • 1, retryTimesWhenSendFailed

The default value is 2, so the producer will try to send retryTimesWhenSendFailed + 1 times at most. Instead of selecting the broker that failed last time, try sending messages to other brokers to ensure that messages are not lost. If the number of recasts exceeds the threshold, the client will ensure that the message is not lost. Recasts occur when remotingExceptions, MQClientExceptions, and partial MQBrokerExceptions occur.

  • 2, retryTimesWhenSendAsyncFailed

Retry times of asynchronous sending failures. The asynchronous retries are performed only on the same broker without selecting other brokers. Messages are not lost.

  • 3, retryAnotherBrokerWhenNotStoreOK

Message flush (primary or secondary) timed out or slave is unavailable (return status is not SEND_OK). Whether to try sending messages to other brokers. Default is false. Very important messages can be enabled.

11. Flow control

Producer flow control as broker processing capacity reaches a bottleneck; Consumer flow control, as consumption capacity reached a bottleneck.

Producer flow control

  • 1, commitLog file locked time exceeds osPageCacheBusyTimeOutMills, parameter defaults to 1000 ms, flow control.

  • 2. If transientStorePoolEnable = true is enabled, the broker is an asynchronous host, and transientStorePool resources are insufficient, the current SEND request is rejected, and flow control occurs.

  • The broker checks the wait time of the send request queue every 10ms. If the wait time exceeds waitTimeMillsInSendQueue, the default is 200ms. The send request is rejected and flow control occurs.

  • 4. The broker controls traffic by rejecting send requests.

Consumer flow control

  • If the number of messages cached locally exceeds the pullThresholdForQueue, the default value is 1000.

  • Consumer message size than pullThresholdSizeForQueue local cache, the default 100 MB.

  • Consumer message span over consumeConcurrentlyMaxSpan local cache, 2000 by default.

  • The result of consumer flow control is reduced pull frequency.

12. Dead-letter queues

Dead-letter queues are used to process messages that cannot be consumed normally.

When a message fails to be consumed for the first time, the message queue automatically retries the message. If the consumption still fails after the maximum number of retries is reached, it indicates that the consumer cannot consume the message correctly under normal circumstances. In this case, the message queue does not immediately discard the message, but sends it to the special queue corresponding to the consumer.

RocketMQ calls these messages, which normally cannot be consumed, dead-letter messages, and the special queues that store them dead-letter queues.

In RocketMQ, consumer instances can be consumed again by resending messages in a dead-letter queue using the Console.

RocketMQ terms

1. Message Model

RocketMQ consists of a Producer, a Broker, and a Consumer. A Producer produces messages, a Consumer consumes messages, and a Broker stores messages.

Brokers correspond to one server during actual deployment. Each Broker can store messages from multiple topics, and messages from each Topic can be fragmented among different brokers.

MessageQueue is used to store the physical addresses of messages, which are stored in multiple Message queues within each Topic. The ConsumerGroup consists of multiple Consumer instances.

2, Producer

Message producer, responsible for generating messages, generally by business systems.

3, Consumer

Message consumers are responsible for consuming messages, and typically the backend system is responsible for asynchronous consumption.

4, PushConsumer

Consumer A type of consumption in which the Broker actively pushes data to the Consumer after receiving it.

Applications typically register a Listener interface with the Consumer object, which calls back the Listener interface method as soon as it receives a message. This consumption mode generally has high real-time performance.

5, PullConsumer

Consumer is a type of consumption in which the application actively invokes Consumer’s pull message method to pull messages from the Broker server, with the application controlling the initiative. Once the batch messages are retrieved, the application initiates the consumption process.

6, ProducerGroup

A set of producers of the same kind who send the same kind of messages and send them logically.

If a transaction message is sent and the original producer crashes after sending, the Broker server contacts other producer instances in the same producer group to commit or backtrack consumption.

7, ConsumerGroup

A collection of the same kind of consumers, which usually consume the same kind of messages and consume logically. Consumer groups make it easy to achieve the goals of load balancing and fault tolerance in terms of message consumption.

Note that the consumer instances of the consumer group must subscribe to the exact same Topic. RocketMQ supports two messaging modes: Clustering and broadcast consumption.

8, the Broker

A message transfer role that stores and forwards messages. Also known as a Server. It is called a Provider in the JMS specification.

9. Broadcast consumption

A message is consumed by multiple consumers, even if they belong to the same Consumer Group, and the message is consumed by each Consumer in the Consumer Group once. The concept of Consumer Group in broadcast consumption can be considered meaningless in terms of message partitioning.

In CORBA Notification specification, all consumption modes belong to broadcast consumption. In the JMS specification, this is equivalent to the JMS Topic (Publish/Subscribe) model

10. Cluster consumption

Consumer instances in a Consumer Group share consumption messages equally.

For example, if a Topic has nine messages and a Consumer Group has three instances (perhaps three processes, or three machines), each instance consumes only three of the messages.

11. Sequential messages

Consume messages in the same order as they are sent,

In RocketMQ, this refers primarily to local ordering. For a class of messages to be sequential, they must be sent in a single-thread order by Producer and sent to the same queue, so that consumers can consume messages in the order that Producer sends them.

12. Common order

A type of sequential message. Normally, complete sequential messages are guaranteed. However, if a communication exception occurs, the Broker restarts.

If the business can tolerate temporary out-of-order messages under cluster exceptions such as a Broker down or restart, the normal order is appropriate.

13. Strict order

Message sequence A type of message that guarantees the sequence regardless of normal or abnormal conditions, but sacrifices the distributed Failover feature, which means that if only one machine in a Broker cluster becomes unavailable, the whole cluster becomes unavailable, greatly reducing service availability.

If the server is deployed in synchronous dual-write mode, this defect can be avoided by automatic switchover on the standby server, but the service may still be unavailable for several minutes. (The automatic switchover function is not implemented yet.) Currently, only the database binlog synchronization strongly depends on strict sequential messages. Most other applications can tolerate transient disorder, and ordinary sequential messages are recommended.

14 and the Message Queue

In RocketMQ, all message queues are persistent, data structures of infinite length. By infinite length, each storage cell in the queue is of a fixed length, and the storage cell is accessed using Offset, which is of type Javalong, 64-bit, and theoretically won’t overflow for 100 years. Therefore, it is considered that the queue length is infinite. In addition, only the data of the last few days is saved in the queue, and the previous data will be deleted according to the expiration time.

You can also think of a Message Queue as an array of infinite length, with offset being the subscript.

15. Tag

Flags set for messages to distinguish different types of messages under the same topic.

Messages from the same business unit can be tagged differently under the same topic for different business purposes. Tags are an effective way to maintain clarity and consistency in your code and optimize the query system RocketMQ provides.

Consumers can implement different consumption logic for different subtopics based on the Tag for better scalability.