RocketMQ is a professional messaging middleware independently developed by Alibaba Group. Based on the highly available distributed cluster technology, RocketMQ provides a series of messaging cloud services, including message subscription and publication, message track query, timed (delayed) message, resource statistics, monitoring and alarm. It is the core product of enterprise-class Internet architecture. RocketMQ message queue has a history of more than 9 years. It provides asynchronous decoupling, peak cutting and valley filling capabilities for distributed application systems, and features required by Internet applications such as massive message accumulation, high throughput and reliable retry. It is the core product used in Alibaba Double 11.

Message queue RocketMQ is an official commercial product of Ali Cloud. Currently, it provides high availability message cloud service in multiple regions of Ali Cloud. Multiple machine rooms are deployed in a single domain, with high availability. Product stability and availability are implemented in accordance with Alibaba’s internal standards without any single point.

Message queue RocketMQ currently provides TCP and HTTP protocol access and supports seven programming languages: Java, C++,.net, Go, Python, Nodejs, and PHP. The RocketMQ message cloud service enables applications developed in different programming languages to quickly access the message queue. Users can deploy the application in Alicloud ECS or enterprise self-built cloud, or embed it in mobile terminal or Internet of Things device to establish connection with message queue RocketMQ for message sending and receiving. Meanwhile, local developers can also access message queue RocketMQ service for message sending and receiving through public network.

Product features

Message queue RocketMQ provides access to multiple programming languages based on TCP and HTTP protocols and multi-dimensional management tools, as well as a series of features for different application scenarios.

Feature overview diagram

Multi-protocol support

  • Support HTTP: RESTful standards, easy to use, fast access, strong cross-network capability, and support seven language clients.

  • TCP: Different from HTTP, the SDK provides more professional, reliable, and stable TCP access.

  • Support for STOMP: A plain-text protocol mechanism similar to HTTP, commonly used for lightweight interaction between scripting languages such as Ruby, Python, and Perl and message queue RocketMQ Broker.

Management tool

  • Web console: support Topic management, producer management, consumer management, message query, message track display and query, resource reports and monitoring alarm management.

  • OpenAPI: Provides apis to facilitate integration of message queue RocketMQ management tools into your own console.

  • Mqadmin Command set: The proprietary cloud output provides a rich set of administrative commands to manage the message queue RocketMQ service by command.

features

  • Transaction messages: Implement X/Open XA-like distributed transaction functionality to achieve the final consistency state of the transaction.

  • Timed (delayed) messages: Allows message producers to specify messages for timed (delayed) delivery up to 40 days.

  • Large messages: Supports a maximum of 4 MB messages.

  • Message trace: The message trace can clearly locate the complete link that the message is sent from the publisher to the message subscriber through the RocketMQ server of the message queue, facilitating fault locating and troubleshooting.

  • Broadcast consumption: Allow all consumers identified by the same Group ID to consume a message once.

  • Sequential messaging: Allows message consumers to consume messages in the order in which they were sent.

  • Reset consumption schedule: Resets consumption schedule based on time, allowing users to trace back messages or discard accumulated messages.

  • Dead-letter queue: Stores messages that cannot be consumed normally in a special dead-letter queue for subsequent processing.

  • Global message routing: synchronously replicates messages between regions to ensure data consistency between regions.

Private Cloud Deployment

  • Expert customization: provide technical scheme design; Expert on-site technical support and training.

  • Flexible deployment: Supports independent deployment of proprietary clouds and hybrid cloud architectures.

  • O&m management and control: Private cloud supports MQadmin command set and Open API

  • Operation and maintenance management tool, convenient control platform integration and unified operation and maintenance.

Message sending and receiving model

Message queue RocketMQ supports a publish/subscribe model, where a message publisher (producer) can send a message to a Topic on the server, and multiple message recipients (consumers) can subscribe to the Topic to receive the message, as shown in the following figure:

Topic

Message Topic, a first-level message type, classifies messages by Topic.

Message

Message, the carrier of information transfer in a message queue.

Message ID

A globally unique identifier of a message, automatically generated by the message queue RocketMQ system, that uniquely identifies a message.

Message Key

The service identifier of a message is set by the message Producer and uniquely identifies a certain service logic.

Tag

Message labels, secondary message types, are used to further differentiate the message categories under a Topic

Producer

Message producers, also known as message publishers, are responsible for producing and sending messages.

Producer instance

An object instance of Producer. Different Producer instances can run in different processes or on different machines. The Producer instance is thread-safe and can be shared between multiple threads in the same process.

Consumer

Message consumers, also known as message subscribers, are responsible for receiving and consuming messages.

Consumer instance

An object instance of Consumer. Different Consumer instances can run in different processes or on different machines. Configure thread pool consumption messages within a Consumer instance.

Group

A class of producers or consumers generally produce or consume the same kind of messages and follow the same logic in publishing or subscribing to the messages.

Group ID

Group identifier.

Exact-once delivery semantics

The exact-once delivery semantics means that a message sent to the message system can be processed by the consumer only Once. Even if a message is re-delivered at the production end, it will be consumed only Once at the consumer end.

Cluster consumption

All consumers identified by a Group ID share consumption messages equally. For example, if a Topic has nine messages and a Group ID has three Consumer instances, then only three of the messages are consumed in a clustered consumption mode, with each instance split equally.

Radio consumption

All consumers identified by a Group ID consume a message once each. For example, if a Topic has nine messages and a Group ID has three Consumer instances, each instance will consume nine messages in broadcast consumption mode.

Timing of the message

The Producer sends a message to the Message queue RocketMQ server, but does not expect the message to be delivered immediately. Instead, the message is delayed until a certain time after the current point in time to be delivered to a Consumer for consumption. The message is a timed message.

Delay message

The Producer sends a message to the Message queue RocketMQ server, but does not expect the message to be delivered immediately. Instead, the message is delivered to the Consumer for consumption after a certain period of time. The message is called a delayed message.

Transaction message

Message queue RocketMQ provides a distributed transaction function similar to X/Open XA. The final consistency of distributed transactions can be achieved through the transaction messages of message queue RocketMQ.

The order message

Message queue RocketMQ provides a type of message that is published and consumed sequentially, divided into globally ordered messages and partitioned ordered messages.

Global order message

For a given Topic, all messages are published and consumed in a strict first-in, first-out (FIFO) order.

Partition order message

For a given Topic, all messages are partitioned according to a Sharding key. Messages within the same partition are published and consumed in a strict FIFO order. Sharding key is a key field used to distinguish different partitions in sequential messages, which is completely different from the key of ordinary messages.

Messages are stacked

The Producer has sent messages to the server of message queue RocketMQ. However, due to the limited consumption capacity of consumers, all messages cannot be correctly consumed in a short time. At this time, the server of message queue RocketMQ keeps messages that have not been consumed, which is the state of message accumulation.

The message filter

Consumers can filter messages based on message tags, ensuring that consumers end up receiving only the filtered message types. Message filtering is done on the server side of message queue RocketMQ.

Message trajectory

In the process of sending a message from the producer to the subscriber’s consumption, it is a complete link information aggregated by the time and place of each relevant node. Message traces enable you to clearly locate the complete link that a message is sent from a producer to a message consumer through the RocketMQ server of the message queue, facilitating problem locating and troubleshooting.

Reset consumption point

Within the time range of message persistence (default 3 days), the message consumer’s consumption of its subscription Topic is reset, using the timeline as the coordinate, and the subscriber will receive the message sent by the message producer to the RocketMQ server of the message queue after the set point in time.

Dead-letter queue

Dead-letter queues are used to process messages that cannot be consumed normally. When a message fails to be consumed initially, message queue RocketMQ automatically retries the message. If consumption still fails after the maximum number of retries is reached, it indicates that the consumer cannot properly consume the message under normal circumstances. At this point, message queue RocketMQ does not immediately discard the message, but sends it to a special queue corresponding to the consumer.

Message Queue RocketMQ refers to these messages, which are not normally consumed, as dead-letter messages, and to the special queues that store them as dead-letter queues.

Message routing

Message routing is used to synchronize messages between different regions to ensure data consistency between regions. Message queue RocketMQ’s global message routing function relies on ali Cloud’s high-quality infrastructure to implement high-speed channel dedicated lines, which can efficiently achieve synchronous replication of messages between different regions at home and abroad.

Message queue RocketMQ is scalable in any environment, the publisher must be a cluster, the message server must be a cluster, and so must the subscribers. High availability at the cluster level is the main difference between RocketMQ and other message servers. Producer sends a message to a message server, and the message server randomly selects a Consumer. If the Consumer succeeds, we consider it successful.

Note: The Server or Server of message queue RocketMQ mentioned in this article contains Name Server, Broker, etc. A server is not the same as a Broker.

System Deployment Architecture

The following figure shows the system deployment architecture.

The concepts involved in the figure are as follows:

  • Name Server: Is a nearly stateless node that can be clustered to provide naming services, update and discover Broker services in message queue RocketMQ.

  • Broker: There are Master brokers and Slave brokers. A Master Broker can correspond to multiple Slave brokers, but a Slave Broker can correspond to only one Master Broker. Once the Broker is started, it needs to register itself with the Name Server. Subsequently, the Topic routing information is regularly reported to the Name Server every 30 seconds.

  • Producer: Keep-alive links are established with one of the nodes (randomly) in the Name Server cluster, Topic routing information is periodically read from the Name Server, and long links are established to the Master Broker providing Topic services. And sends heartbeat periodically to the Master Broker.

  • Consumer: Establish a long connection (randomly) with one of the nodes in the Name Server cluster, periodically pull Topic routing information from the Name Server, and establish a long connection with the Master and Slave brokers providing Topic services. Periodically sends heartbeat messages to Master and Slave brokers. Consumers can subscribe to messages from both Master and Slave brokers, and the subscription rules are determined by the Broker configuration.

A subscription model

The subscription model for message queue RocketMQ is published/subscribe (Pub/Sub), as shown in the figure below.

A Producer cluster is used to represent an application that sends messages. A Producer cluster contains multiple Producer instances, which can be multiple machines, multiple processes of a machine, or multiple Producer objects of a process. A Producer cluster can send multiple Topic messages. Imagine sending a distributed transaction message. If the Producer fails unexpectedly, the Broker will actively call back to any machine in the Producer cluster to confirm the status of the transaction.

Consumer cluster: Used to represent a Consumer messaging application. A Consumer cluster contains multiple Consumer instances, which can be multiple machines, multiple processes, or multiple Consumer objects for a process. Multiple consumers in a Consumer cluster consume messages on an equally shared basis. If the broadcast mode is set, then each instance under the Consumer cluster consumes the full amount of data.

A Consumer corresponds to a Group ID, and a Group ID can subscribe to multiple topics, as shown in Group 1 in the figure. Group and Topic subscriptions can be set directly in the program.

Note: Whether you are successful today depends on your attitude yesterday, your attitude today determines whether you will be successful tomorrow.