In distributed service SOA architecture, no single middleware or application is allowed to exist, and service discovery mechanism is necessary. There are multiple service instances, and the number changes dynamically. The registry provides service management capabilities where service callers obtain information about service providers to make remote calls.

The following describes RocketMQ architecture, cluster management, and some of the key concepts in RocketMQ.

Overall architecture design

When it comes to RocketMQ’s architectural design, its roots in Kafka must be noted. Kafka is a high-performance message middleware, often used in big data scenarios, but because Kafka does not support retry failure, timed messages, transaction messages, sequential messages also have obvious defects, difficult to support taobao transactions, orders, recharge and other complex business scenarios. The Taobao middleware team redesigned RocketMQ and wrote it in Java based on Kafka, so there are some concepts similar to Kafka in RocketMQ.

Kafka, RabbitMQ, RocketMQ and other popular message middleware are all based on publish/subscribe mechanism. Producer sends messages to message server, and consumers subscribe to messages they are interested in from message server. In this process, the message sender and the message consumer are the client, the message server is the server, and both the client and the server need to be aware of each other’s presence through the registry.

The RocketMQ deployment architecture is divided into four main parts, as shown in figure 1:

  • Producer: A role that publishes messages and sends messages to brokers. Producer supports distributed cluster deployment.
  • Consummer: The role of message consumer, responsible for subscribing to message consumption from the Broker and supporting distributed cluster deployment.
  • Broker: Role of message storage. It is responsible for storing, delivering, and querying messages and ensuring high availability of services. It supports distributed cluster deployment.
  • NameServer: a service management role that manages the routing information of Broker clusters and supports distributed cluster deployment.

NameServer is a very simple Topic routing registry that plays a similar role to the Zookeeper relied on in Dubbo and supports dynamic registration and discovery of brokers. It mainly includes the following two functions.

  • Service registration: NameServer receives the registration information of the Broker cluster, saves it as basic routing information, and provides a heartbeat detection mechanism to check whether the Broker is still alive.
  • Routing information management: NameServer stores the routing information of the Broker and provides the queue information for clients to query the Broker. The Producer and Consumer can use NameServer to know the routing information of the Broker cluster to deliver and consume messages.

The basic concept

  • Message: Message, the smallest unit of production and consumption of data. Each message must belong to a Topic. In RocketMQ, each message has a unique MessageID and can carry a Key with a business identity.
  • Topic: a collection of messages, each containing several messages, each belonging to only one Topic. Topic is the basic unit of Message subscription for RocketMQ.
  • Queue: Message Queue, the smallest unit that makes up a Topic. By default, a Topic corresponds to multiple queues, a Topic is a logical concept, a Queue is a physical storage, and the underlying layer actually pulls the messages from the Queue as consumers consume the Topic messages.
  • Tag: A flag set for a message to distinguish between different types of messages on the same topic. Messages from the same business unit can be tagged differently under the same topic for different business purposes. Tags are an effective way to maintain clarity and consistency in your code and optimize the query system RocketMQ provides. Consumers can achieve better extensibility based on the logic that the Tag implements for processing different consumptions of different subtopics.
  • UserProperties: A collection of user-defined properties that are part of Message.
  • ProducerGroup: A set of producers of the same kind who send the same kind of messages and send them logically. If a transaction message is sent and the original producer crashes after it is sent, the Broker server contacts other producer instances in the same producer group to commit or backtrack consumption.
  • ConsumerGroup: A collection of consumers of the same type, usually consuming the same type of message with consistent consumption logic. Consumers make it easy to achieve load balancing and fault tolerance goals in terms of message consumption. Note that the consumer instances of the consumer group must subscribe to the exact same Topic.

Good blog

RocketMQ deployment architecture