I’m participating in nuggets Creators Camp # 4, click here to learn more and learn together!

preface

Hi, everybody. I’m Helu.

I have been in contact with message queue middleware such as ActiveMQ, RocketMQ, Kafka and RabbitMQ since I started working. Although I have been in contact with them, they only stay at the application level and have little understanding of the real principles. So I plan to learn and write, forming a series of articles, demystifying the message middleware.

This is RocketMQ’s first article, so let’s get started.

1. Introduction to RocketMQ

1.1 RocketMQ profile

Apache RocketMQ is a distributed messaging system developed in the Java language. It was developed by the Alibaba team and contributed to Apache at the end of 2016, becoming a top-level project of Apache.

Within Alibaba, RocketMQ serves thousands of applications, large and small, and on Singles’ Day every year, an incredible trillions of messages flow through RocketMQ. (On Singles’ Day in 2017, The whole Alibaba Group’s online message flow through RocketMQ reached trillions, with a peak TPS of 56 million), playing a pivotal role in Alibaba’s strategy of greater China and Taiwan.

Website address: http://rocketmq.apache.org/

1.2 Historical development of RocketMQ

  • Alibaba message-oriented middleware originated from the Multicolored Stone project in 2001, and Notify came into being during this period, which is used for the flow of core messages in transactions.

  • In 2010, B2B began to use ActiveMQ as the message kernel on a large scale. With the rapid development of Ali business, a message middleware supporting sequential messages and possessing massive message accumulation ability was urgently needed. MetaQ 1.0 was born in 2011.

  • In 2012, MetaQ had evolved to version 3.0 and abstracted the general-purpose messaging engine RocketMQ. Since then, RocketMQ has been open sourced, and Alibaba’s messaging middleware has officially gone out of the public eye.

  • In 2015, RocketMQ has experienced many years of Double 11, with outstanding performance in usability, reliability and stability. At the same time, cloud computing took off. Aliware MQ 1.0, based on RocketMQ, was launched by Ali messaging middleware, which began to provide messaging services for thousands of enterprises on Ali Cloud.

  • In 2016, MetaQ passed a new milestone during Singles’ Day with trillion-level message flows, while RocketMQ entered Apache incubation.

  • In 2017, RocketMQ launched version 4.0, which experts say is suitable for e-commerce, finance, big data, and the Internet of Things.

The above content and images are from RocketMQ Field & Principles.

1.3 RocketMQ core concepts

RocketMQ’s architecture consists of four core components: Producer, Consumer, NameServer, and Broker.

Producer

Message producer, responsible for generating messages, generally by business systems. RocketMQ provides three ways to send messages: synchronous, asynchronous, and one-way.

  • Synchronous sending: Synchronous sending means that the sender sends data packets only after receiving the response from the receiver. It is used for important notification messages, such as important notification emails and marketing SMS messages.

  • Asynchronous sending: After sending data, the sender does not wait for the receiver to send back the response and then sends the next data packet. This method is generally used in service scenarios where the link takes a long time and the response time is sensitive, for example, the transcoding service is enabled after a user uploads a video.

  • Unidirectional sending: Unidirectional sending only sends messages without waiting for a response from the server and without triggering a callback function. It is applicable to scenarios that require very short time but do not require high reliability, such as log collection.

Consumer

Message consumers are responsible for consuming messages, and typically the backend system is responsible for asynchronous consumption. – Consumer is also deployed by users, supports Push and Pull consumption modes, supports cluster consumption and broadcast messages, and provides real-time message subscription mechanism.

  • Pull: Pull consumers actively Pull information from the message server. As long as they Pull messages in batches, the user application will start the consumption process. Therefore, Pull is called active consumption.

  • Push: The Push Consumer encapsulates the pull, consumption schedule, and other internal maintenance work of the message, leaving the callback interface performed when the message arrives to the user application. So a Push is called a passive consumption type, but it is implemented to Pull messages from a message server. Unlike a Pull, a Push first registers a consumption listener and starts consuming messages only after the listener is triggered.

NameServer

Organization coordinator in cluster architecture. Mainly responsible for the management of source data, including the management of Topic and routing information, and collecting the working status of brokers, but not message processing.

Broker

A message transfer role that stores and forwards messages.

  • Is the core of RocketMQ responsible for sending, receiving, high availability, etc. (the real work)
  • You need to send information about yourself to NameServer periodically. By default, you need to send information about yourself to NameServer every 10 seconds.

1.4 RocketMQ’s message domain model

RocketMQ’s Message domain model is divided into Message, Topic, Queue, Offset, and Group.

Message

A message is a message to be transmitted.

Topic

Topics can be regarded as a generic category of messages. Different types of messages are distinguished by Topic names, such as User and Order. It is the first level type of message. A message must have a topic.

Tag

Tags, which can be thought of as subtopics, are the second level type of message and are used to provide additional flexibility for users. Using tags, messages for different purposes within the same business module can be identified with different tags for the same Topic. A message can have no Tag.

Group

Groups. A group can subscribe to multiple topics.

It can be divided into ProducerGroup and ConsumerGroup, which represent producers and consumers of a certain type. Generally speaking, the same service can be used as a Group, and the same Group generally sends and consumes the same messages.

Queue

Queues, each Queue is internally ordered, and in RocketMQ there are two types of queues: read queues and write queues. Generally, the number of read queues is the same.

Message Queue

Message queues, in which topics are divided into one or more subtopics, message queues. Multiple message queues can be set up under a Topic, and RocketMQ polls all queues under that Topic to send the message.

The physical management unit of messages. There can be multiple queues under a Topic. The introduction of queues enables message storage to be distributed and clustered, with horizontal expansion capability.

Offset

In RocketMQ, all message queues are persistent, data structures of infinite length. By infinite length, each storage unit in the queue is of a fixed length, and the storage unit is accessed using Offset, which is Java Long, 64-bit, Theoretically it won’t overflow for 100 years, so it’s supposed to be infinite.

You can also think of a Message Queue as an array of infinite length, with Offset being the subscript.

The above contents refer to “Shallow In shallow Out” -RocketMQ by: Ao Bing Link: juejin.cn/post/684490…

2. Deployment and installation

2.1 Installation Package Installation

  • (1) Download the installation package

Download the latest version 4.9.3, wget downloads.apache.org/rocketmq/4….

  • (2) Decompress

Unzip rocketmq – all – 4.9.3 – bin – release. Zip

  • Start Nameserver

Command: bin/mqnamesrv

Background startup command: nohup sh mqnamesr&

There is an error: Please set the JAVA_HOME variable in your environment, We need Java (x64)

Reason: in the/usr/local/rocketmq/bin/runserver then executes. Sh file specifies the JDK directory, but the JDK installation on the machine is not in the directory

Solution: Find the JDK directory specified in the runserver.sh file, comment it out, and re-write your own JDK installation directory.

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/local/jdk1.8.0_321
Copy the code

The system is restarted successfully

  • Start the broker

Command: bin/mqbroker -n 42.194.222.32.9876 (#-n specifies nameserver address and port)

Java HotSpot(TM) 64-bit Server VM Warning: INFO: OS :: Commit_memory (0x00000005C0000000, 8589934592, 0) failed; error=’Cannot allocate memory’ (errno=12)

Cause: RocketMQ is configured in a production environment by default, and the JVM memory size is set to a large value. The startup failed because of insufficient memory. Procedure

Solution: Adjust the default memory size parameters.

Example Modify the runserver.sh file

Modify the runbroker.sh file

The system is restarted successfully

  • (5) Test sending messages

Export NAMESRV_ADDR = 42.94.222.32:9876

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer

Sending succeeded.

  • (6) Test receiving messages

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer

Receiving succeeded.

2.2 Docker Installation

2.2.1 Pulling a Mirror

Docker pull foxiswho/rocketmq: server – 4.9.3

Docker pull foxiswho/rocketmq: broker – 4.9.3

2.2.2 Creating a Nameserver Container

docker create -p 9876:9876 –name rmqserver -e “JAVA_OPT_EXT=-server -Xms128m -Xmx128m -Xmn128m” -e “JAVA_OPTS=-Duser.home=/opt” -v /rmq/rmqserver/logs:/opt/logs -v /rmq/rmqserver/store:/opt/store Foxiswho/rocketmq: server – 4.9.3

2.2.3 Creating a Broker container

docker create -p 10911:10911 -p 10909:10909 –name rmqbroker -e “JAVA_OPTS=-Duser.home=/opt” -e “JAVA_OPT_EXT=-server -Xms128m -Xmx128m -Xmn128m” -v /rmq/rmqbroker/conf/broker.conf:/etc/rocketmq/broker.conf -v / RMQ rmqbroker/logs: / opt/logs – v/RMQ/rmqbroker/store: / opt/store foxiswho/rocketmq: broker – 4.9.3

2.2.4 Starting the Container

docker start rmqserver rmqbroker

After the test, it can send and receive messages normally.

2.2.5 Stopping Deleting containers

docker stop rmqbroker rmqserver

docker rm rmqbroker rmqserver

2.3 Deploy RocketMQ UI management tools

RocketMQ provides a UI management tool called RocketMQ-Console, project address: github.com/apache/rock…

The tool supports both Docker and non-Docker installations. Here we choose to use Docker installations

2.3.1 Pulling a Mirror

Pull styletang/rocketmq - the console - ng: 1.0.0

2.3.2 Creating and starting a container

The run - e "JAVA_OPTS = - Drocketmq. Namesrv. Addr = 42.194.222.32:9876 - Dcom. Rocketmq. SendMessageWithVIPChannel = false" -p 8082-8080 - t styletang/rocketmq - the console - ng: 1.0.0

2.3.3 Browser access test

Access through a browser:http://42.194.222.32:8082/#/Switch the language to Chinese and you can see everything at a glance.

conclusion

That is the main content of this article. This article introduces the basic concepts and installation of RocketMQ. Thank you for reading. If you have any questions, feel free to leave a comment in the comments section

Stay tuned for our next post.