What is RocketMQ

RocketMQ is a distributed messaging and streaming data platform with low latency, high performance, high reliability, trillion-scale capacity, and flexible scalability. RocketMQ is the third generation distributed messaging middleware of Alibaba open source in 2012. On November 21, 2016, Alibaba donated RocketMQ to Apache Software Foundation. On February 20 of the following year, the Apache Software Foundation announced Apache RocketMQ as a top-level project. — Wikipedia

To simplify things, RocketMQ is simply a MessageQueue, or MessageQueue. Even simpler, it’s a queue.

Why RocketMQ

Rocketmq.apache.org/docs/motiva…

According to Ali’s research, ActiveMQ will have IO bottleneck in the case of more and more queues and topics. Kafka can solve I/O bottlenecks, but suffers from low latency and high reliability. So Ali decided to build his own rocket, RocketMQ.

RocketMQ applies to this scenario

  • Decoupling between systems (i.e. no direct RPC synchronous calls between microservices)
  • Peak flow clipping (can be understood as flow limiting, to ensure service stability and maximize performance in the case of heavy flow and high concurrency)
  • Data distribution (MQ has a broadcast mode that can be used by multiple downstream systems after an upstream system publishes a message)

How does RocketMQ install, start, and shut down

Installation (For Windows,Linux, and Mac)

  • Step1: download RocketMQ

Open a browser to download address: www.apache.org/dyn/closer….

  • Step2: Click the link to download it

  • Step3: Download completed

  • Step4: Double-click to decompress

Or decompress the command line

Unzip rocketmq – all – 4.8.0 – source – the zip

  • Step5: Package and compile

mvn -Prelease-all -DskipTests clean install -U

If the following information is displayed:

aaron@aarondeMacBook-Pro rocketmq-all-4.8.0-source-release % mvn-prelease – all-dskiptests clean install -u

zsh: command not found: mvn

It is recommended to install Maven environment first, you can baidu to install.

If you run step5 successfully, but the result is as follows:

This is because your environment configuration file does not include JAVA_HOME. I’ll just add it. As for how to add JAVA_HOME, this self baidu.

When you see the image below, you have compiled successfully and installed successfully

Start the RocketMQ

After compiling RocketMQ, go to the target directory first

CD target/rocketmq – 4.8.0 / rocketmq – 4.8.0

Because RocketMQ consists of a Name Server and Broker.

Here, we enter the command to start RocketMQ’s Name Server.

nohup sh bin/mqnamesrv &

Then run the following command to print the startup log for starting Name Server

tail -f ~/logs/rocketmqlogs/namesrv.log

When you see the following log, you have successfully started the Name Server

Next, to start the Broker, type the following command:

nohup sh bin/mqbroker -n localhost:9876 &

Then print the startup log that starts the Broker with the following command

tail -f ~/logs/rocketmqlogs/broker.log

When you see the following log, you have successfully started the Broker

Send and consume messages

Enter the following command to set the IP and port of NameServer. This is to tell producers and consumers about NameServer’s environment variables.

export NAMESRV_ADDR=localhost:9876

Send a message

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer

Successful sending:

News consumption

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer

Identification of successful Consumption

Close the RocketMQ

Close the Broker

sh bin/mqshutdown broker

Turn off the Borker success flag

Close the NameServer

sh bin/mqshutdown namesrv

Successful closing of NameServer

RocketMQ vs. Kafka

Data reliability

  • RocketMQ supports asynchronous real-time flush, synchronous flush, synchronous replication, and asynchronous replication

  • Kafka uses asynchronous flush mode, asynchronous replication/synchronous replication

Summary: RocketMQ’s synchronous flush is more reliable than Kafka’s on a single machine, without data loss due to OS crashes. Kafka synchronous Replication theoretically performs worse than RocketMQ synchronous Replication because Kafka’s data is organized by partitions, meaning that a Kafka instance can have hundreds of data partitions, whereas RocketMQ has only one data partition per instance. RocketMQ can take full advantage of the I/O group Commit mechanism, bulk transfer of data, and configure synchronous Replication with a performance loss of about 20 to 30 percent compared to asynchronous Replication. Kafka has not tested this personally, but I believe it is theoretically lower than RocketMQ.

The performance comparison

  • Kafka single-machine write TPS is about 1m/SEC and message size is 10 bytes

  • A RocketMQ single machine can write about 70,000 TPS single instances per second, with three brokers deployed on a single machine, running up to 120,000 TPS single instances per second, and message sizes of 10 bytes

Bottom line: Kafka’s TPS runs to millions on a single machine mainly because the Producer side merges small messages and sends them in batches to the Broker.

Why didn’t RocketMQ do that?

  • In the Java language, which is commonly used, too many messages are cached, and GC is a serious problem

  • When the Producer invokes the interface for sending messages, the message is not sent to the Broker and returns a success message to the service. In this case, the Producer breaks down, resulting in message loss and service errors

  • Since the Producer is usually a distributed system and each machine is sending multiple threads, we believe that an online system with a single Producer can produce only a limited amount of data per second, not tens of thousands.

  • The function of caching can be completed by the upper-layer business.

Number of queues supported by a single machine

  • When a Kafka single machine has more than 64 queues/partitions, the Load increases significantly. The more queues, the higher the Load, and the longer the response time of sending messages. The number of Kafka partitions cannot be too high

  • RocketMQ standalone machines support up to 50,000 queues without significant load changes

What’s so good about queues?

  • A single machine can create more topics because each topic is made up of a batch of queues

  • The cluster size of consumers is proportional to the number of queues. The more queues there are, the larger the consumer cluster can be

Real-time message delivery

  • Kafka uses short polling, and real-time is dependent on the polling interval. After 0.8, long polling is supported.

  • RocketMQ uses long polling, which is as real-time as Push, with delivery delays of messages typically within milliseconds.

Consumption failure retry

  • Kafka consumption failure does not support retry.

  • RocketMQ consumption failure Periodic retry is supported. The retry interval is extended

Conclusion: For example, in the recharge application, the operator gateway is invoked at the current moment and the recharge fails, which may be due to pressure from the other party

Force is too much, later call will be successful, such as alipay to the bank deduction is similar to the demand. Retries here require reliable retries, meaning that failed retries are not lost due to a Consumer outage.

Strict message order

  • Kafka supports message ordering, but when an agent goes down, messages get out of order

  • RocketMQ supports strict message ordering. In a sequential message scenario, when a Broker goes down, messages fail to be sent, but not out of order

  • MySQL binary log distribution requires strict message order

Timing of the message

  • Kafka does not support timed messages

  • RocketMQ supports two types of timed messages

  • RocketMQ only supports the timing level, which can be customized by users

  • Alicloud MQ specifies the millisecond delay time

Distributed transaction messages

  • Kafka does not support distributed transaction messages

  • Alicloud MQ supports distributed transaction messaging, and a future open source version of RocketMQ is planned to support distributed transaction messaging

Information query

  • Kafka does not support message query

  • RocketMQ supports querying messages by message identity as well as by message content (specifying a message key when sending a message, an arbitrary string, such as specifying an order number)

Conclusion: Message query is very helpful in locating message loss problems, such as an order processing failure, whether the message was not received or received incorrectly.

Message back

  • Kafka can theoretically trace messages back by offsets

  • RocketMQ supports the ability to trace messages back in time to milliseconds, such as re-consuming messages from a certain hour or a certain second earlier in the day

Conclusion: In a typical business scenario, for example, a consumer does order analysis, but due to the failure of program logic or dependent system, all the messages consumed today are invalid and they need to be consumed again from midnight yesterday. Then the message replay function based on time is very helpful to the business.

Consumption parallelism

  • The consumption parallelism of Kafka depends on the number of partitions configured for the Topic. For example, if the number of partitions is 10, a maximum of 10 machines can consume in parallel (each machine can only open one thread), or one machine can consume in parallel (10 threads can consume in parallel). That is, the parallelism of consumption is consistent with the number of partitions.

  • RocketMQ consumption parallelism comes in two cases

  • The parallelism of sequential consumption is exactly the same as Kafka

  • The out-of-order parallelism depends on the number of Consumer threads. For example, if Topic is configured with 10 queues, 10 machines consuming, and each machine has 100 threads, then the parallelism is 1000.

Message trajectory

  • Kafka does not support message traces

  • Ali Cloud MQ supports message traces

Develop language friendliness

  • Kafka is written in Scala

  • RocketMQ is written in the Java language

Brokerage end message filtering

  • Kafka does not support proxy message filtering

  • RocketMQ supports two proxy message filtering methods

  • Filter by message variables, equivalent to subtopic concepts

  • Upload a piece of Java code to the server that can do any form of filtering on messages, even breaking up the Message body.

Message stacking capability

  • Theoretically Kafka is more stackable than RocketMQ, but RocketMQ can also support hundreds of millions of messages on a single machine, which we believe is sufficient for the business.

Open source community activity

  • The Kafka community updates slowly

  • RocketMQ’s GitHub community has 250 individuals, company users have registered contact information, and the QQ group has more than 1,000 people. MQ ### Commercial support

  • Kafka’s original development team has set up a new company, so far no related products have been seen

  • RocketMQ has been commercialized on Alibaba Cloud and is now available as a cloud service. RocketMQ promises 99.99% reliability to customers and completely eliminates the operation and maintenance complexity of customers’ own MQ products

maturity

  • Kafka is mature in the logging space

  • RocketMQ has a large number of applications in alibaba Group. It generates massive messages every day and successfully supports the massive message tests of Tmall Double 11. It is a sharp tool for data peak clipping and valley filling.

reference

  • Apache RocketMQ
  • RocketMQ