origin

I’ve known Apache RocketMQ since the beginning of open source, so to speak. At that time, it was still young, there was no mature community, there was no good mechanism to operate. In itself, it’s not a mature product.

However, driven by the strong technical background of Ali, with the support of business, it has gradually achieved long-term development, more and more to the public’s vision, and has gradually become an excellent message queue middleware.

In the selection of technology, as a leader, to provide technical support for each technical leader.

Edge to the business

For the traditional business scenarios and related products of our company, message-oriented middleware is generally not involved. Fortunately, with the development and expansion of the company, new products and new business scenarios have been expanded. Our company has also participated in the construction of related business scenarios of smart city.

A lot of products belong to the Internet platform, therefore, the demand for message queue middleware will be born, it is more urgent. With the help of message queues, which reduce the time required and peak cutting, and reduce the system coupling, build more robust platform products.

Practice comes from business and goes to business. This sharing is about the construction of an enterprise service platform.

The service scenario is as follows:

Our company and the government Big Data Bureau, to participate in the construction of a new Internet platform between enterprises, enterprises and government interconnection, information exchange. The platform provides the most convenient communication entrance for enterprises.

According to the needs of the government, enterprises can realize a series of advanced, excellent, convenient and fast government services such as policy declaration and policy enjoyment online. At the same time, enterprises can realize online service application and online communication with other enterprises in the platform. Among them, there will be a module in the enterprise personal center on the platform will be the message center. Shows how the enterprise receives notifications from various sources.

The scene is as follows:

Good luck with the technical

In order to reduce the coupling of the system, message queue middleware is selected to complete the construction of message center.

Technology selection

There are four common open source distributed message queues: Kafka, ActiveMQ, RabbitMQ and RocketMQ.

In the world of distributed message queues, Kafka plays a very important role because of its excellent performance.

Kafka as a streaming platform has three capabilities:

  1. Publish and subscribe record flows, similar to message queues or enterprise messaging systems;
  2. It has fault tolerance and can store record streams in a persistent manner.
  3. When a flow of records is generated (occurs), it can be processed in a timely manner.

Kafka is suitable for two types of applications:

  1. Establish real-time streaming data pipelines to reliably obtain data between systems or applications;
  2. Build real-time streaming applications that transform or react to data streams.

Currently, Kafka is mostly used for big data platform projects with large data volumes.

ActiveMQ is produced by Apache. According to the official website, it is the most popular and powerful open source message bus. ActiveMQ is fast, supports clients and protocols in multiple languages, and can be easily embedded into an enterprise application environment with many advanced features.

ActiveMQ is developed based on Java language and can assume more responsibilities from the convenience of use and the degree of open source. However, currently due to

  1. Low community activity, slow update, increased maintenance costs;
  2. Network data show that ActiveMQ has some puzzling problems and will lose messages.
  3. At present, the official focus is on The next generation of ActiveMQ 6.0 product Apollo, and there is less maintenance for 5.x.
  4. Not suitable for applications with thousands of queues.

Therefore, it is not suitable for updating the speed of iteration of the Internet platform.

RabbitMQ is a popular open source message queue system. It supports a variety of clients, such as Python, Ruby,.NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc. It supports AJAX and persistence. It is used to store and forward messages in distributed systems, and has good performance in ease of use, scalability, and high availability. However, currently due to

  1. Although combined with the concurrency advantages of Erlang language itself, the performance is good, but it is not conducive to secondary development and maintenance;
  2. The broker architecture is implemented, meaning messages can be queued on the central node before being sent to the client. This feature makes RabbitMQ easy to use and deploy, but slow to run because of the added latency of the central node and large message encapsulation;
  3. Complex interfaces and protocols need to be learned, which is costly to learn and maintain.

Therefore, all things considered, there is no choice.

RocketMQ is a distributed queue developed by Alibaba’s r&d team. It focuses on the sequential delivery of messages and features high throughput and reliability. RocketMQ, like ActiveMQ, is implemented in the Java language, designed with reference to Kafka, and has made its own improvements to be better than Kafka in terms of message reliability. RocketMQ has been adopted by several large Internet companies in the industry.

Practice is the sole criterion for testing truth, and the expression in practice is more convincing than words. Within Alibaba, RocketMQ serves thousands of applications, large and small. On The day of Singles’ Day every year, an incredible trillions of messages flow through RocketMQ (on the day of Singles’ Day in 2017, The whole Alibaba Group’s online message flow through RocketMQ reached trillions, with a peak TPS of 56 million), playing a pivotal role in Alibaba’s strategy of greater China and Taiwan.

Currently, RocketMQ was chosen because the community was active and open source, and it stood the test of the business scenario.

Deployment architecture

RocketMQ supports customization of deployment scenarios and highly available deployment solutions.

For the construction of the current platform, there are DMZ zone and Jinhong network zone network isolation environment. There are some complex network requirements for the middleware deployment of the platform.

Use the deployment architecture diagram from the official website:

For our deployment in practice, official deployment practices are also used. The multi-master and multi-slave asynchronous replication mode is adopted. Even if the disk is damaged, the message loss is very small, and the real-time performance of the message is not affected, because after the Master is down, consumers can still consume from the Slave. This process is transparent to the application, does not require manual intervention, and the performance is almost the same as the multi-master mode.

The technical implementation

For RocketMQ client consumers and server producers, Springboot is a very simple technology development framework to implement.

Introduce corresponding POM dependencies

<dependency> <groupId>org.apache.rocketmq</groupId> <artifactId>rocketmq-spring-boot-starter</artifactId> The < version > 2.0.3 < / version > < / dependency >Copy the code

Apply configuration file modification


rocketmq:
  name-server: xxxx:9876
  producer:
    group: base_group_syncMsg
    send-message-timeout: 5000
    retry-times-when-send-failed: 2
    max-message-size: 4194304
Copy the code

Corresponding to add relevant annotations, business code can be implemented.

Easy to use, powerful, very friendly for developers, is a good choice.

Edge and distributed transactions

As I write here, I also describe the many scenarios in which distributed transactions can occur with the adoption of microservice technology solutions. So, in addition to self-implementation, distributed transaction framework, at the same time, we can use message queue implementation.

In this practice, we have had further practice. I will not say more here.

Good luck with the future

In practice, the RocketMQ landing was bumpy, but it worked as expected. The documentation of the community also meets the actual needs of the implementation and is generally good.

Here, also hope that the community do better and better, the future can be expected.