Hello, I’m Koha

Today we are going to talk about the development of RabbitMQ. Rabbit in RabbitMQ means Rabbit. It means to run as fast as Rabbit. Is a lightweight, highly available message queue that supports multiple messaging protocols. RabbitMQ is written in the Erlang language, which is a natural language for high concurrency.

Is not very like the rabbit, the cute rabbit in our work but the role of the big brother, said it is the darling of Ali now it is too much. RabbitMQ: What does MQ stand for? RabbitMQ is an open source Message queuing system. Follow Kohwa to find out how this little bunny developed.

preface

There are many mainstream MQS in the market, such as ActiveMQ, RabbitMQ, RocketMQ, Kafka, ZeroMQ and so on.

Alibaba also used ActiveMQ at first, but with the continuous development of the business, the ActiveMQ IO module appeared a bottleneck. Later, Alibaba could not solve the problem well through a series of optimization, and then Alibaba focused on kafka, the mainstream messaging middleware. But Kafka does not meet their requirements, especially low latency and high reliability.

So RocketMQ is standing on the shoulders of a giant (Kafka) and optimizing it to make it more of an Internet company.

RabbitMQ is a very popular messaging middleware, it has very rich features and advantages: high reliability, flexible routing, high cluster scalability, high availability, support for multiple protocols, support for multiple clients and a rich plug-in system.

RocketMQ is currently widely used in Alibaba Group for transaction, recharge, flow calculation, message push, log streaming processing, binglog distribution and other scenarios.

concept

RabbitMQ is an open source implementation of AMQP developed in the Erlang language.

AMQP: Advanced Message Queue. It is an open standard of application layer protocol, designed for message-oriented middleware. The client based on this protocol can deliver messages with message-oriented middleware, and it is not limited by product, development language and other conditions.

RabbitMQ has its origins in financial systems and is used to store and forward messages in distributed systems. RabbitMQ is very good at ease of use, scalability, and high availability.

Usage scenarios

The application of decoupling

Take the order system that we often say for example, the order system calls the interface of the inventory system generally. As follows:

This approach causes a lot of problems. When the inventory system is not accessible, the destocking of our order system will fail.

When we use the message queue, after the user place an order, the order system carries on the persistence processing, writes the message to the message queue, returns the user order to place a successful order; Subscribe the order message, using the pull/push way, obtain the order information, the inventory system according to the order information, inventory operation.

In this way, even if there is a problem with the inventory system when placing an order, the normal ordering will not be affected, because after placing an order, the order system has been written into the message queue, and other operations are not concerned, so the application of the order system and the inventory system can be decoupled.

Asynchronous processing

Take our user registration for example, users need to send registration email and registration SMS after registration. As follows:

Serial mode:

Parallel mode:

When we use message queues, business logic is no longer necessary and only asynchronous processing is required. As follows:

Traffic peak clipping

Take the second kill that we often mention for example, we must all have experienced the Double 11 of Tmall and Taobao, when this time, we will at a specific time, such as 0 o ‘clock in the evening, the demand per second will suddenly increase, if not to upgrade the system structure, is not able to withstand so many requests, directly will crash the system.

When we use the message queue, at the time of the peak, many users to come in, there are 5000 requests per second, for example, we only need to put this 5000 requests in the message queue, system to handle 2000 requests per second, will be out of the message queue to take out the corresponding number, can only deal with the 2000 requests per second. This will result in hundreds of thousands or more requests being queued for the duration of the seckill. After all, seckill only lasts for a short period of time, and by the time it’s over, there may only be dozens or hundreds of requests queued every second. But the system is still processing 2,000 requests per second. So, when the seckill is over, the system consumes all the messages that are left.

The main features

Reliability: RabbitMQ uses mechanisms to ensure Reliability, such as persistence, transmission confirmation, and release confirmation.

Flexible Routing: Routing messages through Exchange before they are queued. RabbitMQ already provides some built-in Exchange implementations for typical routing functions. For more complex routing capabilities, you can bind multiple Exchanges together and also implement your own Exchange through a plug-in mechanism.

Clustering: Multiple RabbitMQ servers can form a cluster to form a logical Broker.

Highly Available Queues: Queues can be mirrored on machines in a cluster, making them Available in the event that some nodes fail.

Multi-protocol: RabbitMQ supports multiple message queuing protocols, such as STOMP, MQTT, and so on.

Many Clients: RabbitMQ supports almost all commonly used languages such as Java,.NET, Ruby, and so on.

Management UI :RabbitMQ provides an easy-to-use user interface that enables users to monitor and manage many aspects of the message Broker.

Tracing: If a message is abnormal, RabbitMQ provides a message Tracing mechanism so that the consumer can find out what happened.

Plugin System :RabbitMQ provides a number of plugins to extend in many ways. You can also write your own plugins.

Architectural model

Message

A message, which is anonymous, consists of a message header and a message body. The body of the message is opaque, and the header consists of a series of optional attributes, including routing-key, priority (priority over other messages), delivery-mode (indicating that the message may require persistent storage), and so on.

Publisher

The message producer is also a client application that publishes messages to the exchange.

Exchange

A switch that receives messages from producers and routes them to queues on the server.

Binding

Binding for association between message queues and exchanges. A binding is a routing rule that connects a switch to a message queue based on a routing key, so you can think of a switch as a routing table made up of bindings.

Queue

A queue, which is an internal object of RabbitMQ, is used as a container to store messages and is also the destination of messages. A message can be put into one or more queues. The message stays in the queue, waiting for a consumer to connect to the queue and pick it up.

Connection

Network connection, such as a TCP connection.

Channel

Channel, a separate two-way data flow channel in a multiplexing connection. A channel is a virtual connection built on the mainland of a real TCP connection. AMQP commands are sent out through the channel. Whether it is to publish a message, subscribe to a queue or receive a message, these actions are completed through the channel. Because TCP is expensive for the operating system to set up and destroy, the concept of a channel is introduced to reuse a TCP connection.

Consumer

A consumer of a message, representing a client application that gets a message from a message queue.

Virtual Host

A virtual host that represents a batch of switches, message queues, and associated objects. A virtual host is an independent server domain that shares the same authentication and encryption environment.

Broker

Represents a message queue server entity.

Exchange type

There are four types of Exchange message distribution policies: Direct, FANout, TOPIC, and HEADERS. Headers matches the HEADERS of AMQP messages instead of the routing key. In addition, the headers switch is exactly the same as the direct switch, but the performance is much worse.

The Direct key distribution

Direct: If the routing key in the message is the same as the Binding key in Binding, the exchange sends the message to the corresponding queue. It is a perfectly matched, unicast pattern.

If we send a message to the Exchange with routingKey= “error”, the message will be routed to Queue1 (amqp.gen-s9b… Which is the Queue name automatically generated by RabbitMQ) and Queue2 (amqp.gen-agl…) ; If we send a message with a routingKey= “info” or routingKey= “warning”, then the message will only be routed to Queue2. If we send messages with other routingKeys, the messages will not be routed to either Queue.

Fanout

Fanout: Every message sent to a Fanout exchange is sent to all bound queues. Much like a subnet broadcast, each host in the subnet gets a copy of the message. The Fanout type forwards messages the fastest.

All messages sent by the producer Publisher to the Exchange are routed to the two queues in the figure and eventually consumed by two consumers (Consumer1 and Comsumer2).

Topic switch

Topic switch: A Topic switch uses pattern matching to assign the routing key property of a message and matches the routing key to a certain pattern, to which the queue needs to be bound. It cuts the string of routing keys and binding keys into words separated by dots. It also recognizes two wildcards: the symbol “#” and the symbol “”. # Match 0 or more words, match no more than one word.

Our routingKey= “quick.orange.Rabbit” message will be routed to both Q1 and Q2, and our routingKey= “lazy.orange.fox” message will be routed to Q1, A message routingKey= “lazy.brown.fox” will be routed to Q2, and a message routingKey= “lazy.pink.rabbit” will be routed to Q2 (it will only be delivered to Q2 once, Although this routingKey matches both of Q2’s bindingkeys); RoutingKey = “quick.brown.fox”, routingKey= “orange”, routingKey= “quick.orange.male. Rabbit” messages will be discarded because they do not match any BindingKeys.

Installation steps

Install Erlang before installing RabbitMQ. You can download it from the Erlang website. Go to the RabbitMQ website, download the RabbitMQ installation package, and then unzip it. The official website provides installation instructions for different oss: Windows, Debian/Ubuntu, rpm-based Linux, and Mac

Download address

https://www.rabbitmq.com/releases/rabbitmq-server/
Copy the code

download

Wget HTTP: / / https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-generic-unix-3.6.15.tar.xzCopy the code

The installation

The tar XVF - the rabbitmq server - generic - Unix - 3.6.15. Tar. XzCopy the code

configuration

Vim/etc/profile then add the export PATH = $PATH: / opt/rabbitmq/rabbitmq_server - 3.6.15 / sbin according to your own installation PATHCopy the code

Put the change into effect

source /etc/profile
Copy the code

Modify the hosts

vim/etc/hosts
Copy the code

Start the

CD /opt/rabbitmq/rabbitmq_server-3.6.15/sbin/./ rabbitmq-server-detached Start the rabbitmq-server as user rootCopy the code

Check whether the startup is successful

./rabbitmqctl status
Copy the code

This is a success. If you want remote access

/rabbitmq-plugins enable rabbitmq_management (disable firewall for non-native access)Copy the code

access

Springboot project demo

Springboot integration with RabbitMQ is very simple, if only to use very little configuration, SpringBoot provides spring-boot-starter- AMQP message support.

To configure poM files, add spring-boot-starter- AMQP support

Code demo:

Properties file, and configure the rabbitMQ installation address, port, and account information

Code demo:

Configure the queue

Code demo:

The sender

Code demo:

The receiver

Code demo:

test

Code demo:

Built in the cluster

One of RabbitMQ’s best features is built-in clustering, which is designed to allow consumers and producers to continue running in the event of a node crash, and to linearly scale message throughput by adding more nodes. RabbitMQ internally uses the distributed communication framework OTP provided by Erlang to meet these requirements. If a client loses a connection to a RabbitMQ node, it can reconnect to any other node in the cluster to continue producing and consuming messages.

Common concepts in RabbitMQ clusters:

Queue metadata: includes queue names and their properties, such as whether they are persistent or automatically deleted

Switch metadata: switch name, type, and properties

Binding metadata: Inside is a table that records how messages are routed to queues

Vhost metadata: Provides namespaces and security attributes for queues, switches, and bindings within the Vhost

Technology selection

In our usual development of various message queue MQ selection thinking:

If the user access is within the acceptable range of ActiveMQ, and it is really mainly used based on decoupling and asynchronization, ActiveMQ can be considered, which is also close to the use habit of Java engineers, but ActiveMQ has stopped maintenance now, and the concurrency of ActiveMQ is not high. Therefore, it can be used when the business volume is certain.

RabbitMQ, as a pure heritage messaging middleware, has a perfect combination of advanced messaging protocol AMQP, which is irreplaceable in the messaging middleware, but The Erlang language prevents us from in-depth research and control, for the company, the underlying technology is not controlled, but it is open source, with stable support. High activity.

If you have absolute confidence in your company’s technical strength, you can use RocketMQ. However, RocketMQ was born late and updated quickly, which means that there may be many pits in the process of use. Therefore, if your company’s Java technology is not strong, it is not recommended to use it.

Small and medium-sized companies, technical strength is relatively general, technical challenges are not particularly high, ActiveMQ, RabbitMQ is a good choice; For larger companies with strong infrastructure development, RocketMQ is a good choice

When it comes to real-time computing, log collection and other scenarios in the big data space, Kafka is the industry standard, absolutely fine, the community is very active, almost the world’s de facto specification in this field.

In terms of performance, message-oriented middleware using file systems (Kafka, RokcetMq) is the best, so message-oriented middleware based on file system storage is the trend. (File system >KV storage > relational database from storage mode and efficiency)

The last

Message queue, are actually the same, basic usage are the same, the emphasis of the various open source message queue just slightly different, we should according to our own project requirements to decide we should choose what kind of message queue to service for our project, the project selection work is usually at the beginning of the research and development, the team leader had already decided what to do, Generally, it is not our turn to do it, but the interview may be the time to check the relevant knowledge, so we should be familiar with all kinds of message queues and can use them flexibly.