How to solve RabbitMQ problems

What is MQ?

Message Queue (MQ) Message Queue

The communication between software is realized by the queue mechanism. Consumers can pull messages from the specified queue, or subscribe to the corresponding queue, and the MQ server will push messages to them

What is a queue?

Is a data structure that follows the FIFO (First in, first out) principle

Why use MQ and what are the advantages of MQ?

• Asynchronous communication

Improve performance by optimizing asynchronous operations that weren’t necessarily synchronous before

• The business of decoupling

The original module A directly invokes the interface of module B and optimizes it into that the request of module A is sent to MQ, and then the work of module A is finished

MQ will push it to Module B actively, or module B will pull it by itself

• Traffic peak clipping

When a large amount of traffic is sent to the server at a certain time, the server cannot bear it and will break down

In this case, if the requests are all coming out of the message queue, the server can still process the requests in an orderly and stable manner despite the heavy traffic

What are the disadvantages of MQ?

• The availability of the system is reduced and there are external dependencies
• The problem of MQ message loss and repeated consumption needs to be considered
• It takes effort to ensure that messages are sequential and consistent

Common MQ performance comparison

	ActiveMQ	RabbitMQ	RocketMQ	Kafka
Development of language	java	erlang	java	scala
Single machine throughput	All level	All level	Hundreds of level	Hundreds of level
timeliness	Ms level	Us level	Ms level	Ms than level
availability	high A master-slave architecture	high A master-slave architecture	Very high Distributed architecture	Very high Distributed architecture
Message reliability	Low probability of message loss	Basic don’t throw	You can do it almost without losing it	You can do it almost without losing it
Function support	Support all functions	Performance is good Low latency Strong concurrency capability	MQ features are relatively complete Support distributed, good scalability	It is used to collect big data and logs

How does MQ avoid message piling

• Increase consumption rate (cluster approach)
• Consumers get messages in bulk for consumption

How MQ can avoid consumer duplication (idempotence problem)

• Global ID (add flag bit) + Ensure service consistency

How does MQ ensure that messages are not lost

• Message acknowledgement mechanism
• persistence
• An ACK message

How does MQ ensure message order consistency

• Bind to the same consumer and queue

What is the MQ push and pull architecture model?

• After a long connection is established between the MQ server and the consumer, the MQ server will actively push the data to the consumer
• When the consumer first starts, it goes to the MQ server to pull data

What are the consumption patterns of MQ

• Push mode After registering a consumer, RabbitMQ automatically pushes messages to the consumer as they become available. This is the most efficient and timely way.

• The pull pattern is a polling model that sends a GET request and gets a message. If there are no messages in RabbitMQ at this time, it will get an empty reply.

• Automatic confirmation

When a consumer consumes a message, it is automatically confirmed to RabbitMQ.

• Manual confirmation

When the consumer consumes the message, the corresponding function is called manually to ack the message

• Qos prefetch mode

Consumers can request a certain number of messages to be received in advance before a certain number of messages are received, and confirm in batches after processing a certain number of messages

Of course, if the consumer application crashes before the message is acknowledged, all unacknowledged messages will be resended to other consumers

RabbitMQNow that there isconnectionsWhy should there bechannel?

What is the connection

A connection is a TCP connection established between the producer or consumer and the RabbitMQ Broker

Once the TCP connection is established, the client can then create AMQP channels, each of which is assigned a unique ID

A channel is a virtual Connection built on a Connection. Multiple channels reuse a TCP Connection, which reduces the performance overhead and facilitates management

Because an application needs to set up a TCP connection to generate or consume messages from RabbitMQ, TCP connections are very expensive, and performance bottlenecks can arise if there is a spike in usage

Channel multiplexing connection advantages:

• TCP connections are reused to reduce performance overhead and facilitate management
• RabbitMQ ensures the privacy of each channel

When the traffic per channel is not very large, multiplexing a single Connection can effectively save TCP Connection resources with performance bottlenecks

When the traffic of the channel itself is very high, multiple channels multiplexing a Connection will cause performance bottlenecks, which will limit the overall traffic. In this case, multiple connections need to be opened up and the channels are evenly distributed among these connections

RabbitMQThe role of

• Peak peel
• Producer and consumer business decoupling
• Asynchronous communication between services
• Timing task
• Order consumption

Why choose RabbitMQ

There are many MQ options in the market, such as ActiveMQ, ZeroMQ, Appche Qpid. Why choose RabbitMQ?

• Besides Qpid, RabbitMQ is the only messaging server that implements the AMQP standard
• Reliability. The persistence support of RabbitMQ ensures the stability of messages
• High concurrency. RabbitMQ uses Erlang, a language developed for telephone exchanges that comes with a high concurrency aura and high availability features
• Cluster deployment is simple. Erlang is what makes RabbitMQ cluster deployment super simple
• Community activity is high, and according to the Internet, RabbitMQ is preferred

What are the features of RabbitMQ?

• reliable

RabbitMQ uses mechanisms such as persistence, transport confirmation, and publish confirmation to ensure reliability

• Flexible routing

Routing messages through the exchange

RabbitMQ already provides some built-in switches for typical routing

For more complex routing functions, multiple switches can be bundled together, which needs to be implemented through plug-ins

• scalability

Multiple RabbitMQ nodes can form a cluster or dynamically expand nodes in the cluster based on service requirements

• High availability

Queues can be mirrored on the machines in the cluster so that they remain available in the event that some nodes fail

• Multiple protocols are supported

RabbitMQ supports various messaging middleware protocols such as STOMP and MQTT in addition to AMQP

• Multilingual client

RabbitMQ supports almost all common languages such as GO, Java, Python, Ruby, PHP, etc

• WEB Management Interface

RabbitMQ provides an easy-to-use user interface that allows users to monitor and manage messages, nodes in a cluster, and more

• Command plug-in mechanism

RabbitMQ provides a number of plugins to extend it in many ways, but you can also write your own plugins

What knowledge do producers and consumers have?

producers

• Message producer, delivers the message
• Messages typically contain two parts: the body (payload) and label (Label)

consumers

• Consume the message, receive the message

• The consumer connects to the RabbitMQ server and subscribes to the queue

  When consuming a message, only the message body is consumed and the label is discarded

Pits in RabbitMQ message persistence

By default, restarting the server will result in loss of messages. How can we ensure that data is not lost when restarting RabbitMQ?

This is persistence, which requires the following three conditions to restore RabbitMQ data

• The durable value is set to True when the messages are delivered
• The message has arrived at the persistence exchange
• The message has arrived on the persistent queue

How persistence works

Rabbit writes the persistent message to a persistent log file on disk, and when the message is consumed, Rabbit identifies it as waiting for garbage collection

Advantages and disadvantages of persistence

advantages

• Data persistence, data is not lost

disadvantages

• Performance is affected. Disk write performance is lower than memory write performance, which reduces service throughput

RabbitMQ ACK Reply mechanism

ACK response is divided into manual and automatic, each has its advantages and disadvantages

• Automatic ACK is convenient if the message is not important and the loss does not matter

• If the message is too important to lose. It is better to ACK manually after the consumption is completed

  Otherwise it will ACK automatically upon receipt and RabbitMQ will remove the message from the queue. If the consumer goes down or the processing fails, the message is lost

Considerations for the development of ACK mechanism

If you forget the ACK, the consequences are serious

Messages are always redistributed when the Consumer quits. RabbitMQ then takes up more and more content, and because RabbitMQ is running for so long, this “memory leak” can be fatal

Why do we need traffic limiting, consumer traffic control

• At some point, the producer has a lot of messages piled up in the RabbitMQ queue, and when a consumer starts up, a large number of messages are pushed to the consumer, and the instantaneous volume of traffic will suspend the consumer
• An imbalance in efficiency between producers and consumers can result in performance degradation on the consumer side, or server delays or crashes

The composition of the RabbitMQ

• Producers producer
• Consumers consumer
• Switch exchange

It is used to receive and distribute messages

• The message the message
• Queue queue

Used to store the producer’s messages

• Channel is the channel closer

The channel through which messages are pushed

• Connect the connections

TCP connection between the generator or consumer and Rabbit

• The routing key routingKey

Used to assign the generator’s data to the switch

• The binding key BindingKey

Used to bind messages from an exchange to a queue

• The connection manager is ConnectionFactory

A manager used in program code to establish a connection between an application and Rabbit

• VHost

A Vhost can be understood as a virtual broker, or virtual machine

Each has its own queue, exchange, binding, and so on

Have an independent permission system, achieve resource isolation, efficient use of resources

Six modes of RabbitMQ

• single

Simple producers produce messages, put them into queues, and consumers consume messages

• work

When producers can produce messages faster than consumers can consume them, consider using work mode to increase processing speed and load

The Work model is similar to the Single model, except that there are more consumers in the Work model than in the Single model

• publish

Application scenario: Use of simple message queues, one producer and one consumer

• routing

The message producer sends the message to the switch according to the route judgment, the route is a string (info) the current generated message carries the route character (object method), the switch according to the key of the route

Only the message queue corresponding to the routing key can be matched and the corresponding consumer can consume the message

• topic

In topic mode, a message is obtained by multiple consumers, and the target queue of the message can be wildcard with BindingKey

• rpc

By means of remote procedure calls

The storage mechanism

• Persistent message

Persistent messages are written to disk as they arrive in the queue. Persistent messages also have a backup in memory, which improves performance to a certain extent, and can be wiped from memory when memory becomes tight

• Nonpersistent messages

Generally, it is only kept in memory, and when memory is tight, it will be switched to disk to save memory space

There are several possible states for messages in RabbitMQ

• alpha

The message content (including the body, properties, and HEADERS) and the message index are stored in memory

• beta

The message content is kept on disk and the message index is kept in memory

• gamma

The message content is kept on disk and the message index is both on disk and in memory

• delta

The message content and index are on disk

RabbitMQ queue structure?

• rabbit_amqqueue_process

Responsible for protocol-related message processing, i.e. receiving the message published by the producer, delivering the message to the consumer, processing the acknowledgement of the message, etc

• backing_queue

Is the specific form and engine of the message store and provides the associated interface to rabbit_amqqueue_process for invocation

What happens when a switch fails to find a queue that meets the requirements based on its type and routing key?

When we set parameters to the switch, we have a flag called Mandatory

• This parameter is mandatory when the mandatory bit is set to true

If Exchange cannot find an appropriate queue to store messages based on its type and message routingKey, the broker calls the basic.return method to return the messages to the producer

• Mandatory is set to false

The preconditions are the same as above, and the broker simply discards the message

How to ensure the reliability of the information?

• Messages are sent from the producer to MQ

By the transaction mechanism, the confirmation mechanism to safeguard

• MQ own reliability

Guaranteed by persistence, clustering, common mode, mirroring mode

• MQ message to consumer

By basicAck mechanism, dead letter queue, message compensation and other mechanisms to ensure

What are the types of nodes in the cluster?

• The memory node

Ram to write changes to memory.

• Disk node

Disc, disk write operation

RabbitMQ requires at least one disk node

How do I ensure the high availability of RabbitMQ message queues?

There are three modes in RabbitMQ to ensure:

• Stand-alone mode

It is usually started locally, learned and tested on its own, and will not be used in production

• Common cluster mode

Start multiple Instances of RabbitMQ on multiple machines, one on each machine

• Mirror cluster mode

The high availability mode of RabbitMQ

Unlike normal clustering, a queue is created where the metadata (metadata is the RabbitMQ configuration data) and the messages in the queue are stored on multiple instances.

Then each time a message is written to a queue, the message is automatically synchronized across multiple instances of the queue

References:

RabbitMQ Tutorials

Welcome to like, follow and collect

Dear friends, your support and encouragement are the motivation for me to keep sharing and improve the quality

All right, that’s it for this time

Technology is open, our mentality, should be more open. Embrace change, live in the sun, and strive to move forward.

I am nezha, welcome to like the collection, see you next time ~