The introduction
Hello, everyone, I am South orange, from contact with Java to now also have almost two years, two years, from a Java have several data structures do not understand super small white, to now understand a little bit of advanced small white, learned a lot of things. The more knowledge is shared, the more valuable, I this period of time summary (including from other big guy over there to learn, quote) some of the focus in the ordinary study and interview (self think), hope to bring some help to everyone
This is a message middleware article, if you haven’t read it, you can follow it
- [Advanced Path] Message Queue — Principle and Selection (1)
- Message Queues — Principles of RabbitMQ (2)
First, basic concepts
We’ve looked at RabbitMQ from the previous chapter, so let’s look back a little bit:
-
RabbitMQ is easy to install and deploy (single-instance or clustered), has rich functions, and complies with the AMQP standard
-
RabbitMQ clusters are easy to expand and shrink. You can flexibly expand and shrink the RabbitMQ cluster by adding or deleting node instances in the cluster based on service access requirements
-
The enterprise message queue middleware has been verified by a large number of practice cases in the production environment of various companies in the industry, and has high reliability
-
RabbitMQ provides an easy-to-use user interface that allows users to monitor and manage many aspects of the message Broker. If a message is abnormal, RabbitMQ provides a message tracing mechanism to find out what has happened.
-
Supports message persistence, message acknowledgement, and flexible task distribution
-
To achieve high availability, the queues can be mirrored on the machines in the RabbitMQ cluster so that they are still available in the event that some nodes fail.
But that’s just why it’s used, there’s a lot more to learn about RabbitMQ to really make it work for our systems.
1, Message
Messages, which are the foundation of MQ, consist of message headers and message bodies. The message body is the information passed from the producer to the consumer, while the message header consists of a series of optional attributes such as routing-key, priority (priority over other messages), delivery-mode (indicating that the message may require persistent storage), and so on.
2, the Publisher
The message producer is also a client application that publishes messages to the exchange.
3, Exchange
A switch that receives messages from producers and routes them to queues on the server.
4, Routing keys
A routing key is a key that the switch looks at and uses to decide how to distribute a message to the queue
5, the Binding
Binding for association between message queues and exchanges. A binding is a routing rule that connects a switch to a message queue based on a routing key, so you can think of a switch as a routing table made up of bindings.
* * 6, Queue * *
Message queues, which hold messages until they are sent to consumers. It is the container and destination of the message. A message can be put into one or more queues. The message stays in the queue, waiting for a consumer to connect to the queue and pick it up.
7, the Connection
Connect the TCP connection between RabbitMQ and the application server
8 Channel.
Channel, a separate two-way data flow channel in a multiplexing connection. A channel is a virtual connection built on the mainland of a real TCP connection. AMQP commands are sent out through the channel. Whether it is to publish a message, subscribe to a queue or receive a message, these actions are completed through the channel. Because TCP is expensive for the operating system to set up and destroy, the concept of a channel is introduced to reuse a TCP connection.
9, Consumer
A consumer of a message, representing a client application that gets a message from a message queue.
10, Virtual Host
A virtual host that represents a batch of switches, message queues, and associated objects. A virtual host is an independent server domain that shares the same authentication and encryption environment. Each vhost is essentially a mini RabbitMQ server with its own queues, switches, bindings, and permissions. The Vhost is the foundation of the AMQP concept and must be specified at connection time.
Second, communication process
After understanding the basic structure of RabbitMQ, let’s use a picture to explain the communication process of RabbitMQ:
Is not a very beautiful painting, and all at once can be learned before the use of ~
Process analysis:
- 1. The message producer connects to the RabbitMQ Broker, creates a connection, and opens a channel.
- The manufacturer declares the switch type, name, persistence, etc.
- 3. The producer sends the message with attributes such as persistent or not and routing key.
- 4. After receiving the message, exchange routes the message to the queue bound to the current switch according to the routing key.
- 5. After receiving the message, the consumer listens for business processing, and then sends an ACK to confirm that the message has been consumed (either manually or automatically).
- After receiving an ACK, the RabbitMQ Broker deletes it from the queue.
3. Message routing mode
Borrowed the pictures from the big guy
1. Direct Mode
If the routing key in the message is the same as the Binding key in the Binding, the switch sends the message to the corresponding queue. Direct mode is a one-to-one, unicast mode.
In this mode, there is no need to bind Exchange, and a “RoutingKey” is required for message delivery, which can be simply understood as the name of the queue to be sent. However, if the queue name specified in the RoutingKey does not exist in the vhost, the message will be discarded.
2. Fanout
The Fanout switch does not handle routing keys, but simply binds queues to the switch, and each message sent to the switch is forwarded to all queues bound to the switch. Much like a subnet broadcast, each host in the subnet gets a copy of the message. So the Fanout type forwards messages the fastest.
The broadcast mode does not require a RoutingKey. It only needs to bind an Exchange to a Queue in advance. An Exchange can be bound to multiple queues, and a Queue can be bound to multiple Exchanges. But if the Exchange receiving the message is not bound to any Queue, the message is discarded.
However, the broadcast mode has the problem of duplicate consumption because it is forwarded to all bound queues.
3. Topic
Any message sent to a Topic Exchange will be forwarded to any Queue that cares about the Topic specified in the RouteKey
Each queue has a topic of interest, all messages have a “title” (RoutingKey), and Exchange forwards messages to all queues that have a topic of interest that vaguely matches the RoutingKey. The topic pattern requires a RoutingKey, and it also requires an Exchange to be bound to a Queue in advance, providing a topic that the Queue cares about when bound. The Topic pattern has two keywords: # and *.
- “#” means zero or more keywords, such as “#.nanju.#” means that the queue cares about all messages involving Nanju (a message RoutingKey “asD.mq.nanju. Error” will be forwarded to the queue),
*Represents a keyword message, such as”*.nanju.*A message with a RoutingKey of “ASD.nanju. error” will be forwarded to the queue, whereas “asD.mq.nanju. error” will not.
Similarly, if the Exchange does not find a Queue that matches the RoutingKey, it dismisses the message.
Four, working mode
1. Simple mode: one producer, one consumer
Process:
- 1. The producer puts the message into the message queue
- 2. The consumer of a message listens to the message queue and, if there is a message in the queue, consumes it. After the message is consumed, it is automatically removed from the queue (it is likely that the message has not been properly processed by the consumer and has disappeared from the queue, resulting in the loss of the message).
Suitable for the scene:
- Simple chat system
2, work mode: one producer, multiple consumers, each consumer to obtain a unique message
Process:
- 1. The producer puts the message into the message queue
- Consumers of multiple messages listen to the content of the message queue at the same time, and the one who gets the message first is responsible for consuming the message (in the case of high concurrency, a certain message will be used by multiple consumers by default, you can set a switch to ensure that a message can only be used by one consumer).
Suitable for the scene:
- Grab a red envelope
- Random assignment
3. Subscription mode: messages sent by one producer are picked up by multiple consumers
Process:
- 1, the message producer puts the message into the switch
- 2. The switch publishes and subscribes to send messages to all message queues, and consumers of the corresponding message queues get the messages for consumption
Suitable for the scene:
- Mail issued
- Subscribe to the scene
- Message issued
Routing mode: Messages are sent to the switch and a route key is specified. Consumers need to specify a route key when binding queues to the switch
Process:
- 1. The message producer sends the message to the switch
- 2. According to the Routing key, the switch can only match the message queue corresponding to the Routing key, and the corresponding consumer can consume the message.
The Routing mode is similar to the subscription mode except that the Routing key must be specified when the queue is bound to the switch. Messages will be forwarded to the queue that matches the Routing key.
Suitable for the scene:
- Mail issued
- Subscribe to the scene
- Message issued
5. Topic mode: Matches the routing key to a certain mode. In this case, the queue needs to be bound to a mode
Process:
- 1. The message producer generates the message and delivers it to the switch
- 2. According to the key rules, the switch is fuzzy matched to the corresponding queue, and the listening consumers of the queue receive messages and consume them. Each consumer listens to its own queue and sets a routingKey with wild cards
Suitable for the scene:
- User notifications
Use RabbitMQ in practice
1. Ensure the successful delivery of messages
How to ensure the successful delivery of messages? Take the business THAT I want to do this time for example, involves the problem of money and accounting, if the message is not delivered successfully, there will be mistakes. There are too many errors. How is the user experience
There are several general solutions:
- Acknowledgement response mechanism
- Automatic message compensation mechanism
- The message falls into the library, confirming the actual state of the message
- Delayed delivery of messages, double check, callback check
Acknowledgement response mechanism
The acknowledgement reply mechanism is built-in to RabbitMQ and can be set to manual (automatic reply is very bad). When the customer receives the message, it will be displayed for acknowledgement when appropriate. The consumer says I have received the message and RabbitMQ can remove the message from the queue
channel.basicAck(envelope.getDeliveryTag(), false)
Copy the codeTo tell the message server to delete the message
Automatic message compensation mechanism
In Rabbit MQ, a compensation mechanism (i.e., a message retry mechanism) is implemented by default if the business logic fails while the consumer is processing the message. If the business logic is abnormal, the message is not consumed.
Message drop mechanism
Message drop mechanism is to write to the database for persistence, and constantly confirm the state of the message, to ensure the successful consumption of the message. The consumption status of the message can be confirmed in the database, while a machine periodically gets the consumption status of the database message and resends the message if the status is abnormal. At the same time, you can also retry too many times of the message state to fail and register, to the program monkey manual processing ~
### Delayed delivery mechanism for messages
Delay queue, the time delay of the most important features is reflected in its attributes, is not the same as ordinary queue, the common element in the queue waiting to want to be always take early treatment, and delay the elements in the queue is to be removed in the specified time and processing, so delay the elements in the queue is properties with time, Usually a message or task that needs to be processed.
Latency queues are used in RabbitMQ, which can be implemented through dead-letter queues. After a message is set to TTL (time to live), it becomes dead letter if it is not consumed within the time to live. Dead-letter messages are sent to a dead-letter queue, and consumers simply listen to the queue for processing.
2. Ensure the success of message consumption
Successful delivery is only half done, and successful consumption messages are needed to meet our needs.
RabbitMQ defaults to automatic acknowledgment (ACK) that a message has been properly consumed. That is, it is automatically acknowledged that the message has been processed after being sent to a consumer and will delete the message even if the consumer unexpectedly goes down or throws an exception. If the consumer receives the message and then goes down or throws an exception before the processing is complete, The message was lost.
Message confirmation modes are as follows:
- Snaven emode.NONE: Automatic acknowledgement
- AUTO: Confirmed as stated in the circumstances
- MANUAL: Indicates the terms of the policy annexed hereto
By default, message consumers ack messages automatically. If they ack messages manually, the acknowledgment mode needs to be changed. Also, manual validation can be batched to process a large amount of content at once.
3. Message persistence
Messages are delivered to RabbitMQ’s memory, and are lost when the RabbitMQ instance goes down before it can be delivered to the consumer instance.
Messages can be persisted. Exchange, queue, and message can be persisted to hard disk. When RabbitMQ restarts, the persistent Exchange, queue, and message will be reloaded from hard disk and redelivered.
You still need to persist messages into the database.
4, consumption end flow limit
If the RabbitMQ server has tens of thousands of unprocessed messages, we can open a consumer client and a huge number of messages will be sent to us in a moment, but we can’t handle all that data on a single client at the same time, server resources are exhausted and so on.
RabbitMQ provides a qos feature that does not consume new messages unless a certain number of messages have been acknowledged without automatic acknowledgment.
5. Sequential messages
To send sequential messages, the message must be delivered to the same queue, and the consumer can only have one (exclusive mode), and the message must be delivered uniformly (may be merged into a large message, may be split into multiple messages), and the message attribute must be added to the session ID of all messages: The sequence number of the sequence tag, and the SIZE attribute of the sequence message. One message is consumed at a time, processed by a manual ACK, and then the next message is received.
6. Repeat purchases
Repeated consumption is a very dangerous problem in the message queue, especially in the multi-consumer mode, if a message is repeatedly consumed, and it involves money, etc., there is often a capital risk. Therefore, it is necessary to solve the problem of repeated message consumption.
The key to ensuring that messages are not consumed repeatedly is to ensure the idempotence of message queues
Solutions are as follows:
You get this message to do an INSERT operation on the database. That’s easy. Give the message a unique primary key, and even if there is a duplicate consumption, there will be a primary key conflict. If you take this message and do the redis set operation, it is easy to do, because no matter how many times you set the operation, the result is the same, the set operation is originally considered idempotent operation. 3, consumer media to prepare a third party media, to do consumption records. Take Redis as an example, assign a global ID to a message, and write < ID,message> to Redis in the form of K-V as long as the message is consumed. Before consumers start to consume, they can first check whether there is a consumption record in Redis.
Vi. Set up RibbitMQ cluster
RabbitMQ clusters are classified into two types: common clusters and mirroring clusters.
1. Common cluster:
Two nodes (Rabbit01 and Rabbit02) are used as examples. Rabbit01 and Rabbit02 have the same metadata, the structure of the queue, but the message entity exists only in one of the nodes, rabbit01 (or rabbit02).
After A message is queued to Rabbit01 and A consumer consumes from Rabbit02, RabbitMQ temporarily transfers messages between Rabbit01 and Rabbit02, fetching messages from entity A and sending them to Consumer through entity B. So a consumer should try to connect to every node and get a message from it. That is, physical queues must be created on multiple nodes for the same logical Queue. Otherwise, whether the consumer is connected to Rabbit01 or Rabbit02, the export will always be at Rabbit01, which will create a bottleneck.
If the Rabbit01 node is faulty, the Rabbit02 node cannot access unconsumed message entities from the Rabbit01 node. If messages are persisted, they can be consumed only after the Rabbit01 node is restored. If there is no persistence, message loss will occur.
2. Mirror cluster:
On the basis of the common cluster, the required queue is made into a mirror queue, and the message entity will actively synchronize between the mirror nodes, rather than temporarily pull the data when the client fetch, that is, the number of copies of the message will be backed up according to the number of nodes. The side effects of this mode are obvious. In addition to system performance degradation, if there are too many mirror queues and a large number of incoming messages, the network bandwidth within the cluster will be greatly consumed by this synchronous communication. Therefore, it is suitable for high reliability requirements.
Because messages between mirror queues are automatically synchronized and there is an internal master election mechanism, even if the master node breaks down, the use of the whole cluster will not be affected, achieving the purpose of decentralization, so as to effectively prevent message loss and service unavailable
Compared with the normal mode, the storage space of each node in the mirroring mode is very large, and the message backlog capacity of the cluster is very weak (the message backlog capacity cannot be improved by expanding the cluster node capacity). At the same time, message publishers need to copy messages to each cluster node, and the overhead of synchronous network and disk replication increases significantly for persistent messages.
7. What are the disadvantages of system architecture after the introduction of message-oriented middleware
Message-oriented middleware is good, but it can be avoided or avoided as much as possible, as the decoupling of the system creates other problems.
Reduced system availability: With the introduction of MQ, the system has an additional dependency. If the dependency goes wrong, the availability of the system will be reduced. Once middleware is introduced, you must consider how the middleware will be deployed and how high availability will be guaranteed
System stability reduction: the introduction of MQ may cause a variety of messy problems due to network failure, middleware failure, consumer exceptions and other reasons, thus making the system stability decline
Distributed consistency problem: When multiple systems process a business cooperatively, it is not guaranteed that all systems can process the business normally, and the system data may be inconsistent. Therefore, it is necessary to use a distributed transaction scheme with reliable message final consistency to ensure the data consistency
Even if there are so many problems, considering that distributed architecture is now the mainstream, highly coupled system is difficult to meet the growing number of users, distributed environment needs message middleware to ensure the efficiency and reliability of message delivery, so we need to distinguish when to use what kind of message middleware. Know yourself and your enemy, and you will win a hundred battles.
conclusion
This chapter introduces RabbitMQ in a nutshell. The methods of setting up RabbitMQ are not written, and the methods of using RabbitMQ are not described. These are clearly implemented on the Internet, so you can check them out and follow the tutorial to learn more about RabbitMQ.
At the same time, if you need a mind map, you can contact me, after all, the more knowledge is shared, the more fragrant!
