1. Usage scenarios of messaging middleware

There are many scenarios used by message-oriented middleware, but the main ones are: asynchronous, decoupled, peak elimination and valley filling.

2. Advantages and disadvantages of message-oriented middleware

Asynchronous, decoupled, peak elimination and valley filling are the biggest advantages of message-oriented middleware, in addition to these message-oriented middleware can solve some problems in our particular business scenarios.

The disadvantages mainly lie in the availability, complexity and consistency of the system. After the introduction of message-oriented middleware, the availability of middleware needs to be considered, and the complexity is obviously increased, and the problems such as message repetition and sequential consumption need to be considered.

3. The RabbitMQ is what

RabbitMQ is an open source IMPLEMENTATION of AMQP written on the server in Erlang and supports a variety of clients. It is used to store and forward messages in distributed systems, and has good performance in ease of use, scalability, and high availability.

4. The RabbitMQ characteristics

  • reliability
  • High availability
  • Multilingual client
  • Message cluster
  • A variety of protocols
  • Plug-in mechanism
  • Tracking mechanism
  • Flexible routing

5. Install and run RabbitMQ

Go to the official documentation to download RabbitMQ:

www.rabbitmq.com/download.ht…

Follow the steps on the official website to download the file and install it

To start the RabbitMQ service, run sudo systemctl start rabbitmq-server

Run the sudo systemctl status rabbitmq-server command to query the rabbitmq-server status

To enable rabbitmq-server, run sudo systemctl enable rabbitmq-server

6. The RabbitMQ role

administrator

Policymaker and Monitoring anything you can do plus:

  • Create and delete virtual hosts
  • View, create, and delete users
  • View the create and delete permissions
  • Close connections for other users

none

The Management Plugin cannot be accessed

management

Anything a user can do with AMQP plus:

  • List the virtual hosts you can log into through AMQP
  • View queues, exchanges, and Bindings in your virtual hosts
  • Check and close your channels and connections
  • View “global” statistics about your own virtual hosts, including the activities of other users in those virtual hosts

policymaker

Anything management can do plus:

  • View, create, and delete policies and parameters of your own virtual hosts

monitoring

Anything management can do plus:

  • List all virtual hosts, including those they cannot log in to
  • Check connections and Channels for other users
  • View node-level data such as clustering and memory usage
  • View real global statistics for all virtual hosts

7. It agreement

The Advanced Message Queuing Protocol (AMQP) is an open application-layer Protocol standard designed for message-oriented middleware

8.RabbitMQ core concepts

A Producer is a party that delivers messages. The producer creates the message and publishes it to RabbitMQ

A message can generally contain two parts: the message body and additional information.

Message body: In practice, a message body is typically a piece of data with a business logical structure, such as a JSON string. Of course, you can serialize the message body as a step forward.

Additional information: Used to express the message, such as the name of the destination switch, routing keys, custom attributes, and so on

Consumer: the party that receives the message, the Consumer connects to the RabbitMQ server and subscribes to the queue.

Broker: Service node of message-oriented middleware

For RabbitMQ, a RabbitMQ Broker can simply be regarded as a RabbitMQ service node, or an instance of a RabbitMQ service, or as a RabbitMQ server

Virtual Host: A Virtual Host represents a group of switches, message queues, and related objects. A Virtual Host is an independent server domain that shares the same authentication and encryption environment.

Each Vhost is essentially a mini RabbitMQ server with its own queue, switch, binding and permission mechanism.

Channel: a lightweight Connection built on top of a Connection

Most of the operations are done in the Channel interface, including queueDeclare defining queues, exchangeDeclare declaring switches, queueBind binding queues, basicPublish, basicConsume, and so on

RoutingKey: a RoutingKey. When a producer sends a message to the exchange, it specifies a RoutingKey that specifies the routing rules for the message.

With a fixed exchange type and BindingKey, producers can specify a RoutingKey when sending a message to the exchange to determine where the message will go.

Exchange: An Exchange in which producers send messages to exchanges that route messages to one or more queues. If the route is not available, it is either returned to the producer or discarded.

Exchange types commonly used has the fanout Exchange types, the RabbitMQ direct, topic, four kinds of headers

Fanout: a fan switch that routes all messages sent to the switch to all queues bound to the switch. Direct: a direct switch that routes messages to queues where BindingKey and RoutingKey match exactly. Topic: a topic switch similar to Direct But it can do fuzzy matching with wildcards

Headers: a header switch that does not rely on the matching rules of routing keys to route information but matches the HEADERS attribute in the sent message content. This type of switch is not recommended because of poor performance

Queue: The internal object of RabbitMQ for storing messages

RabbitMQ binds the switch to the queue, usually by specifying a Binding key so RabbitMQ knows how to route messages to the queue correctly.

9.RabbitMQ sends messages

The process by which a producer sends a message

  1. A producer connects to a RabbitMQ Broker, establishes a Connection, and opens a Channel
  2. The producer declares a switch and sets properties such as switch type, persistence, and so on
  3. The producer declares a queue and sets its attributes, such as whether it is exclusive, persistent, and automatically deleted
  4. The producer binds the switch to the queue through the routing key
  5. The producer sends a message to the RabbitMQ Broker containing information about routing keys and switches
  6. The corresponding switch looks for a matching queue based on the routing key it receives
  7. If found, the message sent by the producer is queued
  8. If not, discard or roll back to the producer based on the attributes of the producer configuration
  9. Close the channel. Close the connection

The process by which a consumer receives a message

  1. A consumer connects to a RabbitMQ Broker, establishes a Connection, and opens a Channel
  2. The consumer requests the RabbitMQ Broker for a message in the corresponding queue, possibly setting the corresponding callback function
  3. The consumer waits for the RabbitMQ Broker to respond and deliver the message to the corresponding queue
  4. The consumer acknowledges the message received by the ACK
  5. RabbitMQ removes the corresponding acknowledged message from the queue
  6. Close the channel. Close the connection

10. Common problems with RabbitMQ

  • 1. How to ensure that messages are not consumed repeatedly

The consumer does idempotent, and the consumer does message de-duplication through msgId

  • 2. How to ensure the order of messages

One consumer, one queue, one thread consumes messages in a queue

  • 3. How to solve message backlog

    • Fix the problem with the consumer and make sure it gets back up to speed, then stop all existing consumers
    • Create a new topic with 10 times as many partitions and temporarily create 10 or 20 times as many queues
    • Write a temporary consumer program that distributes data. This program is deployed to consume the backlog of data. After consumption, it does not do time-consuming processing, but directly polls and writes to the 10 times as many queues that are temporarily created
    • The consumers are then temporarily deployed on 10 times as many machines, with each batch consuming a temporary queue of data
    • This approach is equivalent to temporarily expanding queue resources and consumer resources by 10 times, consuming data at 10 times the normal speed and increasing consumers’ consumption power
    • Once the backlog is quickly consumed, the original deployment architecture needs to be restored to consume messages with the original consumer machine

  • 4. Order timeout problem

1. Dead-letter queue 2. Scheduled tasks for service compensation

  • 5. Reliability
    • 1. The transaction
    • 2. Enable the confirm mechanism
    • 3. Enable persistence
    • 4. Use manual ACK

11. How to ensure high availability

RabbitMQ has three modes: single-machine mode, common cluster mode, and mirrored cluster mode

(1) Single-machine mode

Single machine mode is generally used in development or test scenarios, but not in production, which is very risky.

(2) Common cluster mode

The normal clustering mode is to start multiple RabbitMQ instances, and the queue you create will be placed on only one Rabbtimq instance, but each instance will synchronize the metadata of the queue. After consuming, if you connect to another instance, that instance will pull data from the instance of the queue.

If an instance of a storage queue goes down, other instances will not be able to pull from the previous instance. If you enable message persistence and allow RabbitMQ to drop messages, they will not be lost until the instance recovers. The main purpose of this solution is to improve throughput by having multiple nodes in the cluster service reads and writes to a queue

(3) Mirror cluster mode

Mirrored cluster mode is the high availability mode of what is called RabbitMQ. Unlike normal cluster mode, the queue you create, the metadata and messages in the queue are stored in multiple instances, and each time you write a message to the queue, the message is automatically synchronized to the queue of multiple instances.

The advantage is that if any one of your instances goes down, the others can use it.

The disadvantage is that the performance overhead is too high and the scalability is low. Synchronization of all instances will lead to high network bandwidth and pressure, and the scalability is low. Each additional instance will contain all the data of the existing queue, and there is no way to linearly expand the queue.

You can add a policy to the RabbitMQ management console that requires data to be synchronized to all nodes, or to a specified number of nodes, and when you create a queue again, apply this policy and data will be automatically synchronized to other nodes.