preface

RabbitMQ is based on the AMQP protocol and can be passed between languages using a common protocol.

Closer agreement

The core concept

  1. Server: Also known as broker, accepts client connections and implements AMQP entity services.

  2. Connection: A connection to a specific broker network. A complete PDF version of the Java Interview handbook has been organized into a document

  3. Channel: Network channel in which almost all operations take place. A channel is a channel through which messages are read and written. A client can establish multiple channels, each representing a session task.

  4. Message: Message, data passed between the server and the application, consisting of properties and body. Properties allows you to modify messages with advanced features such as message priority and latency. Body is the message entity content.

  5. Virtual host: Virtual host used for logical isolation and routing of top-level messages. A Virtual host can have multiple exchanges and queues. A Virtual host cannot have exchanges or queues of the same name.

  6. Exchange: a switch that accepts messages and forwards them to bound queues based on routing keys.

  7. Banding: Virtual connection between exchanges and queues. Binding can include routing keys

  8. Routing key: A routing rule that the VM uses to determine how to route a message.

  9. Queue: Message Queue, used to store messages.

Exchange

Type of switch, Direct, topic, fanout, headers, Wants (Whether persistence is required true required) Auto delete When the last queue bound on Exchange is deleted Exchange is deleted.

  1. Direct Exchange. All messages sent to the Direct Exchange are forwarded to the Queue specified in RouteKey. The Direct Exchange can use the default Exchange (default Exchange). The default Exchange binds all queues, so Direct can be bound using Queue names (as routing keys). Or the consumer and producer routing keys match exactly.

  2. Toptic Exchange, in which a message sent to a Topic Exchange is forwarded to a Queue of all concerned topics in the Routing key. Exchange makes a fuzzy match between the routing key and a Topic, and the queue needs to be bound to a Topic. “#” matches one or more words, and “” matches only one word. For example, “log. #” matches “log.info.test”. “log. “matches only log.error.

  3. Fanout Exchange: Does not deal with routing keys and simply binds queues to the switch. Messages sent to the switch are sent to the queue bound to the switch. Fanout retweets are the fastest.

How to ensure 100% delivery of messages

What is reliability delivery at the production end?

  1. Ensure successful delivery of messages

  2. Ensure that the MQ node is successfully received

  3. The sending MQ node (broker) receives an acknowledgement message

  4. Improve the message compensation mechanism

Reliability delivery guarantee scheme

The message falls into the database and marks the message

A complete PDF of the Java Interview Handbook has been documented

In high concurrency scenarios, each DB operation costs performance per operation. We use delay queues to reduce one database operation.

Message idempotency

If I perform an action, we can perform it 100 times, 1000 times, and the result must be the same for all 1000 times. For example, if a single thread updates an update count-1 a thousand times, the result is the same, so the update is idempotent. If a single thread updates an update count-1 a thousand times without thread-safe processing, the result may not be the same. Therefore, the concurrent update operation is not an idempotent operation. Corresponding to the message queue, even if we receive multiple messages, the effect is the same as consuming one message.

How to avoid repeated message consumption in high concurrency situations

  1. Unique ID + fingerprint code, using the database primary key deduplication. Advantages: Simple implementation Disadvantages: High concurrency has a data write bottleneck.

  2. Use the atomicity of Redis to practice. Idempotent with Redis is something to consider

  • Are databases dropped, and how can data and cache be idempotent (how can Redis and the database both succeed and fail at the same time)?

  • How about if you don’t drop it, you put it all in Redis which is the Redis and database synchronization strategy? And is it 100% successful in cache?

Confirm indicates the confirmation message and Return indicates the Return message

Understand the CONFIRM message confirmation mechanism

  • Acknowledgement of a message means that after a producer receives a posted message, if the Broker receives the message, it sends an acknowledgement to our producer that the Broker received the message.

How to implement confirm confirmation messages.

  • ConfirmSelect () : channel.confirmSelect()

  • Add a listener to a channel: addConfirmListener, which listens for success and failure results that are either resent or logged.

Return message mechanism

The Return message mechanism handles non-routable messages, and our producer sends the message to a queue by specifying an Exchange and a Routinkey, and we consumers listen to the queue for consumption!

In some cases, if we need to listen for unreachable messages when Exchange does not exist or the specified routing key cannot be found, we need to use a Return Listener!

A Mandatory value of true enables listeners to accept unreachable messages and process them. If set to false, the broker will delete the message automatically.

Custom listening on the consumer side

Traffic limiting on the consumption end

Suppose we have a scenario where we have a rabbitMQ server with tens of thousands of unconsumed messages, and then we open up any consumer client and there will be a huge amount of messages sent in a flash, but our consumer can’t handle it all at once.

This will cause your service to crash. There are other situations where there is a mismatch between your producer and consumer capabilities, where the producer side generates a lot of messages and the consumer side cannot consume as many messages in a high concurrency situation.

  • RabbitMQ provides a quality of service (qos) function that does not automatically acknowledge a certain number of messages (set to a consumer or Channel qos).

Void basicQOS(Unit prefetchSize, USHORT prefetchCount,Boolean Global).

  • PrefetchSize :0 Size limit for a single message. Zero means no limit, usually no limit.

  • PrefetchCount: Set a fixed value that tells rabbitMQ not to push more than N messages to a consumer at the same time. If N messages have not been ack, the consumer will block until it ack

  • Global: Whether truefalse applies the above Settings to channels, i.e., whether the above restrictions apply to channels or consumer levels.

Consumer ACK and requeue

  • Consumer end for consumption, if due to abnormal business we can log the record, and then compensation! (Maximum number of attempts can also be added)

  • If the server is down and other serious problems, then we need to manually ack to ensure that the consumer side of the consumption success!

Message back to queue

  • Requeuing is to repost messages to the broker if they have not been processed successfully!

  • Requeueing is usually not enabled in practice.

TTL queue/message

TTL Time to Live Indicates the TTL.

  • Support message expiration time, which can be specified when the message is sent.

  • Support queue expiration time. When a message is queued, it will be cleared automatically if the timeout period of the queue is exceeded.

Dead-letter queue

Dead Letter queue: DLX, dead-letter-exchange

With DLX, when a message becomes a dead message in a queue (that is, without any consumer consumption), it can be republished to another Exchange, the DLX.

There are several ways in which a message can become dead letter:

  1. Message rejected (basic.reject/basic.nack) with requeue=false (not requeued)

  2. TTL overdue

  3. The queue length reaches the maximum. Procedure

DLX is also a normal Exchange, no different from normal Exchanges, it can be specified on any queue, actually set the attributes of a queue. When the queue is dead, RabbitMQ will automatically re-publish the message to the Exchange and route it to another queue. The ability to listen for messages in this queue for processing compensates for the immediate parameter rabbitMQ previously supported.

Dead letter queue setup

  • Set up Exchange and Queue, then bind

Exchange: dlx. Exchange (custom name)

Queue: dlx.queue (custom name)

Routingkey: # (# indicates that any dead letter in a Routingkey will be routed)

We then declare the switch, queue, and bind normally, but we add a parameter to the queue:

arguments.put(“x-dead-letter-exchange”,”dlx.exchange”);

RabbitMQ Cluster mode

  1. Active/standby mode: Implements rabbitMQ high availability clusters. This mode is easy to use when the number of concurrent requests and data is small. Also known as the Warren model. (In this mode, the active node provides write operations while the secondary node provides read operations. In active/standby mode, the secondary node does not provide read and write operations but only performs backup.) If the active node fails, the standby node automatically takes over from the active node to provide services.

  2. Cluster mode: The classic mode is Mirror mode, to ensure 100% data loss, is relatively simple to implement.

  • Mirrored queues, a highly available solution for rabbitMQ data, are typically two to three nodes (three for 100% message reliability solutions) for data synchronization

The shovel mode is the main mode for remote replication. The shovel mode is usually used for remote replication. It relies on federation of rabbitMQ to produce consistently reliable AMQP data.

The rabbitMQ deployment architecture uses dual-center mode (multi-center). A rabbitMQ cluster is deployed in two or more data centers. The rabbitMQ service of each center must provide normal message services and share messages in some queues.

The multi-live architecture is as follows:

The Federation plugin is a high-performance plugin that transmits messages between Brokers or clusters without the need to build clusters. The two parties can use different users or virtual hosts or use different versions of Erlang or rabbitMQ. The Federation plugin can use the AMQP protocol as a communication protocol and can accept discrete transmissions.

A Federation of Exchanges can be seen as Downstream pulling messages from Upstream, but not all of them. Messages must be pulled from Upstream to Downstream only if the Bindings relationship has been explicitly defined on the Exchange, i.e. there is an actual physical Queue to receive messages.

Using the AMQP protocol for inter-agent communication, Downstream will group binding relationships together and bind/unbind commands will be sent to Upstream switches.

Therefore, Federation Exchange only accepts messages with subscriptions.

HAProxy is a free, fast and reliable solution that provides high availability, load balancing and proxy software based on TCP (Layer 4) and HTTP (Layer 7) applications. It supports virtual hosting. HAProxy is especially useful for heavily loaded Web sites that require call hold or seven-tier processing. HAProxy runs on current hardware and can support tens of thousands of concurrent connections. And it operates in a way that makes it easy and secure to integrate into your current architecture while protecting your Web servers from being exposed to the network.

Why does HAProxy perform so well?

  1. The single-process, event-driven model has been significantly reduced. Context switching overhead and memory footprint.

  2. Wherever available, single buffering can complete read and write operations without copying any data, saving significant CPU clock cycles and memory bandwidth

  3. With the splice() system call on Linux 2.6 (>= 2.6.27.19)., HAProxy can implement zero-copy forwarding and zero-starting on Linux 3.5 and above.

  4. Memory allocators allow instant memory allocation in a fixed-size memory pool, which can significantly reduce the time it takes to create a session

  5. Tree storage: focuses on the use of elastic binary trees developed by the author many years ago to achieve O(log(N)) low overhead to hold timer commands, hold run queue commands, and manage polling and minimum connection queues

keepAlive

KeepAlived software mainly realizes high availability function through VRRP protocol. VRRP, short for Virtual Router RedundancyProtocol(Virtual Router RedundancyProtocol), is designed to solve the single point of failure (singleton failure) problem of static routes. It ensures uninterrupted network operation when individual nodes fail. Keepalived – aspect has the function of configuration management LVS, and also has the function of health check on the nodes below LVS, on the other hand, can also realize the high availability of system network services

The role of the keepAlive

  1. Manage LVS load balancing software

  2. Implementing LVS cluster node health check

  3. Failover as a system network service

How does Keepalived achieve high availability

Keepalived Failover between high availability service pairs is achieved through Virtual Router Redundancy Protocol (VRRP). A complete PDF version of the Java Interview handbook has been organized into a document

While Keepalived is working correctly, the Master node sends heartbeat messages to the standby node continuously (in multicast mode) to tell the standby node that it is still alive. When the Master node fails, it cannot send heartbeat messages. Therefore, the standby node cannot detect the heartbeat of the Master node and invokes its own takeover program to take over the IP resources and services of the Master node.

When the active Master node recovers, the standby Backup node releases the IP resources and services that the active Master node takes over and restores to the original standby role.