RabbitMQ

Problem solved by message queuing

The invocation of message queues is usually asynchronous, and should only be considered if it is acceptable to invoke them asynchronously (the result of processing is not immediately known).

1. System decouple

Loose coupling between systems can separate primary and secondary processes.

For example, when users purchase goods on e-commerce websites, the main process is to generate orders and reduce inventory, while the secondary process is to issue coupons and increase user points, which do not need to be handled immediately. At this point, the secondary process can be handed over to the downstream service of the message queue for processing.

2. Flow peak cutting and valley filling

In the second kill scenario, the second kill request is temporarily stored in the message queue, and the service system responds “The second kill result is being processed…”. After that, the system continues to digest messages in the message queue to avoid system crash due to the arrival of traffic peak.

However, since messages are sent and processed asynchronously, users cannot receive the results immediately, which is a disadvantage that must be considered when using message queue technology.

3. Achieve final consistency

The message queue is used to call other microservices, and the retry mechanism of message queue is used to ensure that a series of service calls can run correctly.

Remember to consider idempotence to avoid situations where multiple calls produce different results.

4. Persistent timers

Using the dead letter queue and TTL, when the message expires, the message will be put into the dead letter queue, from the dead letter queue to obtain the message, to achieve the purpose of timer.

Use dead-letter queue as the timer to use there is a problem, is in the normal queue, only check whether the head of the first message expired, other messages do not check, if this leads to each message TTL has long have short, dead-letter queue will lose the function of timer, this problem can be used to solve delay queue plugin.

TTL: Time To Live Expiration Time

How to set: (1) Queue property, all messages in the Queue are the same (2) message itself

In case of a conflict, the smaller value prevails

2. Message queue selection

1. RabbitMQ

Advantages:

  1. Support for AMQP protocol (support for most programming languages)
  2. Message latency is in the microsecond level

Disadvantages:

  1. Message pile-up can lead to significant performance degradation (queue state mechanism causes indexes and message bodies to be put on disk when there are too many messages)
  2. Poor performance (single machine throughput per second 1W magnitude)
  3. Using Erlang language development, secondary development costs high

2. RocketMQ

Advantages:

  1. Java development, read the source code, secondary development is convenient
  2. Messages are delayed in milliseconds
  3. High performance and stability (single-machine throughput of 10W)

Disadvantages:

  1. Integration and compatibility with peripheral systems are not very good

3. Kafka

Advantages:

  1. High reliability and stability
  2. Good compatibility with peripheral systems (especially related to big data and streaming computing)
  3. Scalable, partition, copy, fault tolerance

Disadvantages:

  1. High latency (asynchronous, batch), not suitable for e-shopping scene

Type of message body

  1. Simple TextMessage
  2. Serializable Object (ObjectMessage)
  3. Attribute Collection (MapMessage)
  4. Byte streams (BytesMessage)
  5. Raw value stream (StreamMessage)
  6. No payload (Message)

4. AMQP concept

Message sending process

Publisher specifies the RoutingKey for the message and sends the message to the Exchange, which looks up any (0 to infinity) queues to which the RoutingKey should be routed and routes the message. The Consumer gets the message from the Queue and processes it.

role

  1. Publisher: Message sender. Send a message specifying a RoutingKey to the Exchange, and the Queue receives the corresponding message.
  2. Server: an instance of the MQ service. Also called a Broker.
  3. Virtual Host: indicates a Virtual Host. A Server can have multiple virtual hosts, which are used to isolate projects.
  4. Exchange: indicates a switch. Receives messages from the Producer and routes the messages to the corresponding Queue.
  5. Routing Key: indicates the Routing Key. Specifies the routing rules for messages. Routing Key, Bindings and Exchange need to cooperate with each other.
  6. Bindings: binding. Specifies the binding between exchanges and queues.
  7. Queue: message Queue. A container for storing messages.
  8. Message: the Message. Publisher wants to tell the Consumer the news.
  9. -Penny: Consumer. Responsible for processing the messages sent from Publisher.

The AMQP uses the TCP/IP stream protocol to transmit data, and TCP/IP has no way to define the data probe (the range of a complete data), so the size of the data probe is added to each data probe header.

Type of switch

  1. Fanout: broadcast, no need to consider RoutingKey and Bindings. Routes messages to all queues bound to the exchange.

    (Publish/subscribe: online consumers can receive messages, if the consumer is not online, can not receive messages)

  2. Direct: Routes messages to queues where the RoutingKey exactly matches the BindingKey.

    For a single queue, in queue mode (a message to the queue is received by only one consumer)

    However, if there are multiple different queues, all bound to the same BindingKey, then all queues will receive messages

  3. Topic: Extension of the Direct type, which can use “*”, “#” keywords for fuzzy matching. * “matches one word, “#” matches zero or more words.

    (Publish/subscribe: online consumers can receive messages, if the consumer is not online, can not receive messages)

    RoutingKey and BindingKey are denoted by a “.” split, EX: eroupe.user.news

6. Consumer model

  1. Push mode: equivalent to listener mode
  2. Pull pattern: Consumers actively get messages

Vii. Storage mechanism

Message type

  1. Persistent message

  • Messages are backed up directly on disk as they arrive in the queue

  1. Nonpersistent messages
  • When memory pressure is high, messages are dumped to disk. The MQ is lost after the restart

The message file

  1. The index

  • Record (1) message location (2) whether the message has been received by the consumer (3) whether the message has been ACK by the consumer. The file falls after.idx

  • When the message ontology is small (4096B by default), the message itself is stored directly in the index.

  1. The message body
  • Key-value pair type storage, all queues use the same block of storage space. File drops after.rdq

The message to delete

  • When deleting a message, the record is only deleted from the ETS table (Erlang Term Storage is responsible for recording the mapping of messages in the document), and the message in the document is not deleted

  • Only when the percentage of garbage data is greater than grabage_Fraction (default 0.5) will garbage collection be triggered to merge the files of the two message bodies

The queue status

  1. Alpha: Indexes and messages are stored in memory
  2. Beta: Indexes are stored in memory and messages are stored on disk
  3. Gama: Indexes, messages are all on disk

Eight, message reliability

Ensure that the sender’s message is delivered to the queue

  1. Exception handling

    Not 100% reliable

  2. The Channel transaction

    Expensive and not recommended

  3. Sender confirmation

    Channel Is set to Confirm mode, blocking synchronization. Batch and asynchronous callbacks can be used to improve efficiency.

Ensure that messages are successfully consumed by “consumers”

Set the ACK Mode for the Listener

After a consumer consumes a message, it needs to send an ACK to the Broker, otherwise it will resend the message until it expires.

  1. NONE mode:

    Consumers catch exceptions themselves and risk losing data.

  2. AUTO mode:

    If an exception is thrown, the message is put back in the Queue and resend.

  3. MANUAL mode:

    The consumer manually calls the Channel method to return an ACK.

9. Persistence

MQ is not lost after restart

  1. Exchange Persistent: Durable Parameter = true
  2. Queue Persistence: durable parameter = ture
  3. Message persistence: deliveryMode = 2

10. MQ server security

Blocking producer

Do not let producers push messages into MQ

  1. Free disk space
  2. Disk memory ratio
  3. The available memory

Blocking consumer

Do not continue pushing messages to consumers (for push mode, not pull mode, none-ack mode is not supported)

Channel Sets basicQoS. When the number of unacknowledged messages reaches the upper limit, messages are not sent.

Ensure message idempotency

RabbitMQ only supports “at most” and “at least once” messages for reliable transmission

You must ensure that the results of multiple runs of the message are the same as those of only one run

  1. Database unique index

    When the same order number is inserted into the database, an error is reported and the transaction is rolled back.

  2. Preliminary checks

    If there is no record in the database, the operation is completed using exclusive locks (to avoid concurrency problems).

  3. Message unique ID

    A unique ID is generated for each message, and its existence in the distributed cache is determined before consumption.

Interface idempotency

Upstream, when called, passes a GUID and returns the same result if it is found to have been processed.

Eleventh, improve performance

Improve downstream throughput

  1. Optimize service performance
  2. Adding consumer nodes
  3. Increase the number of concurrent consuming threads

Distributed architecture

For specific cluster construction, HAProxy is used as load balancing in cluster mode and mirroring queue mode is used as high availability

  1. The main standby mode

    • Only one node is working (the backup node cannot read or write) and a shared storage is required – not recommended

  2. Shovel mode

    • Use plug-ins to achieve cross-room replication

  3. Cluster pattern

    • All nodes store metadata information about (1) queues (2) switches (3) bindings (4) vhosts

    • Each node stores its own message fragment

  4. Mirror queue mode

    High availability supplement for cluster mode

    • When a node fails, it is automatically switched to another mirror node in the cluster

    • Multiple copies are used to ensure that messages are not lost

    • The Slave is used only to replace the Master