1. Message producer, message sender, and queue
  • Message Producer Producer: Sends messages to the message queue.
  • Message Consumer: Receives messages from message queues.
  • Broker: The concept comes from Apache ActiveMQ. It refers to the server side of MQ that helps you transfer messages from sender to receiver.
  • Message Queue: A first-in, first-out message storage area. Messages are sent and received sequentially, and once they are consumed and processed, they are removed from the queue.
2. The main consideration in designing the Broker

1) Message dump: it can be delivered at a more appropriate time point or assisted by a series of means to finally reach the consumer machine.

2) Specification of a paradigm and a common pattern to meet the needs of decoupling, final consistency, staggered peaks, etc.

3) In fact, it is simply a message forwarder, which makes one RPC into two RPCS. The sender delivers the message to the broker, which forwards the message first hand to the receiver.

This sums up to two RPCS plus one dump, or three RPCS if confirmation of consumption is to be done.

3. Point-to-point message queue model

The point-to-point model is used for point-to-point communication between message producers and message consumers.

The point-to-point mode contains three roles:

  • Message Queue
  • The Sender (the Sender)
  • The recipient (Receiver)

Each message is sent to a specific queue from which the receiver retrieves the message. Queues hold messages that can be stored in memory or persisted until they are consumed or timed out.

The characteristics of

There is only one Consumer per message (that is, once consumed, the message is no longer in the message queue). There is no time dependency between the sender and the receiver, who must reply to the queue after successfully receiving the message

4. Publish subscribe message model Topic

The publish subscribe model contains three roles:

  • Topic
  • Publisher
  • Subscriber

Multiple publishers send messages to a Topic, and the system delivers these messages to multiple subscribers.

The characteristics of

  • Each message can have multiple consumers: Unlike peer-to-peer, published messages can be consumed by all subscribers
  • There is a temporal dependency between publisher and subscriber.
  • For subscribers to a Topic, it must create a subscriber before it can consume the publisher’s messages.
  • In order to consume messages, the subscriber must remain running.
5. The difference between peer-to-peer and publish-subscribe

The producer sends a message to the queue, and only one consumer receives it.

When a publisher sends a message to a topic, only the subscribers who subscribe to the topic receive the message.

6. Guarantee the order of messages

Based on Queue message model, FIFO can ensure the order of messages.

7. ACK mechanism of messages

The Ackownledge confirmation mechanism for messages,

To ensure that messages are not lost, message queues provide a message Acknowledge mechanism, or ACK mechanism. When a Consumer acknowledges that a message has been consumed and sends an ACK to the message queue, the message queue can delete the ACK

The interest rates. If a Consumer is down/down and does not send an ACK, the message queue will assume that the message has not been processed and will re-send the message to another Consumer for re-processing.

8. Final consistent design ideas

Mainly by way of “record” and “compensation”.

The local transaction maintains business changes and notification messages, landing together, and then the RPC reaches the broker. After the broker has landed successfully, the RPC returns success and the local message can be deleted. Otherwise, local messages are constantly resent by scheduled task polling, which ensures a reliable landing broker for messages.

The broker sends a message to a consumer in a similar manner until the consumer sends a successful confirmation of the purchase.

Ignoring the problem of duplicate messages for a moment, the downstream is guaranteed to receive the message by landing the message twice plus compensation. Then rely on the state machine version number and other ways to do heavy, update their own services, to achieve the final consistency.

If the consumer is too slow to process the goods, it should be allowed to take the initiative to ack the error, and can agree with the broker on the next delivery time.

For messages sent from broker to consumer, it is necessary to record the IP address delivered because it is uncertain whether the loss occurred during business processing or in the case of message delivery. Ask for this IP before deciding to retransmit. Was the message processed successfully? If the inquiry fails, resend it.

Transactions: local transactions, local landing, compensation send. Local transactions do business landing and message landing transactions, not business landing and RPC success transactions. Once the message is successfully landed, there is largely no risk of loss.

9. Transaction support for messages

The sending and receiving processing of messages supports transactions. For example, in a task-centric scenario, a single processing may involve the receiving and processing of multiple messages, which should be within the scope of the same transaction. If a message fails to be processed, the transaction is rolled back and the message is returned to the queue.

10. Message persistence

Message persistence is very important for some key core services. After message persistence is enabled, messages can be recovered from persistent storage after message queues are down and restarted. Messages are not lost and can continue to be consumed and processed.

11. High availability of message queues

In a real production environment, the message queue service of a single instance is unavailable in the event of system problems such as downtime or restart. Therefore, in many scenarios, we expect message queues to have high availability support, for example

RabbitMQ high availability scheme in mirror cluster mode, ActiveMQ also has a high availability scheme based on LevelDB+ZooKeeper, and Kafka Replication mechanism.

12. Message queue selection and application scenarios

For details, see: High Concurrency Architecture series: Distributed message queuing features, models, and application scenarios

This article is published by OpenWrite!