A message model

The evolution of message queues

Message queue model

Full Java learning Materials collection link: wpa.qq.com/msgrd?v=3&u…

Early message queues were designed as a “queue” data structure.

Producers generate messages and join the queue. Consumers receive messages, that is, they exit the queue. The message container on the server is called message queue.

Of course, there may be more than one consumer, the existence of multiple consumers is a competitive relationship, the message is consumed by one of the consumers, the other consumers can not get the message.

Publish and subscribe model

If a person’s message is intended to be consumed by multiple consumers at the same time, the above queue pattern is not applicable, leading to a new pattern, the publish-subscribe model.

In the publish-subscribe model, the sender of the message is called Publisher, the receiver of the message is called Subscriber, and the container where the message is stored on the server is called Topic.

The publisher sends a message to the topic, and subscribers need to subscribe to the topic first. Subscribers who subscribe to a topic can then receive messages sent by the sender.

Publish subscriptions are also compatible with the message queue model, if there is only one subscriber, it is the message queue model.

The message model for RabbitMQ

RabbitMQ uses the same message queue model, but it introduces an exchange concept.

An Exchange, also known as an exchange, is located between a producer and a queue. The data generated by the producer is directly sent to an Exchange, which then sends the message to the corresponding queue according to the configured policy.

RabbitMQ uses bindings to associate switches with queues, usually specifying a BindingKey.

When a producer sends a message, it specifies a RoutingKey, and when a RoutingKey is the same as a BindingKey, it will be sent to the corresponding queue.

Type of the switch

The common switches in RabbitMQ are FANout, Direct, Topic and headers

direct

Direct sends messages according to the RoutingKey and then to the queue corresponding to the BindingKey that matches the RoutingKey.

If the RoutingKey is log, both message queues receive messages. If the RoutingKey is debug, exchange only sends messages to message queue 1.

topic

In Direct, a RoutingKey and a BindingKey match exactly to send a message, and topic extends this by introducing fuzzy matching.

• RoutingKey and BindingKey use. To split strings. Each split string is a matching character;
• In BindingKey, * and # are used for fuzzy matching. * represents a word, and # represents any zero or more words.
• When # is used alone in BindingKey, all messages are received, consistent with the type fanout;

Chestnut:

1. The routing key is test. Rabbitmq message queues 1 and 2 receive messages.

2. No one can receive messages if the routing key is rabbitMQ.

3. Message queue 2 receives the message whose routing key is test.

4. The routing key is rr.info.ww message queue 2 receives the message.

5. No queue can receive messages if the routing key is INFO.

fanout

Messages received by the exchange are sent to all queues bound to the exchange.

headers

A HEADERS exchange does not rely on the matching rules of the routing key to route a message, but matches the HEADERS attribute in the content of the message sent. When sending a message to the exchange, RabbitMQ gets the person headers (also a key/value pair) of the message and compares whether the key/value pair matches exactly the pair specified by the queue/exchange binding. If so, the message will be routed to the queue. Otherwise, a switch that does not route to the headers type of queue would be poor in performance and impractical, and would almost never see it.

Kafka’s message model

A broker is introduced in Kafaka. The broker receives information from producers, sets offsets for messages, and saves them on disk. The broker serves consumers and responds to requests to read partitions by returning messages that have been committed to disk.

The broker also confirms messages to producers and consumers.

A producer sends a message to the broker and can choose to continue sending if it does not receive an acknowledgement from the broker.

Consumers in the same way, on the consumption side, consumers to buy the messages are received and completed its business logic (for example, to save the data in the database), will send the service side consumption successful validation, the broker only after receiving the consumer to confirm, to think that a message has been successfully consumption, otherwise it will give consumers to resend this message, Until the corresponding consumption confirmation is received.

If only one consumption instance is being processed at a time in a topic, and we want to keep the message ordered, the current message cannot be consumed until the next message is consumed. Then, the performance of consumption would be extremely low, at which point the concept of partitioning was introduced.

Topics can be divided into several partitions, each of which is a commit log. Messages are appended to the partition and then read in first-in, first-out order. Note that because a topic generally contains several partitions, the order of messages cannot be guaranteed across the topic, but within a single partition.

At the same time, the introduction of consumer group, consumer is a part of the consumer group, so that one or more consumers will read a branch, but the group will ensure that a partition can only be consumed by one consumer, through multiple consumers, so that the consumption performance is improved.

Each Consumer group has a complete message in the consumption topic, and consumption progress among different Consumer groups is not affected by each other. That is to say, a message once consumed by Consumer Group1 will also be consumed by Consumer Group2. However, a message in a group can only be consumed by one message in the same group.

The offset is used by the consumer to confirm the read data, and it is an accumulation of data, increasing by one each time a data is successfully consumed. Each message has a unique offset in a given partition. The consumer stores the message offset read by each partition on Zookeeper or Kafka, and its read state is not lost if the consumer shuts down or restarts.

The message model for RocketMQ

RocketMQ’s message model is similar to Kafaka’s. Replace Kafaka’s middle partition with a queue, which is RocketMQ’s message model.

But while the message model is similar, the implementation is quite different.