In this paper, the core application scenarios such as asynchronous, decoupled, peak cutting and valley filling, as well as the comparison of common protocols and push-pull modes of messaging middleware are discussed to solve this problem.
What is messaging middleware
As a typical Message Broker component, Message middleware is commonly used in enterprise application systems. It is mainly used for Message communication between distributed systems or components and provides Message communication services with reliable, asynchronous and transactional features. The application of message broker component can reduce the coupling degree between systems and improve the throughput, scalability and high availability of the system.
Distributed Message service mainly involves five core roles, Publisher, MsgBroker, Subscriber, Message Type and Binding, which are described as follows:
- A message publisher is an application that sends messages. An application can send one or more message types, and a publisher sends messages to MsgBroker.
- The trusted message component, known as MsgBroker, is responsible for receiving messages sent by publishers and delivering message distribution to one or more message subscribers based on message type and subscription. The whole process involves core functions such as message type verification, message persistence storage, subscription matching, message delivery and message recovery.
- A message subscriber is an application that subscribes to messages. An application can subscribe to one or more message types. A message subscriber receives messages from a Trusted Message broker.
- Message type: A message type is uniquely identified by TOPIC and EVENTCODE.
- A subscription relationship, which describes a message type to which a subscriber subscribes, is also called a Binding.
Core functional features
It can provide reliable message communication between different application systems, reduce the coupling degree between systems and improve the scalability and availability of the overall architecture.
It provides asynchronous message communication between different application systems, improving system throughput and performance.
The publisher system, message broker component, and subscriber system all support horizontal cluster scaling and dynamic deployment of compute nodes based on the volume of service messages.
Support for transactional messages to ensure consistency between messages and local database transactions.
Remote calls to RPC differ from message MQ
When it comes to message queues, it is necessary to take a look at the essential differences between RPC and MQ. From the definition and positioning of the two, Remote Procedure Call (RPC) mainly solves the problems between Remote communication and does not need to understand the communication mechanism of the underlying network. Message queue (MQ) is a communication model that implements one-way producer-to-consumer communication. The core difference is that RPC is two-way direct network communication, while MQ is one-way network communication into the intermediate carrier. A queue is a special kind of linear table, special in that it only allows deleting at the front end of the table and inserting at the rear end of the table. Like a stack, a queue is a linear table with limited operations. The end that inserts is called the end of the queue, and the end that deletes is called the head of the queue. Adding the qualifier “message” in front of the queue implies that the overall architectural implementation is message-driven. RPC and MQ are essentially two different implementation mechanisms for network communication. The significant differences between RPC synchronous waiting results and MQ in asynchronous, decoupled, peak cutting and valley filling are mainly as follows:
- Architecturally, RPC differs from MQ in that Message has an intermediate node, Message Queue, which stores messages.
- Synchronous invocation: RPC is preferred for scenarios where you wait for the result of processing to be returned immediately.
- MQ is used for performance reasons, such as servers that cannot respond quickly to clients (or clients that do not require real-time responses) and need to be cached in queues. On the other hand, it is more focused on data transmission, so it is more diversified, in addition to point-to-point, subscription and publishing functions.
- As the service grows, MQ can be used when the processing end calls too many downstream services or the processing volume becomes a bottleneck and the synchronous calls are changed to asynchronous calls.
Core Application Scenarios
For the core scenarios of MQ, we analyze the characteristics of asynchronous, decoupled, peak cutting and valley filling, which are different from traditional RPC calls. Especially in the case of the introduction of intermediate nodes, more possibilities and capabilities are increased through the idea of space (with storage capacity) for time (RPC synchronous waiting for response).
Asynchronous communication
For operations that do not require immediate processing, especially those that are time-consuming, asynchronous processing is provided through message queues, with additional consuming threads taking over for asynchronous processing.
The decoupling
Between applications, it provides a message communication mechanism between heterogeneous systems, and uses message middleware to solve the one-way communication mechanism between multiple systems or heterogeneous systems except RPC.
scalability
Because message queues decouple the processing of the main flow, you only need to add additional processing without changing codes or adjusting parameters, facilitating distributed expansion.
Distributed transaction consistency
When data state synchronization between two application systems needs to consider the final consistency of data state, transaction messages provided by message queue are used to achieve data state consistency between systems.
Peak peel
In the case of a surge in traffic, the application still needs to continue to play a role, but such sudden traffic cannot be predicted in advance. Preparing application resources in advance to handle such instantaneous peak access is a huge waste of resources. The pulse-proof capability of using message queues in the event of an emergency provides a guarantee to take over large pulse requests from the foreground and then consume them asynchronously and slowly.
recoverability
The failure of a component does not affect the entire system. Message queuing reduces coupling between applications, so that even if an application that processes messages fails, messages that are queued can still be processed after the system recovers.
In order to ensure
In most usage scenarios, the order in which data is processed is important. Most message queues are inherently sorted and ensure that the data will be processed in a particular order.
A large number of accumulation
The message accumulation capability is used to handle data migration scenarios. Enable incremental message accumulation when full migration of old data is performed. Enable incremental message accumulation after full migration is complete to ensure data consistency and not loss.
Data flow processing
Massive data flows generated by distributed systems, such as service logs, monitoring data, and user behaviors, are collected and summarized in real time or in batches, and then imported to the big data real-time computing engine. In this way, heterogeneous systems can connect to each other using message queues.
Industry messaging middleware comparison
Common protocols of message-oriented middleware
Closer agreement
AMQP is an Advanced Message Queuing Protocol (AMQP), an open application-layer Protocol designed for message-oriented middleware, which provides unified messaging services. The client and message-oriented middleware based on this protocol can transmit messages, regardless of different products of client/middleware and different development languages.
Advantages: Reliable and versatile
The MQTT protocol
MQTT (Message Queuing Telemetry Transport) is an instant messaging protocol developed by IBM and has become an important part of the Internet of Things. The protocol supports all platforms, connects almost any connected object to the outside world, and is used as a communication protocol for sensors and actuators (such as connecting homes via Twitter).
Advantages: simple format, small bandwidth, mobile communication, PUSH, embedded system
STOMP deals
Streaming Text Orientated Message Protocol (STOMP) is a simple Text Protocol designed for Message Oriented Middleware (MOM). STOMP provides an interoperable connection format that allows clients to interact with any STOMP message Broker.
Advantages: Command mode (not Topic/Queue mode)
XMPP agreement
XMPP (Extensible Messaging and Presence Protocol) is based on the Extensible Markup Language (XML) and is mainly used for INSTANT Messaging (IM) and online field detection. It is suitable for quasi-real-time operation between servers. The core is xmL-based streaming, a protocol that could eventually allow Internet users to send instant messages to anyone else on the Internet, even if their operating systems and browsers are different.
Advantages: General public, strong compatibility, extensible, high security, but XML encoding format takes up a large bandwidth
TCP/ IP-based custom protocol
Some special frameworks (such as Redis, Kafka, rocketMQ, etc.) do not strictly follow THE MQ specifications according to their own needs. Instead, they encapsulate a set of binary codec protocols based on TCP/IP and transmit them through network socket interfaces to achieve the relevant functions of MQ standard specifications.
Comparison of message-oriented push and pull patterns
Push mode: the server is not only responsible for message storage and request processing, but also needs to save Push state, save subscription relationship and consumer load balancing. The real-time performance of push mode is better; If the push capacity is greater than the consumption capacity, it can lead to consumer crashes or massive message loss
The main advantages of Push mode are:
- Low requirements for users, convenient for users to obtain the required information
- Timeliness is good, the server pushes more dynamic information to the client in real time
The main disadvantages of Push mode are:
- The information pushed may not meet the personalized needs of the client
- If the Push message is larger than the consumption rate of consumers, a coordinated QoS mechanism is needed to achieve consumer feedback
Pull mode: In addition to consuming messages, the client also saves message offset, temporary message storage and recover in abnormal cases. Unable to obtain messages in a timely manner, large amounts of data may cause broker message accumulation.
The main advantages of the Pull mode are:
- Strong pertinence, can meet the personalized needs of clients
- The client obtains on demand, and the server only passively receives the query and responds to the query request of the client
The main disadvantages of the Pull mode are:
- The real-time information is poor. It is difficult for the client to obtain the real-time information updated on the server
- Client users have high requirements and need to maintain sites
The relevant data
Suggest to learn the following technical documentation for more detailed technical details and implementation principle of deepening understanding of the message middleware and application of open source code can be downloaded at the same time, local debugging the corresponding code, deepen the understanding and grasp of the concept of the principle of the technology, and in the actual production more master different message queue application scenarios, Use message-oriented middleware efficiently and correctly.
RocketMQ Information: github.com/apache/rock…
Kafka data: kafka.apache.org/documentati…
Alibaba Cloud RocketMQ documentation: help.aliyun.com/document_de…