Message queue learning basics

What is MOM

MOM is message-oriented Middleware, a type of software designed for asynchronous, loosely coupled, reliable, scalable, and secure communication in distributed applications or systems. The overall idea of a MOM is that it acts as a message mediator between a message sender and a message receiver, providing a whole new level of loose coupling.

The idea of MOM is that applications A and B do not send messages directly. Before, A and B directly sent messages, there were many efficiency problems. For example, B did not accept the messages in time after A sent them, so A would be blocked there all the time. The concurrency is not good. MOM is designed to solve this problem by adding A message-oriented middleware between A and B instead of letting A and B interact. A puts the message in the middle of the message and can go off to do other things. When B comes to the message-oriented middleware, A doesn’t need to know or care. So it’s more efficient and it provides concurrency, so when B goes, it can notify A through status, notifications, callbacks, etc. There are many technologies in the market to realize this idea, such as IBM (MQSEVICES), Microsoft(MSMQ) and BEA’s MessageMQ. In the stage of contention of a hundred schools of thought, they all achieve their own, there is no unified implementation standards. At this time, SUN to achieve a unified standard appeared JMS unified implementation specification. JMS has two main messaging models, point-to-point and publish-subscribe.

What is a message queue

A message queue is a container that holds messages during transmission, receives messages and stores them as files. A message queue can be consumed by one or more consumers.

Message queue middleware is an important component in distributed system, mainly solving application coupling, asynchronous message, traffic cutting and other problems. Achieve high-performance, highly available, scalable, and ultimately consistent architectures. It is an indispensable middleware for large distributed system.

Message queue advantages

1. Transfer data from one application to another, or from one module of the software to another
2. Responsible for the establishment of network communication channels, reliable data transmission
3. Ensure that data is not retransmitted or lost
4. It can realize cross-platform operation and integrate mechanic data transfer service for software on different operating systems

Application scenarios of message queues

The following describes the common usage scenarios of message queues in practical applications. The scenarios are divided into asynchronous processing, application decoupling, traffic cutting and message communication.

Asynchronous processing

Scenario After a user is registered, the user needs to send a registration email or registration information in two traditional ways: serial mode and parallel mode

Serial way

After the registration information is successfully written into the database, the system sends a registration email and then a registration SMS message. After all tasks are completed, the system returns the registration information to the client

parallel

After the registration information is successfully written to the database, you can send the registration email and SMS at the same time. After all tasks are completed, information is returned to the client. Compared with serial mode, parallel mode can improve execution efficiency and reduce execution time.

According to the above process, the user response time of the basic equivalent to time of writing data to database, send registered mail, registered to send SMS messages after writing a message queue, can return to the results, write the message queue time soon, almost can be ignored, also has the system throughput can be up to 20 QPS, ascension nearly three times than the serial way, 2 times better than the parallel mode.

The application of decoupling

Scenario After a user places an order, the order system notifies the inventory system.

The traditional way is: the order system calls the inventory system interface. As shown below:

1. Assuming that the inventory system access fails, the order reduction inventory fails, resulting in the order creation failure
2. The order system is over-coupled to the inventory system

How to solve the above shortcomings? Message queues need to be introduced, and the architecture after message queues are introduced is as follows:

Message queues are introduced to decouple applications

• Order system: after the user places an order, the order system carries on the data persistence processing, and then writes the message to the message queue, and returns the order creation success
• Inventory system: use pull/push to obtain order information, inventory system according to the order information, inventory operation.

If the inventory system is not working properly when placing an order. It also does not affect the normal order, because after the order is written to the message queue, the order system does not care about its subsequent operations. Thus, the order system and inventory system are decoupled.

Flow cutting front

Traffic peak-clipping is also a common scenario in message pairs and is commonly used in split-kill or group-grab activities.

Application Scenario The application is suspended due to heavy traffic. To solve this problem, messages are usually queued at the front of the application.

1. Can control the number of people participating in activities;
2. It can relieve the huge pressure of high flow on application in a short time;

The system figure of flow cutting mode is as follows:

Flow cutting mode system diagram

1. When the server receives a user request, it first writes to the message queue. At this time, if the number of messages in the message queue exceeds the maximum number, the user request is directly rejected or the user returns to the error page.
2. The second kill service reads request information in the message queue according to the second kill rule for subsequent processing.

Log processing

Log processing refers to the use of message queues in log processing, such as Kafka, to solve the problem of large log transfer.

Log processing refers to the use of message queues in log processing, such as Kafka, to solve the problem of large log transfer. The architecture is simplified as follows:

The architecture in which message queues are applied to log processing

• Log collection client: is responsible for log data collection, periodic write write Kafka queue;
• Kafka message queue: receives, stores, and forwards log data.
• Log processing applications: subscribe to and consume log data in the Kafka queue;

The application of this architecture in the actual development can refer to the case: Sina Technology Sharing: How do we carry 3.2 billion real-time log analysis and processing

Technical architecture design of services

1. Kafka: Message queue that receives user logs.
2. Logstash: Logs are parsed and output to Elasticsearch as JSON.
3. Elasticsearch is a schemaless, real-time data storage service that organizes data through index and provides powerful search and statistics capabilities.
4. Kibana: ELK Stack is a data visualization component based on Elasticsearch.

Message communication

Message communication means that message queues generally have efficient communication mechanisms built in, so they can also be used for pure message communication. Such as implementing point-to-point message queues, chat rooms, etc.

Point-to-point communication

Point-to-point communication architecture design

In the design of point-to-point communication architecture, client A and client B share A message queue to realize message communication.

Chat room communication

Chat room communication architecture design

Client A, client B, and client N subscribe to the same message queue to publish and receive messages, so as to realize the design of chat communication scheme architecture.

JMS message service

Talk about message queues without mentioning JMS. JMS(Java Message Service) JMS is called Java Message Service, which is a technical specification for MOMS on the Java platform. Designed to simplify enterprise application development by providing standard apis for generating, sending, receiving, and processing messages, similar to the abstractions of JDBC and relational database communication.

API is a messaging service standard/specification that allows application components to create, send, receive, and read messages based on the JavaEE platform. It is distributed communication that is less coupled, message service more reliable and asynchronous.

In the EJB architecture, message beans integrate seamlessly with JM message services. In the J2EE architectural pattern, there is the message server pattern, which is used to decouple messages directly from applications.

Commonly used concept

• Provider: JMS interface implementation written in pure Java (such as ActiveMQ)
• Domains: message passing methods, including P2P and Pub/Sub
• Connection Factory: The client uses the Connection factory to create a Connection to the JMS Provider
• Destination: The object to which the message is addressed, sent, and received

A message model

In the JMS standard, there are two message models: P2P (Point to Point), Publish/Subscribe (Pub/Sub)

P2P model

P2P mode consists of three roles: Queue, Sender and Receiver. Each message is sent to a specific queue from which the receiver retrieves the message. Queues hold messages until they are consumed or timed out.

P2P message domains use queues as destinations. Messages can be sent and received synchronously or asynchronously, and each message is sent to a Consumer only once. Consumer can use a MessageConsumer. The receive () synchronously receive messages, but can be by using MessageConsumer. SetMessageListener () to register a MessageListener realize asynchronous receive.

Multiple consumers can be registered to the same queue, but a message can only be received by one Consumer, who then confirms the message. And in this case, the Provider polls all registered consumers.

The characteristics of P2P

1. There is only one Consumer per message (that is, once consumed, the message is no longer in the message queue and no other Consumer can get it).
2. Senders and receivers are time-independent, meaning that when a sender sends a message, it does not affect whether the receiver is running or not.

3. The consumer must acknowledge receipt of the message

   Upon receipt of the message, the consumer must confirm that the message was received, otherwise the JMS service provider will assume that the message was not received, and the message can still be received by others. Verification can be performed automatically without human intervention.

4. Non-persistent messages are sent at most once

   A non-persistent message can be sent at most once, indicating that the message may not be sent. The possible causes are as follows:

   1. The JMS service provider is down, causing the loss of non-persistent information

   2. The message in the queue expired and was not received

5. Persistent messages are sent strictly once

   We can set the more important messages to be persistent, so that messages are not lost due to the failure of the JMS service provider or other reasons.

It contains three roles: Topic, Publisher and Subscriber. Multiple publishers send messages to a Topic, and the system delivers these messages to multiple subscribers.

Topic does not hold messages for subscribers unless explicitly specified. Of course, this can be achieved through Durable subscriptions. In this case, the Provider stores messages for the subscriber when it disconnects from the Provider. When a persistent subscriber reconnects, it will receive all messages that were not consumed during the disconnection period.

The characteristics of the Pub/Sub

• Each message can have multiple (0,1…) Subscribers. Each message can have multiple consumers. If newspapers and magazines are the same, anyone who subscribes can get it.

• There is a temporal dependency between publisher and subscriber. A subscriber can only consume messages published after they subscribe, and a subscriber for a Topic must create a subscriber before consuming the publisher’s messages. This requires subscribers to subscribe before producers can publish. That is, the subscriber must run and then wait for the producer to run, which is different from the point-to-point type.
• In order to consume messages, the subscriber must remain running. That is, a subscriber must remain active and wait for a message published by the publisher. If the subscriber runs after the publisher publishes the message, it cannot get the message published by the previous publisher.

To mitigate such strict time dependencies, JMS allows subscribers to create a persistent subscription. This way, the subscriber can receive the publisher’s message even if it is not activated (running). The Pub/Sub model can be used if you want to send messages that can be processed without any processing, by a single message maker, or by multiple consumers.

News consumption

In JMS, messages are produced and consumed asynchronously. For consumption, JMS messagers can consume messages in two ways.

1. Synchronous subscribers or receivers receive messages through the receive method, which blocks until the message is received (or timed out).
2. Asynchronous subscribers or receivers can also register no message listeners. When the message arrives, the listener’s onMessage method is automatically called.

Connection represents the Connection (wrapper around TCP/IP sockets) established between the client and the JMS system. Connection can generate one or more sessions. Like ConnectionFactory, there are two types of Connection: QueueConnection and TopicConnection.

4.Session

Session is the interface for manipulating messages. Session can be used to create producers, consumers, messages, and so on. Session provides transaction functionality. When you need to use a session to send/receive multiple messages, you can put those send/receive actions into a single transaction. Also, QueueSession and TopicSession are divided.

5. Producers of messages

The message producer is created by the Session and used to send the message to the Destination. Again, there are two types of message producers: QueueSender and TopicPublisher. You can call the message producer’s methods (send or publish methods) to send messages.

JMS programming practice

Take ActiveMQ for example

public class JMSDemo {
        ConnectionFactory connectionFactory;
        Connection connection;
        Session session;
        Destination destination;
        MessageProducer producer;
        MessageConsumer consumer;
        Message message;
        boolean useTransaction = false;
        try {
                Context ctx = new InitialContext();
                connectionFactory = (ConnectionFactory) ctx.lookup("ConnectionFactoryName"); / / use ActiveMQ: connectionFactory = new ActiveMQConnectionFactory (user, password, getOptimizeBrokerUrl (broker)); connection = connectionFactory.createConnection(); connection.start(); session = connection.createSession(useTransaction, Session.AUTO_ACKNOWLEDGE); destination = session.createQueue("TEST.QUEUE"); // Producer sends messages. Producer = session.createProducer(destination); message = session.createTextMessage("this is a test"); // Consumer = session.createconsumer (destination); message = (TextMessage) consumer.receive(1000); System.out.println("Received message: "+ message); / / consumer asynchronous receive consumer. SetMessageListener (newMessageListener() {
                        @Override
                        public void onMessage(Message message) {
                                if(message ! = null) {doMessageEvent(message); }}}); } catch (JMSException e) { ... } finally { producer.close(); session.close(); connection.close(); }}Copy the code