JB test tour – learn about MQ

“What is MQ”

According to one section, MQ(Message Queue), called message queue, is a first-in, first-out data organization in the underlying data structure.

Generally used to solve application decoupling, asynchronous messaging, traffic pruning and other problems, to achieve high performance, high availability, scalable and ultimately consistent architecture.

MQ main products include: RabbitMQ, ActiveMQ, RocketMQ, ZeroMQ, Kafka

From the above, it is easy to see that MQ is a cross-process communication mechanism used to send messages upstream and downstream. Personally, MQ is a bit like an intermediary, where landlords post rental information, the information is placed in the intermediary, and tenants rent a house through the intermediary.

» Benefits of using MQ

Here’s a more mundane example:

The interviewer wants HR to hire the right person early, so it starts like this:

HR asks the interviewer when he is free, and sends the candidate’s materials to the interviewer, and sees the interviewer review them and make a conclusion before leaving. After a long time, everyone feels it is very troublesome. HR thinks the candidate is good, while the interviewer thinks the candidate is not suitable, which leads to disputes.

After that, HR tells the interviewer that I will put the information on the desk. Please remember to read it when you are free. Then every time the interviewer sees the information on the desk, he will pick it up and read it.

In this scenario, HR is the producer, the interviewer is the consumer, and the desk is MQ.

The benefits of using MQ are addressing application decoupling, asynchronous messaging, and traffic pruning:

• When HR wants to give information, they don’t need to know whether the interviewer is available or not, they just need to put the information on the table, so that everyone can have time to do other things and save everyone’s time. With decoupling applied, each member is independent and not affected by other members. The interviewer doesn’t care who put the information in, and the HR doesn’t care who sees the information
• If another group has a need (and it’s the same type of job, such as Java back-end development), the HR will still have it on the desk, and both interviewers will just have to pull it off. For asynchronous messaging, HR can just put the data on the desk and do something else, which is much more efficient than looking at it in person in the first place
• HR does not need to pay attention to when the interviewer looks at the data, nor to how long they look at the data, reducing conflicts. The frequency of data given by HR is not fixed, and the length of time for the interviewer to read the data is not fixed, so the interviewer only needs to read the conclusion within a fixed time, and there will not be so much pressure.

» Disadvantages of using MQ

Of course, there are many problems to be solved with MQ, such as data being lost innocently, the same data being given multiple copies, data being robbed, data being given to interviewer A only to be given to interviewer B, etc.

» When to use MQ

Message model

What is a JMS

Java messaging service refers to an API for asynchronous communication between two applications. It provides a common set of interfaces for standard messaging protocols and messaging services, including creating, sending, reading messages, and so on, to support Java application development.

Why JMS is needed

In JAVA, how do two applications send messages to each other if they don’t know much about each other, or even if they might be deployed in different places?

For example, an application A is deployed in India and another application is deployed in the United States, and then whenever A triggers something, B wants to get some update information from A.

Of course, it is also possible that more than one B may be interested in A’s updates, and there may be N applications similar to B that want to get the updates from A.

In this case, JAVA provides the best solution, JMS, that perfectly solves the problem discussed above.

Point-to-point model

In this model, there are the following concepts: message Queue, Sender, Receiver

Each message is sent to a specific queue from which the receiver retrieves the message. Queues hold messages until they are consumed or timed out.

• Multiple consumers are supported
• There is only one consumer per message (once a message is consumed, the message is no longer in the message queue)
• There is no time dependency between sender and receiver, which means that when a sender sends a message, it does not affect whether the receiver is running or not
• After receiving a message successfully, the receiver must reply to the queue successfully

If you want every message sent to be processed successfully, you need a point-to-point model.

The goddess wants to talk to backup A, so she talks to backup A. That’s point-to-point. Only one person gets the message

Publish and subscribe model

A message producer (publication) publishes a message to a topic, and multiple message consumers (subscription) consume the message. Unlike the peer-to-peer approach, messages posted to a topic are consumed by all subscribers.

In this model, there are the following concepts: Topic, Publisher, Subscriber

The client sends the message to the topic. Multiple publishers send messages to a Topic, and the system delivers these messages to multiple subscribers.

• Each message can have multiple consumers
• Publishers and subscribers are time-dependent; they can only receive messages when the client creates a subscription, and the subscriber must always be active to receive messages
• The subscriber creates a persistent subscription. This way, the subscriber can receive the publisher’s message even if it is not activated (running).

The Pub/Sub model can be used if you want to send messages that can be processed without any processing, either by one consumer, or by multiple consumers.

The goddess posted a circle of friends that all her back-up friends could see. It’s called publish/subscribe.

The difference between the two models

Under the point-to-point model, there is no repeat consumption.

Point to point, a queue can have multiple consumers, producers to send a message to the queue, consumers can use the message queue and consumption, once the message is consumption, the queue there will be no storage, and other consumer could not consumption to have been the news, if there has been no consumer processing, this message will be saved, until consumers are available.

Publish and subscribe model, can be repeated consumption.

Under publish subscription, only subscribers who subscribe to a topic will receive messages sent by the publisher. Note that all services subscribed to this topic can receive messages, so that the effect of message copy can be achieved

“MQ in the Workplace”

Although the above example illustrates the MQ application scenario using a recruitment example, you may still have some questions about how it works on the job, so let’s talk about the work scenario.

» asynchronous

I was responsible for a requirement called “old with new”, and the general process is as follows:

1) After placing an order, the user will first judge the identity of the user. 2) If it is a new user, then judge whether there is an inviter. 3) If there is an inviter, then judge the identity of the inviter

In this case, the user’s flow will change:

The problem with this, obviously, is that the added logic runs the risk of clogging the ordering main process.

Since synchronous processing can be a problem, let’s change it to asynchronous instead and change it to something like this:

The advantage of asynchrony is that it does not clog the ordering process even if the old logic has problems with the new logic.

A few days after such a good day, the problem came again: the old with new business frequent changes, resulting in frequent version of the ordering system, there are quality risks.

» use MQ

Since more and more dependent on ordering system business in order to guarantee the stability of the order system, business level must be decoupled, only need to pay the success news to other business, after they received the notice to handle, we just own process, follow-up and other business systems, direct subscription we send payment success message.

“Problems with MQ”

• How can message queues be highly available?
• How to ensure that messages are not re-consumed?
• How do I handle message loss?
• How do I ensure that messages are sequential?
• How to handle a large message backlog in message queues?

These problems are encountered in the actual work, and are often test points, which will also be mentioned below. A simple understanding can be.

“MQ Product Comparison”

In terms of choice, companies tend to use Kafka and RocketMQ.

“Test points for MQ”

» producers

• Is the generated data format consistent with the definition
• Whether the data was successfully pushed to the queue
• Whether the data has been successfully promoted to the corresponding topic
• How to handle push failure
• How to deal with the repeated push of the same data
• Push messages in different order, pay attention to queue priority
• Tweet time
• What can I do if the queue capacity reaches the upper limit and cannot be pushed

» consumers

• Whether the consumed messages come from the subscribed topic
• The message was consumed, whether it was cleared
• Producers push too fast, consumption speed is too slow (jam), what will happen
• Unable to consume unsubscribed topic messages
• After the producer pushes the message, the consumer receives the same message content as the producer pushes it
• How do you handle repeated messages, such as idempotent messages
• Deal with the timeout
• Message processing failure
• Whether the priority of the consumption message is consistent with that of the push
• Consumption message time
• What happens when customers are down, messages pile up and no one processes them
• Whether messages can be consumed normally

» queues

• Check whether messages are lost after the outage is recovered
• Outage plan, how long can be restored, if unable to restore the plan
• Whether different message formats can be recognized and forwarded normally

“Summary”

Come and go, spent a week time to finish this pile of information, the mq has been measured before, but didn’t know much about this thing, from the introduction, selection, test point, deepen the impression of mq, but never done mq performance testing with automated testing, so the temporary no result can be output, if subsequent have the similar experience, will share.

We are very curious about how to solve the problem of inconsistent messages. So we plan to write about distributed transactions in the next chapter. We would like to know more about the details