Introduction to message queues and common message queues

The actual application of message queue includes the following four scenarios:

  • Application coupling: Multiple applications process the same message through message queues to avoid the failure of the whole process caused by interface invocation failure.
  • Asynchronous processing: Multiple applications process the same message in the message queue and process messages concurrently between applications, which reduces the processing time compared with serial processing.
  • Peak clipping: widely used in second kill or buying activities, to avoid the application system hanging due to excessive flow;
  • Message driven system: the system is divided into message queue, message producer and message consumer. The producer is responsible for message generation, and the consumer (possibly multiple) is responsible for message processing.

Application of the coupling

Specific scenario: users use QQ album to upload a picture, face recognition system will carry out face recognition of the picture, the general practice is that the server received the picture, the picture uploading system immediately call face recognition system, call completion and then return success, as shown below:

This method has the following disadvantages:

  1. Face recognition system is adjusted failure, resulting in picture uploading failure;
  2. High delay, face recognition system processing is completed, and then returned to the client, even if the user does not need to know the results immediately;
  3. Image uploading system and face recognition system call each other, need to do coupling;

To use message queues:

After the client uploads an image, the image uploading system writes the image information, such as UIN and batch, to the message queue, and returns success. The face recognition system is timed from the message queue to take data to complete the new image recognition.

At this point, the picture uploading system does not need to care about whether the face recognition system processes these picture information, and when to process these picture information. In fact, because the user does not need to know the face recognition results immediately, the face recognition system can choose different scheduling strategies, according to the idle time, busy time, normal time, to process the picture information in the queue.

Asynchronous processing

Scenario: To register an application, the system sends a registration email and an authentication SMS message. There are two ways to handle these two operations: serial and parallel.

  • Serial mode: after the new registration information is generated, the registration email is sent first, and then the verification SMS is sent.

    In this mode, the authentication SMS needs to be sent and then sent back to the client.

  • Parallel processing: after the new registration information is written, it is processed by sending SMS and email in parallel.

    In this mode, sending SMS messages and emails is returned to the client after processing.

Assuming that the processing time of the above three subsystems is 50ms and the network delay is not taken into account, the total processing time is:

Serial: 50+50+50=150ms Parallel: 50+50= 100ms

  • If message queues are used

Then the total response time depends on the time of writing to the message queue. The time of writing to the message queue itself can be very fast and can be ignored basically. Therefore, the total processing time is increased by 2 times compared with serial and double compared with parallel

Current limiting peak clipping

Specific scenario: When shopping websites carry out seckilling activities, due to the instantaneous traffic is too large and the server receives too much traffic, the traffic will surge, and the relevant system cannot handle the request or even crash. After joining the message queue, the system can fetch data from the message queue, which is equivalent to a buffer in the message queue.

This method has the following advantages:

  1. The incoming message queue, instead of being directly processed by the business processing system, makes a buffer, which greatly reduces the pressure of the business processing system.
  2. The queue length can be limited. In fact, the last user to queue cannot kill the item in seconds. These requests can be discarded directly, returning the information that the activity is over or the item is sold out.

Message-driven systems

Specific scenario: the user uploaded a batch of photos, face recognition system needs to cluster all the photos of the user, after clustering is completed by the reconciliation system to regenerate the user’s face index (speed up the query). The three subsystems are connected by message queues. The processing results of the former stage are put into queues, and the latter stage gets messages from the queues to continue processing.

This method has the following advantages:

  1. Avoid calling the next system directly and causing the current system to fail;
  2. Each subsystem can be more flexible in the way of message processing. It can choose to process the message when it is received, choose to process the message regularly, or divide the time period to process the message according to different processing speed.

If you feel that you have gained something after reading it, please click “like”, “follow” and add the official account “Niumi Technology” to read more wonderful history!! :