What is MQ?

Message queuing (MQ) is an application-to-application communication method. Applications communicate by writing and retrieving application-specific data (messages) in and out of queues, without the need for dedicated connections to link them. Messaging refers to programs communicating with each other by sending data in messages rather than by direct calls, which are typically used for techniques such as remote procedure calls. Queuing refers to applications communicating through queues. The use of queues removes the need for receiving and sending applications to execute simultaneously.

Why use MQ?

Usage scenarios

  • The decoupling
  • asynchronous
  • Peak clipping

Scenario, for example,

The decoupling

  • Data-driven task dependencies

    What is task dependence? For example, Internet companies often conduct some data statistics tasks in the early hours of the morning, and there is a certain dependence between these tasks, such as:

    1) Task3 requires the output of Task2 as input

    2) Task2 needs to use the output of Task1 as input

    So, tast1, Task2, task3 have a task-dependent relationship, and task1 has to do it, task2 has to do it, task3 has to do it.

    • Do not use the MQ

      A common way to implement this requirement is to manually schedule the execution using Cron:

      1) Task1, 0:00, experience execution time is 50 minutes

      2) Task2, 1:00 (10 minutes buffer is reserved for task1), and the experience execution time is 50 minutes

      3) Task3, 2:00 p.m.

    The disadvantages of this approach are:

    1) If the execution time of one task exceeds the reserved buffer time, an error result will be obtained, because the post-task is not clear whether the pre-task is successfully executed. In this case, the task needs to be manually rerouted and the schedule may need to be adjusted

    2) As the execution time of the total task becomes longer, a lot of buffer should always be reserved. If the pre-task is completed in advance, the post-task will not start in advance

    3) If a task is dependent on multiple tasks, the task will be called critical path, and it is difficult to reflect the dependency relationship in the schedule table, which is prone to errors

    4) If the execution time of one task is to be adjusted, the execution time of multiple tasks will be adjusted

    • Using the MQ

      The optimization scheme is to use MQ decoupling:

      1) Task1 starts on time and sends a “task1 done” message when it finishes

      2) Task2 subscribs to the “task1 done” message, executes the message immediately after receiving it, and sends a “Task2 done” message after receiving it

      3) Task3

    The advantages of using MQ are:

    1) There is no need to reserve buffer. After the upstream task is executed, the downstream task will always be executed at the first time

    2) Depend on multiple tasks, be dependent on multiple tasks are very easy to handle, just need to subscribe to the relevant message

    3) The execution time of downstream tasks does not need to be adjusted due to the change of task execution time

    MQ is only used to deliver messages that upstream tasks have completed, not real input and output data.

asynchronous

  • Upstream does not care about execution results

  • Upstream focuses on execution results, but execution takes a long time

    • Do not use the MQ

      Upstream calls downstream directly

      Disadvantages of direct calls:

      1) The upstream needs to wait for the downstream execution result synchronously

      2) The downstream system is faulty and the upstream system cannot be used

      3) Upstream code needs to be modified for downstream addition

    • Using the MQ

      Use MQ to achieve asynchronous, decoupled effects

      Advantages of using MQ:

      1) The upstream does not need to wait for the completion of the downstream execution to speed up the upstream response

      2) Failure of the downstream system does not affect the operation of the upstream system

      3) Add downstream simply by subscribing to MQ

Peak clipping

  • Request peak

    For example, system A receives more than 100 concurrent requests per second during most of the day. However, the number of concurrent requests per second spikes to more than 10,000 between 12:00 and 1:00 p.m. However, the maximum number of requests that the system can handle per second is only more than 1,000.

    • Do not use the MQ

      At this time, if the system hard resistance will cause the system to hang

    • Using the MQ

      By using MQ to limit traffic, requests that cannot be processed by the system pile up in MQ and the system can continue to consume requests in MQ after peak times.

      Advantages of using MQ:

      1) The system will not hang due to peak requests

Problems with the introduction of MQ

Introducing a new technology will inevitably bring new problems while solving existing ones.

  • Reduced availability

    The more external dependencies a system introduces, the more likely it is to fail, and the failure of MQ causes the entire system to become unavailable.

  • Complexity enhancement

    Repeated consumption, message loss, message sequencing, and so on are all things that need to be considered when MQ is introduced.

  • Consistency problem

    A system processing directly return success, people think you this request is successful; But the problem is, if BCD three systems, BD two system write library success, the result of C system write library failure, will lead to data inconsistency.