Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”

This article has participated in the “Digitalstar Project” and won a creative gift package to challenge the creative incentive money.

preface

First to throw A question everyone to think about: in A distributed system, how can we guarantee the order of multiple requests, such as A/B two systems, the system in A business processing orders, A to B, the system sends the request three times, to insert the order operation, and then modify the order status and finally increase the user points.

However, these three requests fell on different machines respectively, and the operation of inserting the order was delayed due to some accidents. The operation of modifying the order was executed first, but there was no order information at this time, which resulted in unexpected results.

In the face of this situation, how should we avoid it? This requires understanding of what Flower elder brother said today: in distributed services, how to ensure the order of requests.

Adding access Services

We can add an access service between system A and system B. For example, in the above scenario, A complete business contains three requests, and each request carries A unique ID as the identification to the access service. According to the ID, the access service distributes all the corresponding requests to A certain machine. All three requests can then be handled by the same service.

Increase the queue

Increase 【 】 after access, can let the same id request distribution to the same machine, as long as the system sends A sequence, the system B also will received order, but there will be A problem, is that if took the form of A multithreaded processing in system B receives the request, it still will be the order of consumption.

In fact, the idea to solve this problem is to increase a memory queue, will be the same ID of the request, all thrown into a memory queue, let these several requests queued consumption, so that you can ensure the order of multithreading.

Distributed locks ensure orderliness

In general, the above two steps basically meet our business requirements, but in fact, there are extreme cases that cause sequential problems, such as abnormal order distribution to system B during access service distribution. For some systems that require 100% sequential interfaces, we can do this by connecting distributed locks.

As shown in the figure above, now three requests have arrived in system B. Each request carries a unique ID and the sequential number ORD of the request. Before executing the service logic in each request, the ZooKeeper lock will be requested, and only the thread that has obtained the lock can execute it:

  • If the delete request obtains the lock first, it will go to the database to determine whether the new request and modification request has been completed before executing the business logic. If not, the lock will be released.
  • If ord=1, it can execute the request. Therefore, it will insert data into the database and release the lock.
  • If the delete request obtains the lock, it will still query the database and find that the modification request has not completed the operation, so it continues to release the current lock resource.
  • The order status is modified and the lock is released after the modification request obtains the lock and the record already exists in the database.
  • The delete request obtains the lock, completes the delete operation, and finally releases the lock.

defects

Through the above three steps, 100% consumption sequence can be achieved, but after this operation, the throughput of the system is severely reduced, especially the introduction of distributed locks, which becomes the bottleneck of the system concurrency.

In addition, we needed to ensure the availability of the middleware introduced, resulting in a series of solutions that increased the complexity of the system, which was a challenge for future maintenance. Therefore, the best solution is whether it can be solved in the business logic, such as merging several steps into one step or replacing them with other solutions to avoid the root.