In an office in Shenzhen, a handsome programmer is hard at work moving bricks. The boss rushed to the front and said to me: “Recently we received a large number of user feedback said that after finishing the order did not timely delivery, there have been a lot of user complaints, you hurry to deal with it.” I quickly nodded good. Then I started to check the order data and found that the remaining quantity of some goods in the inventory system was actually negative, so it was impossible to ship (nonsense, how can we ship when there is no inventory?). “, so I put the code up and down left left right several times, did not find any problems, and then unconvinced in the test environment tried several times, or did not find… At this time my heart is probably like this

Just when I had a splitting headache and doubted my life, I met the operation and maintenance elder brother, who told me that the deployment scheme of the system had changed recently, from the previous single node deployment scheme to the distributed cluster deployment scheme. Oh oh oh! Oh, I see. I think I know the problem.

Single-node deployment solution

  • The user initiates an order request
  • Check whether stock is sufficient
  • If the inventory is sufficient, the order is generated and the inventory is deducted. (The issue of distributed transactions is not discussed in this article.)
  • Response user order success or failure

Single-node deployment is normal when the concurrency is small and the response time of the whole process is optimistic, but if either the order system or the inventory system goes down, the whole business process will be interrupted. (The coupling degree is too high, and there is a single point of failure). Therefore, we decided to switch to a distributed cluster deployment solution to solve the single point of failure and improve system availability.

The issue of system coupling can be handled by middleware and is not discussed in this article. It should be noted that synchronized is used to control concurrent operations in the inventory system, otherwise this solution will still have thread safety issues

Distributed cluster deployment solution

  1. He and Huang place orders to the order system at the same time. Due to the load balancing strategy of the order system, he’s request is handed over to order system 1 for processing, and Huang’s request is handed over to order system 2 for processing
  2. Order system 1 inquired inventory system 1 and found that the inventory was sufficient
  3. Order system 2 checks inventory system 2 and finds sufficient inventory
  4. Both order system 1 and order system 2 find that the inventory is sufficient, so they inform inventory system 1 and inventory system 2 to reduce the inventory, which leads to the negative deduction of the quantity of goods in the inventory system, which is also called inventory oversold phenomenon.

The root cause of this problem is that synchronized locks cannot be shared between nodes in a distributed cluster. Now that the problem has been analyzed, the solution to the problem is in sight. We can use a distributed lock that can be shared across applications to lock the inventory reduction process so that no other ordering system can perform the same operation while the order system is doing the inventory reduction. This ensures thread-safety in a distributed cluster environment.

  for(int i=0; i<3; i++){ System.out.print("Say important things three times: like, attention, share!!");
  }  
Copy the code