Client double commit? How many times did the timed task execute? Don’t worry, distributed locks are here to help

When multiple different threads access the same resource, race relationship will be formed, and malicious events such as the resource being processed twice will occur. So a mechanism is needed to ensure that a resource is handled by only one service at a time. This is where the mechanism of distributed lock service arises.

There are many ways to implement a distributed lock service. Generally, a distributed lock service requires three characteristics

  • Exclusivity: For a lock, at any given time, only one user can acquire the lock
  • Avoiding Deadlocks: Make sure that you do not deadlock, even before releasing the lock
  • Fault tolerance: Unlocking operations are guaranteed as long as a majority of the nodes in the service cluster are alive

Redis implementation

Distributed locks with Redis Distributed locks with Redis

An overview of the

Redis has become one of the most popular implementations because of its high efficiency and simple operation. As a KV database, Redis ensures that the key is globally unique, and the implementation method is relatively simple. Prior to version 2.6.12, two commands were required to create a time-sensitive record that could not be overwritten as a lock

// precludes EXPIRE lock 10 from EXPIRE lock 10 // precludes EXPIRE lock 10 from EXPIRE lock 10

But methods that require two statements break their atomicity, causing unexpected problems. In version 2.6.12, a new set parameter was added to make it possible to implement both commands in a single statement

SET lock 1 EX 10 NX

Here’s what happens in a real world scenario

Has the lock been destroyed by a non-owner?

To prevent other threads from deleting the lock by mistake, we also need to use Lua script to realize atomic determination of whether the thread destroying the lock is the lock holder. The solution is that different threads destroy the lock with their own marks to destroy it. The idea of Lua code is as follows:

if redis.call("GET",KEYS[1]) == ARGV[1]
then
    return redis.call("DEL", keys[1])
else
    return 0
end

Processing data takes too long and another thread gets the lock?



In the real world, there will be A situation where two threads (A and B) are trying to pick up the lock, A wins and B loses, but A is processing data very slowly or is performing GC. At this time, the lock has expired and B has taken the lock. B quickly finished processing the data and has submitted it. After A finished processing the data and submitted it, the data was processed twice.

Although it is possible to make the expiration time redundant or to use a daemon to automatically renew it, both of these options have the potential to cause deadlock.

Let me know in the comments section how to solve this problem. Thank you.

Redis Cluster Mode with Lock?

The use of distributed locks in a clustered Redis environment is a little more complicated. For example, if a thread succeeds in a set lock on the primary node of Redis, but the primary node crashes and the data has not been synchronized to other nodes, the effect of the lock will be lost. To solve this problem, Redis authors designed RedLock scheme to deal with the problem of lock failure after master-slave switch. The Redlock design is based on two premises

  • Deployment method uses multiple instances to deploy separately
  • Nodes to more, the official recommendation of more than 5 and the locking process is also changed to
  • Gets the current UNIX time
  • Lock each Redis node with the same key and globally unique value successively, and set the timeout time less than the expiration time of the lock
  • The lock is considered successful only if more than half of the nodes return successfully and the time in use is less than the lock expiration time
  • In essence, the design of Redlock is still based on the fault tolerance problem in the distributed system, that is, as long as most nodes are available, then the whole system can also provide external services. For Redlock scheme also led to the industry two big men Antirez and Martin’s debate, here is not much to describe, Martin’s discussion of distributed lock can be seen how to do distributed lock.

ZooKeeper implementation

ZK is a distributed file service system, and ZNode is its basic node, which is also the key to realize the distributed lock service mechanism. For ZK, the path of ZNode is unique, which provides support for the mutual exclusion of distributed locks. When creating a node, the -e parameter can be used to make the node become a temporary node. Temporary means that when the connection is broken, the modified node will destroy itself. In this way, if a thread has an abnormal crash when processing data, the lock will be released automatically, and there will be no deadlock. Finally, the ZooKeeper cluster is also guaranteed to be highly available, so as long as half of the ZKs on it are still working properly, there will be no problems.

The actual process is as follows:

  1. Thread A and thread B create /lock node (create-e /lock 1)
  2. Thread A takes the lock and starts processing the data. Thread B did not get the lock registration listener and is waiting for notification
  3. When thread A finishes processing the data, delete the /lock node (delete /lock) and destroy the lock

Herding problem

In reality, there will be multiple threads trying to obtain the lock, and the losers who fail to obtain the lock will uniformly register the watcher. In this way, the message needs to be pushed to all threads when it needs to be notified to each thread, resulting in a lot of waste of resources. But we don’t actually need to notify all threads, just one thread. Therefore, temporary sequential nodes will be used as the implementation of the lock, and different threads will monitor the data of the previous node to realize queuing monitoring. Avoid sending each notification to multiple threads.

Comparison with the REDIS scheme

  1. As ZooKeeper itself provides distributed coordination function, it has high availability, strong consistency and other capabilities, so the lock model is more robust than Redis
  2. The existence of Watcher avoids the waste of resources caused by non-locking threads retrying
  3. ZooKeeper is weaker than the Redis solution because it takes more time to create and destroy nodes than Redis does to create and delete records