Recently, I used distributed lock in my work, and then I checked the realization of many distributed locks. I am familiar with Redis, or the usage of Redis is relatively simple, so I looked up the way that Redis uses setnx to implement distributed locks. One article search to the most times, much to the original article, I don’t know which one is just stick a see links blog.csdn.net/lihao21/art…
Reids > setnx(key,value) // Set key. redis > delete(key) // Delete keyCopy the codeIf the key exists, the setting fails and -1 is returned. If the key does not exist, 0 is returned.
Implementation method 1:
lock_key = "distribute_lock"
def get_lock() :
while True:
lock = redis_client.setnx(lock_key,1)
if lock: # setup successful
break
else:
time.sleep(0.5) If not, wait until 0.5 to continue retrieving. You can also set the number of retries.
return True
if get_lock():
Do your work, handle critical resourcesRedis_client. delete(lock_key) // To prevent deadlocks, release locks in time after critical resources are processed.Copy the codeCan you see what’s wrong with this code?
The problem is that the last lock is released. If the process hangs or the redis link breaks during a delete operation, the distributed lock will never be released, resulting in a deadlock. So the next optimization is to be able to release the lock in time if the process dies. What do you have in mind? Timeout mechanism.
Implementation method two:
In order to avoid the problem of method one, timeout mechanism was added.
setnx(key, <current Unix time + lock timeout + 1>)
Copy the codeThe value of the key is set to a timeout.
lock_key = "distribute_lock"
def get_lock() :
while True:
lock = redis_client.setnx(lock_key,<now_time+ lock timeout + 1>) The direct setup succeeded
if lock: # setup successful
break
now_time = <current Unix time + lock timeout + 1>
lock_time_out = redis.client.get(lock_key)
if now_time > lock_time_out: # check whether timeout occurs
If you have timed out
redis_client.delete(lock_key) Delete this key and reset it.
lock = redis_client.setnx(lock_key,<now_time+ lock timeout + 1>)
if lock: # setup successful
break
else:
time.sleep(0.5) If not, wait until 0.5 to continue retrieving. You can also set the number of retries.
return True
if get_lock():
Do your work, handle critical resourcesRedis_client. delete(lock_key) // To prevent deadlocks, release locks in time after critical resources are processed.Copy the codeThink about it a little bit. What could go wrong here?
- If process P1 has acquired the lock, processes P2 and P3 are constantly checking whether the lock has timed out.
- Then the P1 process hung and did not remove the lock in time.
- After some time, p2 and P3 both detect that the lock has timed out, i.e., now_time > lock_TIME_out is valid.
- P2 first removes the lock and then sets the lock timeout so that P2 can acquire it.
- P3 also detects that the lock timed out, but it is slower than the execution speed, and directly deletes the lock. In fact, P3 deletes the lock set by P2.
Implementation Method 3
What are the key issues with approach 2? P3 does not check whether a new process has acquired the lock when deleting the lock. To avoid this, p3 uses this command when executing the set operation:
getset(lock_key,now_time+lock timeout + 1)
Copy the codeThis command returns the old value and sets it to the new value. Determine whether the current time is greater than the old value of lock_key before setting. If the value is greater than that, the timeout has occurred and the lock has been obtained. If the above situation occurs again, both P2 and P3 detect a lock timeout, and P2 deletes and acquires the lock. P3 perform the getSet operation and compare the current time with the old lock_key value (set by P2). The current time is smaller than the old lock_key value. Failed to obtain the lock, continue to wait for the next round.
Secondly, when the final deletion, it can not be directly deleted like the previous several times. To determine first, the current time is less than the lock timeout time in the delete. Avoid removing locks set by other processes. The procedure is as follows:
def get_lock() :
LOCK_TIMEOUT = 3
lock = 0
lock_timeout = 0
lock_key = 'distribute_lock'
# acquiring a lock
whilelock ! =1:
now = int(time.time())
lock_timeout = now + LOCK_TIMEOUT + 1
lock = redis_client.setnx(lock_key, lock_timeout)
if lock == 1 or (now > int(redis_client.get(lock_key))) and now > int(redis_client.getset(lock_key, lock_timeout)):
break
else:
time.sleep(0.001)
if get_lock():
# dou your lock.
# releases the lock
now = int(time.time())
if now < lock_timeout:
redis_client.delete(lock_key)
Copy the codeKeep going. What’s the problem?
- +1 Current Unix time + Lock timeout +1
+1 is because in Redis 2.4, the expiration delay is less than 1 second — that is, even if the key is expired, it can still be accessed within 1 second after expiration, while in Redis 2.6, the delay is reduced to less than 1 millisecond.
- What’s wrong with this code? 1) Because the client generates the expiration time, it is mandatory to synchronize the time of each client in distributed mode. 2) When a lock expires, if multiple clients execute jedis.getSet() at the same time, only one client can be locked, but the expiration time of the lock on that client may be overwritten by other clients. 3) The lock does not have an owner identifier, that is, any client can unlock it.
(1) First of all, the first problem exists, but generally for the project of the same distribution group, time must be synchronized, and which machine time you have seen using local time? They are generally connected to the Internet to synchronize time with the time zones set around the world by the Internet. (2) Redis is single threaded, so there is no such situation. It is impossible to execute simultaneously. Even though clients A,B, and C send getSet to Redis at the same time (accurate to nanosecond). Redis also executes in queue order. Therefore, it is absolutely guaranteed that only one client can obtain the lock, process the business and finally release the lock. Other clients must return a result longer than the current time, which may result in lock failure, and it is impossible for any other client to unlock. “At the same time,” you might say. But when you say “at the same time”, I know, brother you don’t know the bottom layer of Redis, redis is single-threaded. Unless you change their source code, and even if you change their source code, if you change their single-threaded design philosophy, you say you don’t agree with the redis authors. Redis’s single threading, however, does not affect its performance.
- The problems I found in practice.
if lock == 1 or (now > int(redis_client.get(lock_key))) and now > int(redis_client.getset(lock_key, lock_timeout))
Copy the codeRedis_client.get (lock_key) is None, int(None) will cause type conversion errors if the lock is removed during redis_client.get(lock_key) check. Then if it is None, it can be set to 0, and there is no problem because the lock itself is removed, which is the same as the timeout, so optimization is proposed.
if lock == 1 or (now > int(redis_client.get(lock_key) or 0)) and now > int(redis_client.getset(lock_key, lock_timeout) or 0)
Copy the codeSecondly, I optimized a class for ease of use:
class DistributeLock(object) :
def __init__(self, lock_key=None, lock_timeout=2) :
"" :param lock_key: specifies the key of a distributed lock :param lock_timeout: specifies the timeout period of the lock.
self.LOCK_TIMEOUT = lock_timeout
self.lock = 0
self.lock_timeout = 0
self.lock_key = lock_key The key of the distributed lock
self.redis_client = Initialize the Redis client.
def __enter__(self) :
# acquiring a lock
whileself.lock ! =1:
now = int(time.time())
self.lock_timeout = now + self.LOCK_TIMEOUT + 1
lock = self.redis_client.setnx(self.lock_key, self.lock_timeout)
if lock == 1 or (now > int(self.redis_client.get(self.lock_key) or 0)) and now > int(
self.redis_client.getset(self.lock_key, self.lock_timeout) or 0) :break
else:
time.sleep(0.001)
def __exit__(self, ex_type, ex_value, traceback) :
# releases the lock
now = int(time.time())
if now < self.lock_timeout:
self.redis_client.delete(self.lock_key)
# Usage:
with DistributeLock(lock_key="your_distribute_lock",lock_timeout=3) :# do your work.
do_work() No need to worry about lock acquisition or release.
Copy the code