This is a story about a Redis writer seeking a hammer, describing his battle with another god of distributed systems. The battle was too close to call and there was no winner. If we had to pick the biggest winner, it would be the melon eaters. Let eat melon net friend deep experience: the idea that looks so impeccable, careful scrutiny, is not so seamless.

It’s time to share once a week, as usual, and talk about your life in a different way.

The picture above was taken by a teammate during a bike ride when I was in college. The place where I took the photo is called Oxback Mountain, a very low name, but actually very beautiful place.

It’s a tough ride up the hill, super bad roads and really bad slopes. It was torture, a complete test of willpower.

When we were nearly out of ammunition and nearly two hours from the summit, a truck driver offered to take us to the top for free. We said no.

He said, “I will only ride oxback Mountain once in my life. It’s not so bad now. If I get a ride up there, I’ll regret it later. So I prefer to push.”

The next afternoon there was a snowstorm on the mountain. So we southern kids played around in this rare snow scene. Snowball fights, snowmen, skiing……

At night, when the snow stopped, I saw something I will never forget. Starry sky, beautiful to let a person want to cry of starry sky!

Later, in many scenes of my life, I would think of xing’s words: I will only do this once in my life, AND I don’t want to leave any regrets. I think of the night’s bright, palpable stars.

When you can’t hold on to something, it is indeed a commendable spirit.

All right, back to the article.

Background bedding

When interviewing, whether you put Redis on your resume or not, it’s almost a no-go topic.

In order to get to the question about Redlock’s fairy fights, we have to go through an interview riff:

1. What should be paid attention to when Redis does distributed lock?

2. What problems will arise if Redis is a single point of deployment?

3. So how are you going to solve the single point problem?

4. Are there any problems in cluster mode, such as master/slave mode?

5. Do you know how Redis solved the problem of unreliable cluster mode?

6. Why don’t you give a brief introduction to Redlock?

What do you think is wrong with Redlock?

Obviously, the above is a regular interview serial question. There are many other Redis sites that can be inserted in between, but I won’t extend them here.

Single point of Redis to do distributed locking is not reliable, leading to the emergence of distributed locking solutions based on Redis cluster mode.

Distributed lock solution based on Redis cluster mode is still not reliable, the author of Redis proposed a solution of Redlock.

The Redis author’s solution to Redlock was found unreliable by another distributed system guru, and a battle began between them.

In light of the battle, there is more discussion.

The battle was too close to call and there was no winner. If we had to pick the biggest winner, it would be the melon eaters. Because for melon eaters like me, there’s a lot to learn from the clash of the two gods.

Let you deeply understand: the idea that seems so perfect, careful scrutiny, is not so seamless.

Therefore, this article is based on the following five modules.

First to a wave of persuasion: this article nearly 1.2W words, watch carefully. It doesn’t matter if you can’t see it, a “like” is the biggest encouragement for me. Ollie give!

A single point of Redis

In my experience, when talking about Redis in an interview, 90% of my friends will say ** : Redis is used for hot data caching in our project **.

And then a hundred percent of the interviewers will ask,

Besides Redis for caching, what other Redis based uses have you seen?

The next 80 percent of your friends will say: We also used Redis for distributed locks.

(Of course, Redis in addition to caching, distributed locking also has a lot of strange technology nifty, not the focus of this article, you can have a look at the interested.)

The interviewer will then say:

Why don’t you describe (or write pseudocode) the details of redis-based locking and unlocking?

Note that the interviewer is talking about the details of locking and releasing locks. The devil is in the details.

The interviewer wants to hear the following key points:

Key point 1: atomic command locking. Because in some “disrepair” articles, the lock operation of Redis is to set the key first and then set the expiration time of the key. The root reason for this is that atomic locking was not supported in earlier versions of Redis. I don’t need to tell you what the problem would be if it wasn’t atomic, right? If you don’t know, go home and wait.

With 2.6.12, atomic locking can be implemented by sending the following command to Redis:

SET key random_value NX PX 30000

Key point 2: When you set the value, you put random_value. Not when you throw in an “OK”.

First, explain the meaning of several parameters in the above command:

Random_value: IS a random string generated by the client that is guaranteed to be unique among all lock requests from all clients for a long enough period of time.

NX: The set succeeds only when the key to be set does not exist. This ensures that only the first client can acquire the lock, and no other client can acquire the lock until it is released.

PX 30000: indicates that the lock has an automatic expiration time of 30 seconds. Of course, 30 seconds is just an example, and the client can choose the appropriate expiration time.

Again, explain why value needs to be set to a random string. This is also the third key point.

Key point 3: Value is set to a random number to release the lock more securely. When releasing the lock, you need to check whether the key exists and whether the value corresponding to the key is the same as the value specified by me. So you can see that there are three operations: get, judge, and delete. To ensure atomicity, we need to use lua scripts.

(Basically, these key points are answered, and the interviewer moves on to the next question. Regular warm-up quiz, guys, remember that.

Cluster pattern

The interviewer will then ask:

From what we’ve just discussed, we already have a good way to acquire locks and release locks. Based on the Redis singleton, this approach is secure enough, assuming that the singleton is always available. What if the Redis node fails?

We can talk about Redlock directly. But don’t panic. In order to show off your extensive knowledge, you will need to talk about Redis clusters by yourself.

In order to avoid the problems caused by node failure, we can adopt the Redis cluster method to achieve high availability of Redis.

There are three Redis cluster modes: master-slave mode, Sentinel mode, and Cluster mode

In the master/slave mode, the data copy on the slave node is guaranteed. However, after the master node fails, the slave node needs to be manually switched to the master node. It is very simple, but rarely used in a real production environment.

The Sentinel mode is an upgraded version of the primary/secondary mode. In this mode, the primary node that responds to an exception is offline subjectively or objectively, and the primary/secondary switchover is performed. It guarantees high availability.

The cluster mode ensures high concurrency. All data is shared across the cluster, and different keys are placed in different Redis. Each Redis corresponds to a portion of the slot.

(The above three modes are also the focus of the interview, can say a lot of out, because it is not the focus of this article will not be described in detail. You need to show that you know what you’re talking about when you come across a question.

In the cluster mode described above, there is still a problem because the nodes communicate asynchronously. If a lock has just been placed on the Master node, but the data has not been synchronized to Salve. When the Master node hangs, the lock on it is lost. When the new Master node comes out (manual switchover in Master/slave mode or a failover in sentry mode), the same lock can be obtained again. The same lock is obtained twice.

The lock has been picked up twice, so it’s no longer secure. A secure lock, distributed or not, is held by only one client at any one time.

Redlock profile

To solve this problem, the authors of Redis propose an algorithm called Redlock.

In a distributed Redis environment, we assume that there are N Redis masters. These nodes are completely independent of each other and there is no master-slave replication or other cluster coordination mechanism.

Having described how to safely acquire and release locks under a single point of Redis, we ensured that we would use this method to acquire and release locks on N instances.

In the example below, we assume that there are five completely independent Redis Master nodes running on five servers to ensure that they do not go down at the same time.

From the official website, we can know that a client must go through the following five steps to obtain the lock:

Steps describe source: http://redis.cn/topics/distlock.html

1. Obtain the current Unix time, in milliseconds.

2. Use the same key and random value to obtain locks from N instances in sequence. In Step 2, when setting a lock to Redis, the client should set a network connection and response timeout that is less than the lock expiration time. For example, if your lock expires automatically in 10 seconds, the timeout should be between 5 and 50 milliseconds. This prevents the client from waiting for a response when Redis has already failed on the server. If the server does not respond within the specified time, the client should try another Redis instance as soon as possible.

3. The client obtains the lock usage time by subtracting the start time (recorded in Step 1) from the current time. The lock is successful if and only if it has been taken from most of the Redis nodes (three nodes in this case) for less time than the lock expires.

4. If a lock is obtained, the actual validity time of the key is equal to the validity time minus the time used to obtain the lock (calculated in Step 3).

5. If, for some reason, the lock fails to be acquired (not in at least N/2+1 Redis instances or the lock time has expired), the client should unlock all Redis instances (even if some Redis instances are not locked at all).

As you can see from the above steps, as long as most of the nodes are working properly, you can ensure that Redlock works properly. In this way, we can solve the previous single point Redis case where the node hangs, due to asynchronous communication, resulting in lock failure.

However, the security of the lock after a faulty restart cannot be resolved. Consider the following scenario:

We have three nodes A, B, and C.

1. Client 1 is locked successfully on CLIENT A and client B. C The lock fails to be added.

2. Node B crashes and restarts, but the lock of client 1 on node B is not persisted due to the persistence policy. Client 2 applies for the same lock and locks USER B and user C successfully.

3. The same lock is held by both client 1 and client 2.

(Redis persistence strategy again, all knowledge, friends)

For example, Redis’ AOF persistence defaults to write to disk once per second, known as fsync, so at worst, 1 second of data can be lost.

Of course, you could set it to fsync every time you modify the data (fsync=always), but this would severely degrade Redis performance and violate its design philosophy. (I haven’t seen it used in this way either, perhaps too rarely.)

Also, you think you won’t lose data if you perform fsync? Naive, the real system environment is complex, which is beyond the scope of Redis. Up to the server, system problem.

Therefore, according to Murphy’s Law, lock failure due to node restart is always possible, as mentioned above.

To solve this problem, the authors of Redis proposed the concept of delayed restarts.

This means that when a node crashes, do not restart it immediately, but wait a certain amount of time before restarting it. The wait time should be greater than the lock expiration time (TTL). This is done to ensure that any locks the node participates in expire before restarting. Equivalent to the previous account write-off after the ability to participate in the back of the lock operation.

But there is a problem: the node is not working during the waiting time. So if most of the nodes hang, it’s waiting. This causes the system to become unusable, because the system will not be able to lock any locks during the TTL.

Another important aspect of the Redlock algorithm is its lock release operation.

When a lock is released, a lock release operation is initiated on all nodes. The purpose of doing this is to resolve the possibility that during the lock phase, the node receives the lock request and the set succeeds, but the client thinks that the lock is not successful because the response packet returned to the client is lost. Therefore, lock release operations must be initiated on all nodes.

You might think this is not routine?

Some details are just like this, after saying it, I feel that it is just so, but it is possible that I just can’t think of this point, leading to the emergence of the problem, so we will say: details, the devil is in the details.

Well, the introduction probably said here, interested friends can go to see the official website, add.

English: English: http://redis.cn/topics/distlock.html https://redis.io/topics/distlock

All right, after all this time, all this time, we can finally get into the fairy fight.

The fairy to fight

Immortal 1: Antirez, author of Redis. Some of my friends are not very sensitive to English names, so I will call him curly brother.

Immortal 2: Distributed domain expert Martin Kleppmann, let’s call him Rapunzel.

After looking at the photo of the two gods above, and looking at my hair, which is getting thinner for writing this article, I couldn’t help crying out. Maybe a “like” is the only way to calm me down.

At the end of the Redlock page, Curly curls writes: “If you are someone who uses distributed systems, your opinion is very important.

So, “hammer for hammer”! This hammer, hammer out a large number of melon eating netizens, among them there is no lack of professionals in the relevant fields.

Big hair makes a hammer

The story starts from the article “How to Do Distributed locking” published by Changfa Brother on February 8, 2016:

Address: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

This part directly translates to:

As part of my research for this book (Design for Data-Intensive Applications), I came across an algorithm called Redlock on the Redis website. The algorithm claims to implement fault-tolerant distributed locks (or, more specifically, leases) in Redis, and the page requires feedback from distributed system personnel. This algorithm got me thinking, so I spent some time writing my article. Since Redlock already has more than 10 separate implementations, and we don’t know who is already relying on this algorithm, I thought it was worth sharing my notes publicly. I won’t discuss other aspects of Redis, some of which have already been criticized elsewhere.

You said you wanted feedback, I’ll give you feedback. And you have other problems with this thing, so I won’t talk about it. (In fact, the author also said in this article that he likes and uses Redis, but he thinks the Redlock algorithm is not rigorous.)

Long hair brother mainly around the following picture for the expansion:

If you don’t get it at first glance, I’ll give you another Chinese version from The 2017 book “Design for Data-intensive Applications” :

As you can see from the picture above, the key words of applying for lease and lease expiration are mentioned. Lease is actually a lock with a timeout period.

And in the book, the caption at the bottom of the picture goes something like this, and you smack smack:

For example, HBase is designed to ensure that only one client can access a file at a time. If multiple clients attempt to write to the file at the same time, the file will be damaged. So the picture above explains:

1. Client 1 applies for a lock and successfully obtains the lock. The client then did a long GC which resulted in the STW condition.

2. During the STW, the lock obtained by client 1 expires, and the lock becomes invalid.

3. Because the lock of client 1 has expired, client 2 can successfully obtain the lock after applying for it.

4. Client 2 starts and finishes writing files.

5. When client 1 recovers from the STW, it does not know that its lock has expired and continues to write files, causing the files written by client 2 to be damaged. And as you can see, it does not satisfy the principle that the lock is held by only one client at any time, that is, it does not satisfy mutual exclusion.

It’s not in the book, but if you buy it, isn’t the lock service here saying Redis?

Some friends will say, that client 1 write file, and then determine whether their lock has expired?

You’re really a smart cookie, so let me ask you, GC can happen at any time, what if GC happens after judgment?

What if the client uses a language that doesn’t have GC?

**GC is not the only cause of thread pauses, my friends. ** There are many reasons for this. Take a look at the example given in The book of Long Hair brother:

To sum up, even if the lock service is normal, the shared resource is accessed by more than one client simultaneously because the lock is held for a certain time, due to client blocking, long GC, or network reasons.

In fact, the above brother long hair directly said in the book: this is not correct implementation.

You are a multi-product, isn’t it a bit like the previous task hasn’t been completed due to the unreasonable setting of Redis lock expiration time, but the lock time has expired, and the latter task has also applied for the lock?

Redission actually has its own guard dog mechanism for this kind of scenario. But it’s beyond the scope of this Redlock discussion, so I won’t describe it here.

What’s rapunzel’s solution?

It’s called a fencing token.

When using locks and leases to protect concurrent access to resources, He believes that the node whose lock expires due to an exception cannot affect other normal parts. To achieve this goal, he can use fencing, which has always been quite simple.

Suppose that each time the lock service grants a lock or lease, it also returns a fencing token, which increases with each grant.

Then, the client is required to include the fencing token each time it sends a write request to the storage system. The storage system verifies the token. If the request for a higher token has already been processed, the request is rejected.

Like the picture below:

1. Client 1 acquires a lock with a timeout and gets token 33, but then falls into a long pause until the lock expires.

2. Client 2 has obtained the lock and token 34, and then sends a write request (and token 34) to the storage service.

3. Client 1 then recovers and attempts to write with token number 33. The storage server rejects the write request for token number 33 because it records that it has recently completed a higher token number (34).

This version number mechanism reminds me of Zookeeper. When using ZK as the lock service, you can use the transaction identifier ZXID or the node version cversion as the fencing token, both of which satisfy the monotonically increasing requirement.

In Rapunda’s mechanism, it is essentially required that the resource itself must proactively check the token information held by the request, and reject all write requests with lower tokens if it is found that a higher token has already been processed.

, however, not all resources are the inside of the database data, we can use the version number to support additional token check, so for not to support additional token check resources, we can also use this idea to bypass this restriction, such as in the case of access to the file storage service, we can embed the token in the file name.

In summary, additional checking mechanisms are needed to avoid request processing outside of lock protection.

Checking tokens on the server side may seem complicated, but it is recommended that the system service not assume that all clients will behave as expected. From a security perspective, the server must guard against this abuse from the client.

This is similar to how we, as back-end developers, cannot trust data from the front end or other interfaces and must verify it.

At this point, Rapunzel completes his setup and turns his head to RedLock, which he believes is a distributed lock that relies heavily on the system clock.

He gives an example:

1. Client 1 successfully obtains the lock from Redis nodes A, B, and C. D and E cannot be accessed due to network problems.

2. The clock on node C jumps forward, causing the lock on node C to expire.

3. Client 2 successfully obtains the lock of the same resource from Redis nodes C, D, and E. A and B cannot be accessed due to network problems. Now, both client 1 and client 2 think they have the lock.

This scenario is possible because Redlock relies heavily on the system clock, so if the system time becomes inaccurate, the security of the algorithm is not guaranteed.

Rapunzel used this example to support his previous point: a good distributed algorithm should be based on asynchronous models, the security of the algorithm should not depend on any timing assumptions, that is, time as a security guarantee. In the asynchronous model, program pause, message delay or even loss in the network, system time error and other factors should not affect its security, only affect its activity.

To put it bluntly, in extreme cases, distributed systems can’t give a result in a finite amount of time, not a wrong result.

Such algorithms actually exist, such as Paxos, Raft. Clearly, Redlock’s level of security is inadequate by this standard.

As for frizzy’s suggestion of delayed startup, Frizzy’s answer was: What’s wrong with you delaying startup? Delayed start does not yet depend on reasonably accurate time measurements.

It may be that Rapunda doesn’t think the example of a clock jump is a good one. Everyone might think that a clock jump is unrealistic because they are confident that the correct NTP configuration will correct the clock.

In this case, he gives an example of a process pause that could cause an algorithm to fail:

1. Client 1 initiates A lock request to Redis nodes A, B, C, D, and E.

2. Each Redis node has returned the request result to client 1, but client 1 enters the long GC phase before receiving the request result.

3. Long GC, resulting in lock expiration on all Redis nodes.

4. Client 2 applies for and obtains the lock on A, B, C, D, and E.

5. Client 1 recovers from the GC phase and receives the results of the previous step 2 request from each Redis node. Client 1 thinks it has successfully acquired the lock.

6. Both client 1 and client 2 now believe that they hold the lock.

In fact, as long as we are clear about the locking process of Redlock, we know that this situation has no effect on Redlock, because in Step 5, after client 1 recovers from the GC phase, in the Redlock algorithm, If a lock is obtained, the key’s true validity time is equal to the validity time minus the time used to acquire the lock.

So client 1 checks that the lock has expired and will no longer consider itself successful in obtaining the lock.

And curly Hair later said it in a riposte.

But, thinking about it, I don’t think that’s what Long Hair was trying to do. In addition to the above problems, he would like to highlight that it is not good for a lock to expire before it is used. You give me a lock, I don’t use it, and you expire?

In addition to the above mentioned points, The elder brother of long hair also offered a kind of his own experience:

What do we get locks for?

For him it’s all about efficiency and correctness. He described them as follows:

If it’s for efficiency, it’s to coordinate the clients so they don’t have to do the same work. In this scenario, even if the lock fails occasionally, it’s just possible that two clients do the same job, resulting in either a slight increase in cost (you end up paying AWS 5 cents more than you would have) or inconvenience (for example, the user ends up getting the same email notification twice).

For correctness, lock failure should never occur because it could mean data inconsistencies, data loss, file corruption, or other serious problems. (Say a patient gets twice the dose)

In the end, Rapunzel concludes: Neither fish nor fowl.

For the scenario of improving efficiency, using distributed lock and allowing occasional lock failure, the locking scheme using single Redis node is sufficient, simple and efficient. Redlock is too heavy.

It is not a strong enough algorithm because it is time dependent in high accuracy scenarios. Redlock did not maintain correctness.

What technology should be used?

For Rapunzel, consider a solution like Zookeeper, or a database that supports transactions.

Curly Hair fights back

The day after Posting How to Do Distributed locking, Curly Hair hit back with a post titled Is Redlock Safe? In the article.

Address: http://antirez.com/news/101

The big guy is the big guy, and curly’s riposte is clear and smooth. He concluded that There were two main aspects that Rapunzel felt unsafe about Redlock:

1. Distributed locks with the automatic expiration function require a fencing mechanism to prevent clients from using the lock after the expiration date. In this way, shared resources are mutually exclusive. Rapunzel says Redlock doesn’t have that.

2. No matter how problem “1” is solved, the algorithm is inherently unsafe because it makes time-keeping assumptions about system models that cannot be guaranteed in real systems, says Changfa.

For the first point, Curly Has listed five points to counter this problem. One important point is that although Redlock does not provide a monotonically increasing token like fencing, it does have a random string, which can be used as a token to achieve the same effect. When interacting with a shared resource, we check to see if the token has changed, and if no acquisition-modify-write back operation is performed.

The final result was a soul-searching question: why use a distributed lock and require it to provide such strong security when there is already a fencing mechanism that can maintain mutually exclusive access to resources even if the lock fails?

However, the second problem, network delay or GC pause, as we have analyzed previously, has no impact on the security of Redlock, indicating that Curling brother actually considered the time factor when designing.

But if It was Rapunzel who suggested that the clock jump, it’s clear that Rapunzel knew that if the clock jump, Redlock’s security wouldn’t be guaranteed, it was his lifeline.

But for the two examples that Rapunzel raised when he wrote about clock jumping:

1. The O&M personnel manually change the system clock.

2. A large clock update event is received from the NTP service.

Curly Hair hit back:

Firstly, the operation and maintenance personnel manually modify the clock, which belongs to human factors. I can’t do anything about it. They just want to fuck you. Tighten up. Don’t do it.

The second problem is that a large clock update is received from the NTP service. O&m is required to ensure this problem. When you need to update a large amount of time to the server, you should do it in a small number of times. Multiple changes, each update time as small as possible.

The debate about this place depends on whether you believe the man with long hair must jump in time, or whether you believe the man with curly hair can jump in time and we can handle it.

Here’s an article about clock jumping, which is also the result of this fight:

https://jvns.ca/blog/2016/02/09/til-clock-skew-exists/

The final conclusion: clock jumping exists.

We’ve all had the experience of jumping clocks, remember on the last day of 2016 when the concept of a “leap second” was introduced? Resulting in the 07:59:60 spectacle on January 1, 2017.

The center of the fight

After such a round, in fact, the focus of the fight is very clear, is the impact of large delay on distributed lock.

As for the impact of a large delay on Redlock, as analyzed by Rapunzel, the lock expires and the business is not finished. Curler sees this effect not only on Redlock, but also on other distributed locks with auto-release locks.

As for the big delay issue, I found the following conversation between two gods on a social media platform:

Curly hair asks: I was wondering if, after my post, we could all agree that a big message delay wouldn’t hurt The Redlock operation.

I agree with you on message latency between client and lock server. However, the latency between the client and the resource being accessed is problematic.

So curly-haired agrees that large system clock jumps will cause Redlock to fail, based on his riposte and a social media post. He differs from Rapunzel in that he believes that large clock jumps can be avoided in real systems with good operations.

So here, the two immortals seem to have reached a balance, to achieve the debate on the common ground.

Fight to summarize

As a practitioner of the Internet industry and a user of distributed system, I have benefited a lot from reading their article and the knowledge points derived from this article, so I write this article as a learning summary and share it with you. There are still many deficiencies in this article, please bear with me.

As the article begins, there are no winners in this debate. Apparently Curler didn’t convince Rapunzel, because Rapunzel’s 2017 book, Design for Data-Intensive Applications, includes a section called Unreliable Clocks

In fact, at the end of the argument, Rapunzel summed up the debate in a very emotional way, saying:

The following translation from: https://www.jianshu.com/p/dd66bdd18a56

The most important thing for me is this: I don’t care who is right or wrong in this debate — I only care about what we can learn from the work of others so that we can avoid repeating the mistakes of the past and make the future better. Great things have been done for us: we can build better software on the shoulders of giants.

Be sure to examine any ideas thoroughly, by proving them and checking that they stand up to scrutiny. That’s part of the learning process. But the goal should be to gain knowledge, not to convince others that you are right. Sometimes it just means to stop and think.

Eat melon net friend harvest

And by “melon eating” I mean me.

Write this article my harvest is still quite big, first I bought long hair elder brother’s “data intensive application system design” book, read a few sections, found this book is really good, Douban score 9.6, recommend.

Secondly, I finished the weekly task of this week, although it was very difficult to write from noon on Saturday to 3 am on Sunday…

Then he ate another melon, melon within melon.

On Friday, the Redis website went down for a short period of time, which is not unusual, but the page showed that Redis was not accessible. Now that’s a little bit interesting.

While writing this, I found this on one of Curly’s social media platforms:

It’s not OOM I care about, it’s Curly Hair running Redis on a $5 / month virtual machine with 1 gigabyte of memory. Ha ha ha, shock, this melon tastes good.

Finally, because curly brother is an Italian, due to the recent epidemic, sichuan expert group to help Italy, big thank Chinese. In fact, this user’s answer is very good: return the favor.

Let the epidemic pass, world peace.

One last word (for attention)

When I write here, it is already more than 3 o ‘clock in the morning, but because I have been following the fierce discussion of these two gods, my thinking is unusually clear.

I can’t tell who’s right and who’s wrong when I’m done. In my opinion, everyone’s starting point for system design is different. There is no perfect architecture and no universal architecture, but a good architecture balanced between perfect and universal architecture is a good architecture.

I glanced at the word count of the article and almost broke through 1.2W words. Maybe it’s another article that nobody reads, but it doesn’t matter, as long as one person reads my article and thinks it’s helpful.

Give me a thumbs up, I’m tired of writing articles, don’t fuck me, I need some positive feedback.

If you find something wrong, please leave a message and point it out to me so that I can modify it. (I have this quote in every technical article, and I mean it.)

Thank you for reading, I insist on original, very welcome and thank you for your attention.

I am why Technology, a nice sichuan man who is not a big shot, but likes to share, warm and interesting.

Reference links http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html 1. 2. https://redis.io/topics/distlock http://antirez.com/news/101 https://www.jianshu.com/p/dd66bdd18a56 4. 5. 3. the data-intensive application system design

Welcome to pay attention to the public account [WHY Technology], adhere to the output of original. Share technology, taste life, wish you and I progress together.