preface
1. What is distributed lock
Distributed locking is a way to control synchronous access to shared resources between distributed systems. In distributed systems, they often need to coordinate their actions. If different systems or hosts on the same system share one or a group of resources, the access to these resources must be mutually exclusive to prevent interference and ensure consistency. In this case, distributed locks are required.
2. Why use distributed locks
To ensure that a method or attribute can only be executed by the same thread at the same time in the case of high concurrency, Java concurrent processing apis (such as ReentrantLock and Synchronized) can be used for mutual exclusion control in the case of single application deployment.
In a stand-alone environment, Java provides many apis for concurrent processing. But with the needs of the development of business, the original single standalone deployment system was evolved into the distributed cluster system, due to the distributed system multithreading, multi-process and distribution on different machines, it will make the original single deployment of concurrency control lock strategy fails, pure Java API does not provide the ability of distributed lock. To solve this problem, you need a mutual exclusion mechanism across JVMS to control access to shared resources, and that’s where distributed locking comes in!
Here’s an example:
Machines A and B are A cluster. The programs on machines A and B are the same and have high availability.
A,B machines have A timed task, every night at 2 o ‘clock in the morning need to execute A timed task, but this timed task can only be executed once, otherwise it will report an error, that A,B two machines in the execution of the time, you need to grab the lock, who grab the lock, who execute, who can not grab, do not need to execute!
3. Lock handling
Using locks in a single application: (single process multithreading)
synchronize
Distributed lock A method of controlling synchronous access to resources between distributed systems
Distributed lock is a way to control the synchronization and sharing of resources between distributed systems
4. Implementation of distributed lock
Data – based optimistic locking implements distributed locking
Distributed locks based on ZooKeeper temporary nodes
Distributed lock based on Redis
Redis knowledge summarized a mind map to share with you
5. Redis distributed lock
Acquiring a lock:
In the set command, there are many options to modify the behavior of the command. Here is the basic syntax of the options available for the set command
Redis 127.0.0.1:6379 > SET the KEY VALUE [EX seconds] [PX milliseconds] [NX | XX] - EX seconds SET specified expiration time (in seconds) - PX milliseconds Set the specified expiration time in milliseconds. -nx: set the key only when the key does not exist. -xx: set the key only when the key already existsCopy the codeMethod 1: Recommendation
``` private static final String LOCK_SUCCESS = "OK"; private static final String SET_IF_NOT_EXIST = "NX"; private static final String SET_WITH_EXPIRE_TIME = "PX"; public static boolean getLock(JedisCluster jedisCluster, String lockKey, String requestId, int expireTime) { // NX: Guaranteed mutual exclusion String result = jediscluster. set(lockKey, requestId, SET_IF_NOT_EXIST, SET_WITH_EXPIRE_TIME, expireTime); if (LOCK_SUCCESS.equals(result)) { return true; } return false; } ` ` `Copy the codeMethod 2:
public static boolean getLock(String lockKey,String requestId,int expireTime) {
Long result = jedis.setnx(lockKey, requestId);
if(result == 1) {
jedis.expire(lockKey, expireTime);
return true;
}
return false;
}
Copy the codeNote: recommended method 1, because setnx and expire in method 2 are two operations, not an atomic operation. If setnx has a problem, it will be a deadlock situation
Release the lock:
Mode 1: The del command is implemented
public static void releaseLock(String lockKey,String requestId) { if (requestId.equals(jedis.get(lockKey))) { jedis.del(lockKey); }}Copy the codeMethod 2: Redis + Lua script is recommended
public static boolean releaseLock(String lockKey, String requestId) {
String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return
redis.call('del', KEYS[1]) else return 0 end";
Object result = jedis.eval(script, Collections.singletonList(lockKey),
Collections.singletonList(requestId));
if (result.equals(1L)) {
return true;
}
return false;
}
Copy the code6. Distributed lock of ZooKeeper
The following is a map of ZooKeeper knowledge points to share with you
6.1 ZooKeeper Implements distributed locks
After you understand the principle of locking, you will find that Zookeeper is born as the embryo of a distributed lock.
First of all, each Zookeeper node is a natural sequential number generator.
When a child node is created under each node, whenever the creation type selected is EPHEMERAL_SEQUENTIAL or PERSISTENT_SEQUENTIAL, a sequential number is appended to the new child node. This sequence number is the sequence number of the last generation plus one
For example, if you create a node “/test/lock” for issuing numbers and use it as the parent node, you can create children under the parent node with the same prefix, assuming that the same prefix is “/test/lock/seq-“, and specify the ordered type when creating children. If it is the first child created, the child is /test/lock/ seQ-0000000000, the next node is /test/lock/ seQ-0000000001, and so on.
Secondly, the incremental nature of Zookeeper nodes can stipulate that the node with the smallest node number obtains the lock.
A ZooKeeper distributed lock first needs to create a parent node (PERSISTENT node), and then each thread that wants to acquire the lock will create a temporary sequential node under this node. As the sequence number is increasing, the one with the smallest row number can be specified to obtain the lock. Therefore, before attempting to occupy the lock, each thread first determines whether it is currently the smallest row and, if so, acquires the lock.
Third, the node monitoring mechanism of Zookeeper can ensure the orderly and efficient way of locking.
Before each thread preempts the lock, it creates its own ZNode. Similarly, to release the lock, you need to delete the Znode. If the node is not the node with the smallest row number, the node is waiting for notification. Who are you waiting for? No one else is needed, just wait for notifications from the previous Znode. When a Znode is deleted, it is your turn to own the lock. The first one informs the second, the second one informs the third, beating the drum and passing the flowers back in turn.
Zookeeper node monitoring mechanism can be said to be very perfect to achieve this kind of information transmission like beating drums and passing flowers. The specific method is that each Znode waiting for notification only needs to listen on linsten or Watch to monitor the node in front of it, and the node immediately in front of it. As long as the previous node is deleted, it makes another judgment to see if it is the node with the smallest serial number. If so, it gets the lock.
Why is Zookeeper’s node listening mechanism perfect?
One dragon, end to end, back to front, not afraid to cut off in the middle? For example, in a distributed environment, if the first node fails to be deleted by the program due to network problems, server crashes, or other reasons, the next node will wait forever.
In fact, the internal mechanism of Zookeeper ensures that subsequent nodes can normally monitor deletion and lock acquisition. Try to create a temporary ZNode instead of a permanent zNode when creating the numbered node. If the client of the ZNode loses contact with the Zookeeper cluster server, the temporary ZNode will be deleted automatically. The next node in line also receives the delete event, which acquires the lock.
Zookeeper’s node listening mechanism is perfect. There’s another reason.
Zookeeper avoids herding by connecting heads to tails and listening from the back. Herding is when every node fails, all the nodes listen and react, and that puts a lot of pressure on the server, so with temporary sequential nodes, when a node fails, only the node behind it responds.
6.2 ZooKeeper provides an example for implementing distributed locks
Zookeeper implements distributed locking through temporary nodes.
import org.apache.curator.RetryPolicy; import org.apache.curator.framework.CuratorFramework; import org.apache.curator.framework.CuratorFrameworkFactory; import org.apache.curator.framework.recipes.locks.InterProcessMutex; import org.apache.curator.retry.ExponentialBackoffRetry; import org.junit.Before; import org.junit.Test; /** * @className ZookeeperLock * @description TODO * @author lingxiangxiang * @date 2:57pm * @version 1.0 **/ public Class ZookeeperLock {// Define shared resources private static int NUMBER = 10; Private static void printNumber () {/ / business logic, seconds kill System. Out. The println (" * * * * * * * * * business methods start * * * * * * * * * * * * \ n "); System.out.println(" current value: "+ NUMBER); NUMBER--; try { Thread.sleep(2000); } catch (InterruptedException e) { e.printStackTrace(); } System. Out. Println (" * * * * * * * * * end of the business method * * * * * * * * * * * * \ n "); } public static void main(String[] args) {// define the RetryPolicy 1000 wait time (ms) 10 retry times RetryPolicy policy = new ExponentialBackoffRetry(1000, 10); / / define the zookeeper client CuratorFramework client = CuratorFrameworkFactory. Builder () The connectString (" 10.231.128.95:2181,10.231. 128.96:2181,10.231. 128.97:2181 "). The retryPolicy (the policy). The build (); // Start the client.start(); // Define a lock in ZooKeeper final InterProcessMutex lock = new InterProcessMutex(client, "/ myLock "); // start a thread for (int I = 0; i <10; I++) {new Thread(new Runnable() {@override public void run() {try { printNumber(); } catch (Exception e) { e.printStackTrace(); } finally {// try {lock.release(); } catch (Exception e) { e.printStackTrace(); } } } }).start(); }}}Copy the code7. Distributed locking based on data
When discussing the use of distributed locks, we tend to rule out database-based solutions, which are instinctively not “advanced” enough. From the perspective of performance, the performance of database-based solutions is indeed not excellent. The overall performance comparison is: Cache > Zookeeper, ETCD > database. Others suggest that database-based solutions are problematic and unreliable. The database schema may not be suitable for frequent write operations.
Let’s take a look at database-based (MySQL) solutions, which generally fall into three categories: table-record based, optimistic locking, and pessimistic locking.
7.1 Based on table records
Perhaps the easiest way to implement distributed locking is to simply create a lock table and manipulate the data in that table. When we want to acquire a lock, we can add a record to the table, and when we want to release the lock, we can delete the record.
For better demonstration, let’s create a database table as follows:
CREATE TABLE 'database_lock' (' id 'BIGINT NOT NULL AUTO_INCREMENT,' resource 'int NOT NULL COMMENT' lock ', 'description' varchar(1024) NOT NULL DEFAULT COMMENT 'description ', PRIMARY KEY (' id') Uiq_idx_resource '(' resource') ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT=' mysql ';Copy the codeGets the lock
We can insert a piece of data:
INSERT INTO database_lock(resource, description) VALUES (1, 'lock');
Because resource is the only index in the database_lock table, other requests submitted to the database will be reported as an error and will not be successfully inserted, only one can be inserted. Insert successfully, we get the lock
Remove the lock
INSERT INTO database_lock(resource, description) VALUES (1, 'lock');
This implementation is very simple, but the following points need to be noted:
This type of lock has no expiration time, and failure to release the lock will cause the lock record to remain in the database, preventing other threads from acquiring the lock. This defect can also be easily fixed, such as doing a scheduled task to periodically clean up.
The reliability of this lock depends on the database. You are advised to configure a standby database to avoid single points and improve reliability. This type of lock is non-blocking because an error is reported when the data fails to be inserted and the lock needs to be accessed again. If you need blocking, you can create a for loop, a while loop, etc., until the INSERT succeeds and then return.
This lock is also non-reentrant because the same thread cannot acquire the lock again without releasing it because the same record already exists in the database. If you want to achieve reentrant lock, you can add some fields in the database, such as the host information, thread information, etc., so when you get the lock again, you can query the data first. If the current host information and thread information can be found, you can directly assign the lock to it.
7.2 optimistic locking
As the name implies, the system considers data updates to be conflict-free in most cases and only checks data for collisions when a database update operation is committed. If the result of the test is inconsistent with the expected data, a failure message is returned.
Optimistic locking is mostly implemented based on the version of data logging mechanism. What is a data version number? In the version solution based on database tables, the version number is read together by adding a “Version” field to the database table, and the version number is added by 1 when the data is updated later. During the update, the system compares the version numbers. If the version numbers are the same and have not changed, the operation succeeds. If the version numbers are inconsistent, the update fails.
In order to better understand the use of database optimistic locks in real projects, here is also the industry cliche inventory example. An e-commerce platform will have an inventory of goods, and users will operate the inventory when making purchases (inventory minus 1 means that one item has been sold). If only one user does the operation, the database itself can guarantee the correctness of the user’s operation, while in the case of concurrency, there are some unexpected problems:
For example, when two users purchase a commodity at the same time, the actual operation at the database level should be inventory reduction by 2. However, due to the high concurrency, the first user completes the purchase, reads the current inventory and performs the operation of inventory reduction by 1, because this operation is not completely completed. The second user to purchase the same goods, query the inventory at this time may be not minus 1 operating inventory led to the emergence of dirty data [thread unsafe operation], usually if it is a single JVM using JAVA’s built-in lock can ensure thread safety, if in the case of multiple JVM, using distributed lock can be achieved, later will fill 】 【 This article focuses on the database level.
To address the above problem, optimistic database locking can also be used to ensure thread-safety, which is usually done at the code level:
Select goods_num from goods where goods_name = "goods_name "; Update goods set goods_num = goods_num -1 where goods_name = "";
The above SQL is a group, usually query the current goods_num, and then delete the goods_num by 1 to modify the inventory. In the case of concurrency, this statement may lead to the overselling of an item with 3 inventory after two people buy the item with 2 inventory left. So how is database optimistic locking implemented?
First define a version field to be used as a version number. Each operation will look like this:
Select goods_num,version from goods where goods_name = "goods_num "; Update goods set goods_num = goods_num -1,version = increment where goods_name =" 本 页 "and version= query version;
In fact, optimistic locking can be achieved with an update timestamp (updated_at) in a similar way to the version field: the front-line of the update operation retrieves the current update time and, when the update is committed, checks whether the current update time is the same as the update timestamp obtained at the start of the update.
7.3 pessimistic locks
In addition to adding and deleting records in database tables, we can also use the database’s own locks to achieve distributed locking. Add FOR UPDATE to the end of the query statement, and the database will add pessimistic locks, also known as exclusive locks, to the database table during the query. When a pessimistic lock is added to a record, other threads cannot add pessimistic locks to other rows.
Pessimistic locks, as opposed to optimistic locks, always assume the worst and assume that data updates will conflict in most cases.
While using pessimistic locks, we need to pay attention to the level of locks. MySQL InnoDB causes rows to be locked only if the primary key (or index) is explicitly specified, otherwise MySQL will perform table locks.
When using pessimistic locks, we must turn off the MySQL database’s auto-commit property (see the example below) because MySQL uses autocommit mode by default, which means that when you perform an update, MySQL commits the result immediately.
mysql> SET AUTOCOMMIT = 0; Query OK, 0 rows affected (0.00 sec)
This allows you to execute the business logic after obtaining the lock using FOR UPDATE, and then release the lock using COMMIT.
We can use the previous database_lock table to specify usage. Suppose thread A needs to acquire the lock and perform the corresponding operation. The steps are as follows:
SELECT * FROM database_lock WHERE id=1 FOR UPDATE; .
STEP2- execute the business logic.
STEP3- Release lock: COMMIT
The last
I here organized a: Redis and ZK related information documents, Spring family barrel series, Java systematic information, (including Java core knowledge points, interview topics and the latest 20 years of the Internet, e-books, etc.) friends who need to pay attention to the public number [procedure Yuan Small wan] can be obtained.