Seems to have not written a blog for a long time, taking advantage of this period of time more leisure, especially to summarize the concurrent problems and solutions encountered in the process of business system development, I hope to help you 😁
Problem of repetition
1. “Equipment A Strange Body”
Time to go back to the long, long ago a late at night, then I develop multimedia advertising control system has just launched, the production of the company on the first line, fresh store dozens of large and small multimedia hardware equipment normal after connected to the Internet, is a a registered by me and to have online access to the multimedia advertising broadcast control system. The registration process is summarized as follows:
After each device is registered in the system, a new record is added to the device table in the database to store the device information. Everything was going smoothly until the registration of device A broke the silence… After device A was registered, I suddenly found that two new records were added to the database device table, and they were two identical records! I was beginning to think I was seeing things… A closer look, is indeed two new, and even the device unique identifier (underline, later to test) and the creation time are exactly the same! Looking at the screen, I was lost in thought… Why are there two? In my registration logic, I will check whether the device already exists in the database before falling into the database. If it exists, I will update the existing device. If it does not exist, I will add it. So I was puzzled, by this logic, where did the second identical number come from?
2. Concurrent requests behind the truth
After some investigation and reflection, I found that the problem may be the registration request. Device A may send multiple HTTP registration requests to the cloud center at the same time. The cloud server was deployed on multiple Docker containers at that time. Through checking the logs, it was found that two containers received registration requests from device A at the same time. From this, I speculate: Equipment A sent two registration request at the same time, the two requests, respectively, at the same time playing into different containers in the cloud, according to my registration logic, the two containers after receiving the registration request, at the same time to query the database table equipment, equipment list at this time is not A record of the equipment, so the two containers are carried out the operation of the new, because of the fast, Therefore, there is no difference in the creation time of the two new records, which is accurate to the second.
3. Concurrent new extensions
Since concurrent additions can be problematic, can concurrent updates be problematic?
The solution
Resolving Concurrent Adding
1. UNIQUE INDEX
Database tables are created by creating unique indexes for fields that are unique (such as the device unique identifier above) or by creating joint unique indexes for fields that are unique when combined.
So when complicated with new, as long as there is a new successful, other new operation is because the database the exception thrown (Java. SQL. SQLIntegrityConstraintViolationException) and failure, we only need to deal with the situation of the new failure.
Note that uniquely indexed fields need to be non-empty, because a null field value invalidates the unique index constraint
2. Java distributed lock
After a distributed lock is introduced into the program, you must obtain the distributed lock before performing a new operation. Otherwise, the new operation fails.
This can also solve the problem of data duplication caused by concurrent inserts, but the introduction of distributed locks also increases the complexity of the system. If there are unique fields on the database data to be dropped, it is recommended to use the method of unique index.
In the process of building distributed locks, we need to use Redis. Here we take the distributed locks used for device registration as an example.
Distributed lock simple Q&A:
Q: What exactly is a lock?
A: Lock is essentially a string stored in Redis and generated based on specific rules (in the example, it is a fixed prefix + unique identifier of the device), which means that each device has its own lock when it is registered. Because there is only one lock, even if the device has multiple registration requests coming at the same time, Only the request to obtain the lock was successful.
Q: What is lock acquisition?
A: The string generated by the same device based on the same rules is always the same. Before performing A new operation, check whether the Key exists in Redis. If the Key exists, it means that the lock fails to be obtained. If the Key does not exist, it will be stored in Redis. If the storage is successful, it means that the lock is successfully obtained. If the storage fails, it still means that the lock fails to be obtained.
Q: How does the lock work?
A: As mentioned above, the string (Key) generated by the same device based on the same rules is always the same. Before the current thread performs the new operation, check whether the Key exists in Redis first. If the Key exists, it indicates that another thread has successfully acquired the lock and is doing the new operation that the current thread wants to do. The current thread does not need to perform any further operations (yes, you are redundant)
If the Key does not exist, it means that no other thread has acquired the lock, and the current thread can proceed with the next operation — save the Key quickly in Redis. When the storage of the Key fails, it means that another thread saved the Key first and acquired the lock successfully. The current thread can be deactivated.
If and only if in Redis into the Key success, also said the current thread finally acquiring a lock is successful, can be at ease for the back of the new operation, the new operation during other want to do the same thread lock, and since he couldn’t get all exits only bye 👋, the current thread execution after remember to release the lock (remove the Key from the Redis).
The distributed lock code used for registration is as follows:
Public class LockUtil {// Autowired private RedisService RedisService; @value ("${redis.register.prefix}") private String REDIS_REGISTER_KEY_PREFIX; // Lock expiration time: That is, the maximum time that the thread can operate after acquiring the lock, after which the lock is automatically released (invalid). @value ("${redis.register.timeout}") private Long @value ("${redis.register.timeout}") private Long REDIS_REGISTER_TIMEOUT; @param deviceMacAddress Mac address of the device @return */ public Boolean getRegisterLock(String) deviceMacAddress) { if (StringUtils.isEmpty(deviceMacAddress)) { return false; } // Obtain the lock String (Key) String redisKey = getRegisterLockKey(deviceMacAddress); If (redisservice.exists (redisKey)){return false; if (redisservice.exists (redisKey)){return false; } // Start the lock. Note that you need to use SETNX (because multiple threads may arrive at this step at the same time to start the lock. Use SETNX to ensure that one and only one setting is returned successfully.) Boolean setLock = redisService. SETNX (redisKey, null); If the thread has not released the lock by the expiration time, Redis, which holds the lock, will ensure that the lock is finally released, so as to avoid deadlock. Boolean setExpire = redisservice. expire(redisKey, REDIS_REGISTER_TIMEOUT); Boolean setExpire = redisservice-expire (redisKey, REDIS_REGISTER_TIMEOUT); If (setLock && setExpire) {return true; if (setLock && setExpire) {return true; } // If the lock is set successfully but the expiration time fails to be set, manually clear the lock Key redisservice.del (redisKey). return false; } @param deviceMacAddress Mac address of the device */ public void delRegisterLock(String deviceMacAddress) { redisService.del(getRegisterLockKey(deviceMacAddress)); } /** * obtain the key of the distributed lock during device registration * @param deviceMacAddress deviceMacAddress (the MAC address of each device is unique) * @return */ private String getRegisterLockKey(String deviceMacAddress) { return REDIS_REGISTER_KEY_PREFIX + "_" + deviceMacAddress; }}Copy the codeAn example of using locks in normal registration logic is as follows:
public ReturnObj registry(@RequestBody String device){ Devices deviceInfo = JSON.parseObject(device, Devices.class); Boolean registerLock = lockutil.getregisterlock (deviceinfo.getmacAddress ()); if (! RegisterLock) {log.info(" Failed to obtain device registration lock, current registration request failed!" ); return ReturnObj.createBussinessErrorResult(); ReturnObj result = registerDevice(deviceInfo); Lockutil.delregisterlock (deviceinfo.getmacAddress ())); return result; }Copy the codeResolving concurrent updates
1. Do concurrent updates really cause problems?
When simultaneous updates or successive updates occur and have no impact on the business, there is no need to do anything to avoid adding unnecessary complexity to the system.
2. Optimistic locking
Repeat updates can be avoided by optimistic locking, that is: In a database table to join a “version (version)” field, in the query before making the update operation record, write down the query out the version number, in the actual update operation after judge query to find out whether the version number of the previous and current database which record the version number is consistent, if, in the current thread from the query to update this time, No other thread updates this record; If not, it indicates that other threads have changed the record during this period, and the update operation of the current thread is no longer safe and must be abandoned.
Judge SQL example:
update a_table set name=test1, age=12, version=version+1 where id = 3 and version = 1
Copy the codeOptimistic locking through the way of the version number, at the last minute of update judge yourself before the data read from the database are being modified, its efficiency is higher than the pessimistic locking, because in the current thread query and the last update before this time, other threads can read the same record as usual, and update it first.
Pessimistic locking
Pessimistic locking is the opposite of optimistic locking. When the current thread queries the data to be updated, it locks the data and does not allow other threads to modify the data before its own update is complete.
By using select… “For update” tells the database “I’m going to update this data, lock it for me”.
Note: FOR UPDATE only applies to InnoDB and must take effect in a transaction. FOR UPDATE only applies to InnoDB and must take effect in a transaction. FOR UPDATE only applies to InnoDB and must take effect in a transaction. Therefore, when pessimistic lock is used, the query condition should be able to locate a certain row or several rows, and do not cause full table lock
Zero basic learning Java programming, you can join my ten years of Java learning garden, technical exchange, resource sharing, answering questions, experience sharing.