This article directory
-
background
-
Simple idempotent implementation
2.1 Database record Judgment
2.2 Solving concurrency Problems
- Universal idempotent implementation
3.1 Design Scheme
3.1.1 Universal Storage
3.1.2 Simple to use
3.1.3 Support for annotations
3.1.4 Multi-level Storage
3.1.5 Concurrent Read and write
3.1.6 Execution process
3.2 Idempotent interface
3.3 Idempotent annotations
3.4 Automatically Distinguish repeated Requests
3.5 Storage Structure
3.6 Source Code address
background
Answer group members’ questions: are there any general schemes and practices for idempotence?
What is idempotent is not explained in this article. I believe that we all know, and have encountered similar problems and have their own solutions.
Basically, idempotence in all business systems is handled separately, but it is not to say that it cannot be handled in a unified way, which requires more considerations.
Personally, I think the core business is suitable for the business side to deal with by itself. For example, there is a payment record table for order payment. An order can only be paid once, and the idempotent effect can be achieved through the payment record table.
There are also non-core businesses that have idempotent requirements. For example, network problems, try again. The user clicks multiple times. This scenario still requires a common idempotent framework to handle, which makes business development easier.
Simple idempotent implementation
Idempotent realization is not complicated, there are many schemes, first introduced based on database record scheme to achieve, and then introduced the general scheme.
Database record judgment
Take the payment scenario from the beginning of this article. The business scenario is that an order can only be paid once, so we will judge whether the order has been paid before payment, if not, the payment will be made, if the payment has been made, the payment will be successful anyway, idempotent.
This method requires an extra table to store the actions that have been done before to determine whether they have been done before.
It’s like you’re old and you’re still a single geek. This time you are worried at home ah, your mother every day to introduce you to the little sister. You dress up every weekend to meet the little sister your mom sets you up with.
Go before you have to record it, the first week of August I saw XXX, THE second week I saw YYY, if the third week let you go to see XXX, if this time you don’t like XXX, you will turn out your little book to see, this before seen, there is no need to see again, otherwise see more embarrassing ah.
Concurrency problem solving
There is no business logic problem in determining whether a payment can be made by querying the payment record. But there are problems in concurrent scenarios.
The order of 1001 initiated two payment requests, and the current two requests queried the payment record at the same time, but neither was found. Then they both started to follow the payment logic, and finally found that the same order was paid twice, which is the idempotency problem caused by concurrency.
There are many concurrent solutions, from simple ones that directly use the unique index of the database to more complicated ones that use distributed locks to lock the same resource.
For example, if we lock the order 1001, if two payment requests are initiated at the same time, only one request can obtain the lock at the same time, and the other request can fail to obtain the lock directly, or it can wait for the completion of the previous request.
If you wait for the previous request to complete and continue processing, you can check that 1001 has been paid and return that the payment was successful.
Universal idempotent implementation
In order to make people more focused on the development of business functions, I think the idempotent operation of simple scenarios can be handled by unified encapsulation. The realization of universal idempotent is introduced below.
Design scheme
General store
In general, when we do idempotent programs, we query first, and then do the corresponding operations based on the results of the query. The same resources are also locked to avoid concurrency problems.
Locking is universal. The non-universal part is determining whether the operation has been done before, so we need to have a universal store to record all operations.
Using a simple
Provide universal idempotent components, injection of the corresponding class can achieve idempotent, shield locking, record judgment and other logic.
Support for annotations
In addition to code to control idempotent, in order to make the use of more simple, but also need to provide annotations to support idempotent, users only need to add corresponding annotations on the corresponding business method, to achieve idempotent.
Multistage storage
Multi-level storage needs to be supported. For example, tier-1 storage can be realized by Redis. The advantages are high performance and suitable for 90% of scenarios. In many scenarios, you can set a certain expiration period to disable keys automatically to prevent duplicate requests within a short period of time.
Tier-2 storage supports databases such as Mysql and Mongo, and applies to scenarios where storage is permanent or has a long duration.
You can specify what is used for tier-1 storage and what is used for tier-2 storage through configuration. This scenario is well suited for implementation using the policy pattern.
Concurrent read and write
The introduction of multi-level storage inevitably involves concurrent read and write scenarios, which can be supported in two ways, sequential and concurrent.
The order is to write to tier-1 storage first, then to tier-2 storage, and the same is true for reading. The problem with this is that performance is a bit wasted.
Concurrency is the simultaneous writing and reading of multiple threads to improve performance.
Idempotent execution flow
Idempotent interface
Idempotent interface definition
Public interface DistributedIdempotent {/** * idempotent execution * @param key idempotent key * @param lockExpireTime Lock expiration time * @param FirstLevelExpireTime level-1 storage expiration time * @param secondLevelExpireTime level-2 storage expiration time * @param timeUnit storage timeUnit * @param readWriteType Read/write type * @param execute Logic to execute * @param fail Key already exists, * @return */ <T> T execute(String key, int lockExpireTime, int firstLevelExpireTime, int secondLevelExpireTime, TimeUnit timeUnit, ReadWriteTypeEnum readWriteType, Supplier<T> execute, Supplier<T> fail); }Copy the codeuse
Public String idempotentCode(String key) {return idempotentCode(String key) {return distributedIdempotent.execute(key, 10, 10, 50, TimeUnit.SECONDS, ReadWriteTypeEnum.ORDER, () -> {system.out.println () -> {system.out.println (); ); return "success"; }, () -> {system.out.println (" duplicate... ") ); return "fail"; }); }Copy the codeIdempotent annotations
Using annotations makes it much easier to use, such as in our transactions, caches and so on to simplify the logic.
Idempotent scenarios can also define common annotations to simplify usage, add annotations to business methods that need to support idempotent, and configure basic information.
IdempotentHandler is the method executed after the idempotent rule is triggered, that is, we implement the Supplier fail parameter when idempotent in code. The implementation is using ali-Sentinel current limiting, fusing after processing that logic.
In idempotent scenarios, if the execution is repeated, it usually returns the same result as normal execution.
/** * idempotentHandler = "idempotentHandler", readWriteType = ReadWriteTypeEnum.PARALLEL, SecondLevelExpireTime = 60) public void idempotent(String key) {system.out.println (" ); } public void idempotentHandler(String key, IdempotentException e) {system.out.println (key + ":idempotentHandler already executed. ..." ); }Copy the codeAutomatically distinguish duplicate requests
Code processing idempotent, need to pass idempotent Key, annotation processing idempotent, support configuration Key, support SPEL expression. Both of these are unique statements that need to be determined at the time they are used as idempotent statements.
Another idempotent scenario that is common is to prevent repeated commits or timeout retries for network problems. If the same operation is requested multiple times, you can obtain a unique ID before the operation and give it to the back end at each request, thus identifying the uniqueness of the entire request.
I currently have a function to automatically generate unique identifiers. Simply speaking, MD5 is performed according to the request information. If the MD5 value does not change, it is considered to be the same request.
The contents required for MD5 include request URL parameters, request body, and request header information. The information in the request header will be all concatenated if the user Key is not specified. If the request header userId is configured as the userId, only the userId is used.
An idempotent Key is automatically generated at the entrance to the request, and if spelKey is not specified when using idempotent annotations, the automatically generated Key is used.
Storage structure
Redis: The storage type is String. Key is idempotent Key and Value is 1 by default.
Mysql: create a record table. (Expired data needs to be cleaned regularly or stored permanently)
CREATE TABLE 'idempotent_record' (' id 'int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', `key` varchar(50) NULL DEFAULT '', `value` varchar(50) NOT NULL DEFAULT '', 'expireTime' timestamp NOT NULL COMMENT 'expireTime ',' addTime 'timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT' create time ', PRIMARY KEY (' id ') ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT=' InnoDB ';Copy the codeMongo: the same as Mysql, convert to Json format. Mongo automatically creates collections.
Code word is not easy, if you can do a triple punch, thank you!
About the author: Yin Jihuan, a simple technology lover, the author of Spring Cloud Micro-service – Full stack Technology and Case Analysis, Spring Cloud Micro-service Introduction combat and Progress, the founder of the public account Monkey World.
