Official account: Java Xiaokaxiu, website: Javaxks.com

Author: Super Xiaoding, link: mydlq.club/article/94/

What is idempotence

Idempotence is a mathematical and computer concept in which an operation of one element is idempotence, acting on any element twice will have the same effect as acting on it once.

[in computer programming, an idempotent operation is characteristic of the impact of its arbitrary executed multiple times are the same as the influence of a single execution. Idempotent function or idempotent method refers to repeat, can use the same parameters and function to get the same results. These functions will not affect the system state, also need not worry about repeat will cause changes to the system.]

What is interface idempotence

In HTTP/1.1, idempotency is defined. It describes that one or more requests to a resource should have the same effect on the resource itself (except for network timeouts), that is, the first request has side effects on the resource, but subsequent requests do not have side effects on the resource.

The side effect here is that you don’t spoil the results or have unexpected results. That is, any number of executions will have the same impact on the resource itself as one execution.

3. Why is idempotence needed

When the interface is called, it normally returns a message and does not commit again. However, problems may occur when the following situations occur, such as:

• Front-end repeated submission of forms: When filling in some forms, the user completes the submission. In many cases, due to network fluctuations, the user does not respond to the successful submission in time. As a result, the user thinks that the submission is not successful, and then clicks the submit button all the time.
• Malicious user brushing: For example, when the user voting function is implemented, if a user repeatedly votes for a user, the interface will receive the voting information repeatedly submitted by the user, and the voting result is seriously inconsistent with the facts.
• Interface timeout repeated submission: In most cases, the HTTP client tool enables the timeout retry mechanism by default. When a third party invokes an interface, a retry mechanism is added to prevent request failures caused by network timeout. As a result, one request is submitted multiple times.
• Repeated consumption of messages: When using MQ messaging middleware, repeated consumption occurs if the messaging middleware fails to submit consumption information in a timely manner.

The biggest advantage of using idempotent is that the interface guarantees any idempotent operation, avoiding unknown problems with the system caused by retries and so on.

The influence of idempotence on the system

Idempotency is to simplify the logical processing of the client side and can place repeated submission and other operations, but it increases the logical complexity and cost of the server side, mainly:

• The function of parallel execution is changed to serial execution, which reduces the execution efficiency.
• Complicate business functions by adding additional control idempotent business logic;

Therefore, it is necessary to consider whether to introduce idempotency when using the interface. According to the actual service scenario, except for special service requirements, interface idempotency is generally not needed.

5. Idempotency of Restful apis

There are idempotent lines and methods that cannot guarantee idempotent lines in several popular Restful HTTP interface methods, as follows:

• The square root is idempotent
• X is not idempotent
• – It may or may not be idempotent, depending on the actual business logic

Method type	Whether the power etc.	describe
Get	Square root	The Get method is used to Get resources. It generally does not and should not change system resources and is idempotent.
Post	x	The Post method is typically used to create new resources. It adds data every time it executes, so it’s not idempotent.
Put	–	The Put method is generally used to modify resources. This operation determines whether it is idempotent by case. In the update operation, it can also keep idempotent by updating directly according to a certain value. However, updates that perform a summation operation are non-idempotent.
Delete	–	The Delete method is generally used to Delete resources. This operation determines whether the data is idempotent. If the data is deleted according to the unique value, the effect of deleting the same data is the same. Note, however, that a delete with a query condition is not necessarily idempotent. For example, after a batch of data is deleted based on a condition, a new data that meets the condition is deleted again.

How to realize idempotence

Scheme 1: Unique primary key of the database

Scheme described

The realization of unique primary key of database mainly uses the unique constraint of primary key in database. Generally speaking, unique primary key is more suitable for idempotency of “insert”, which can ensure that only one record with the unique primary key can exist in a table.

When using a database unique primary key to achieve idempotent properties, it is important to note that the primary key is generally not augmented in the database, but is used as the primary key by a distributed ID (see the article designing distributed ids in Java) to ensure global uniqueness of ids in a distributed environment.

Applicable operation:

• The insert
• Delete operation

Restrictions on use:

• Need to generate globally unique primary key ID;

Main process:

Main process:

• ① The client performs the creation request and invokes the server interface.
• ② The server executes the business logic, generates a distributed ID, acts as the primary key of the data to be inserted, and then executes the data insertion operation and the corresponding SQL statement.
• ③ The server inserts the data into the database. If the data is inserted successfully, the interface is not invoked repeatedly. If a duplicate primary key exception is thrown, the record already exists in the database and an error message is returned to the client.

Solution 2: Optimistic database lock

Scheme Description:

Optimistic locking [database normally only applies to implement the “update” process, we can advance the corresponding data in the table to add a field, as the current version identification data. So every time the data update of the database table, will be the version identification as a condition, value of the version identification for the last time to update the data values.]

Applicable operation:

• The update operation

Restrictions on use:

• Additional fields need to be added in the corresponding service table of the database;

Description Example:

For example, the following data table exists:

id	name	price
1	Millet mobile phone	1000
2	Apple mobile phone	2500
3	Huawei mobile phones	1600

To prevent repeated updates each time an update is performed, we usually add a version field to record the current version of the record, so that the value is added during the update, so that the update operation can determine that the information under the corresponding version has been updated.

id	name	price	version
1	Millet mobile phone	1000	10
2	Apple mobile phone	2500	21
3	Huawei mobile phones	1600	5

If version=5 is updated, you must specify the version number to be updated each time.

UPDATE my_table SET price=price+50,version=version+1 WHERE id=1 AND version=5
Copy the code

Select * from SQL WHERE id=1 AND version=5; select * from SQL WHERE id=1 AND version=5; This preserves the idempotent nature of the update, and multiple updates have no effect on the results.

Solution 3: Anti-duplicate Token

Scheme Description:

[In case of continuous client clicks or timeout retries by the caller, such as order submission, this operation can be implemented by Token mechanism to prevent repeated submission. In simple terms, the caller first requests a global ID (Token) from the backend when invoking the interface. The backend requests the global ID together with the request (the Token is best placed in Headers). The backend needs to treat the Token as a Key. User information is sent to Redis as Value to verify Key Value content. If Key exists and Value matches, delete command is executed, and then the following service logic is normally executed. If there is no corresponding Key or Value mismatch, repeat error message is returned, so as to ensure idempotency operation.

Applicable operation:

• The insert
• The update operation
• Delete operation

Restrictions on use:

• Need to generate globally unique Token string;
• Need to use the third-party component Redis for data validation;

Main process:

• ① The server provides an interface to obtain a Token. The Token can be a serial number, distributed ID or UUID string.
• ② The client invokes the interface to obtain the Token. In this case, the server generates a Token string.
• ③ Then the string is stored in the Redis database and the Token is used as the Redis key (note the expiration time).
• ④ Return the Token to the client. After the client gets it, it should be saved in the hidden field of the form.
• ⑤ When the client submits the form, it stores the Token into the Headers, and then carries the Headers with the service request.
• ⑥ After receiving the request, the server obtains the Token from the Headers and checks whether the key exists in Redis based on the Token.
• ⑦ The server determines whether the key exists in Redis. If the key exists, it will delete it and then execute the business logic normally. If not, throw an exception and return a duplicate error message.

Note that performing Redis lookups and deletions in concurrent cases requires atomicity, otherwise idempotency may not be guaranteed in concurrent cases. It can be implemented using distributed locks or Lua expressions to unregister queries and deletes.

Scheme 4. Downstream transmission of unique serial numbers

Scheme Description:

The so-called request sequence number, in fact, is each request to the server with a unique serial number in a short period of time, the serial number can be an ordered ID, or an order number, generally generated by the downstream, when calling the upstream server interface to attach the serial number and ID for authentication.

When the upstream server receives the request information, it combines the serial number with the downstream authentication ID to form a Key for operating Redis, and then queries Redis to see if there is a Key value pair corresponding to the Key. According to the result:

• If yes, it indicates that the downstream request of the sequence number has been processed. In this case, you can directly respond to the error message of repeated request.
• If not, the Key is used as the Key of Redis, the downstream Key information is used as the stored value (for example, some business logic information transmitted by downstream merchants), the Key value pair is stored in Redis, and then the corresponding business logic can be executed normally.

Applicable operation:

• The insert
• The update operation
• Delete operation

Restrictions on use:

• Require a third party to deliver a unique serial number;
• Need to use the third-party component Redis for data validation;

Main process:

Main steps:

• ① The downstream service generates the distributed ID as the serial number, and then performs the request to invoke the upstream interface, along with the “unique serial number” and the requested “authentication credential ID”.
• ② The upstream service carries out security effect test to detect whether there are “serial number” and “credential ID” in the parameters transmitted downstream.
• ③ The upstream service checks whether there is a Key consisting of the serial number and authentication ID in Redis. If there is a Key, the upstream service throws a repeated exception message and responds to the downstream error message. If it does not exist, the combination of the serial number and authentication ID is used as the Key, and the downstream Key information is used as the Value, which is then stored in Redis, and the incoming service logic is normally executed.

When inserting data into Redis in the previous step, be sure to set the expiration time. This ensures that if the interface is repeatedly called within this time range, it will be able to make a judgment call. If you do not set the expiration time, it is likely that unlimited data will be stored in Redis, causing Redis to not work properly.

7. Example of interface idempotent implementation

Here we use the duplicate proof Token scheme, which guarantees idempotency in different request actions. The implementation logic can be seen in the “duplicate proof Token” scheme above, and the code to implement this logic is written below.

Maven introduces dependencies

Maven tools are used to manage dependencies, and SpringBoot, Redis, and Lombok dependencies are introduced in pom.xml.

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion > 4.0.0 < / modelVersion > < the parent > < groupId > org. Springframework. Boot < / groupId > The < artifactId > spring - the boot - starter - parent < / artifactId > < version > 2.3.4. RELEASE < / version > < / parent > < the groupId > mydlq. Club < / groupId > < artifactId > springboot - idempotent - token < / artifactId > < version > 0.0.1 < / version > <name>springboot-idempotent-token</name> <description>Idempotent Demo</description> <properties> < Java version > 1.8 < / Java version > < / properties > < dependencies > <! --springboot web--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <! --springboot data redis--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.apache.commons</groupId>  <artifactId>commons-pool2</artifactId> </dependency> <! --lombok--> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>Copy the code

2. Set parameters for connecting to Redis

Configure the parameters for connecting to Redis in the Application configuration file. The basics of Spring Boot will not be introduced, but the latest tutorial is recommended to see the following tutorial.

As follows:

Spring: redis: SSL: false host: 127.0.0.1 Port: 6379 database: 0 timeout: 1000 Password: lettuce: pool: max-active: 100 max-wait: -1 min-idle: 0 max-idle: 20Copy the code

3. Create and verify the Token utility class

Create a Service class for manipulating tokens, which contains the Token creation and verification methods:

• Token creation method: Use the UUID tool to create a Token string, set idempotent_token: + Token string as the Key, use user information as the Value, and save the information in Redis.
• Token authentication method: Receives Token string parameters, adds Key prefix to form Key, passes in value value, executes Lua expression (Lua expression ensures atomicity of command execution) to search for corresponding keys and delete them. After the command is executed, verify the returned result. If the result is not null or zero, the verification succeeds. Otherwise, the command fails.

import java.util.Arrays; import java.util.UUID; import java.util.concurrent.TimeUnit; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.data.redis.core.script.DefaultRedisScript; import org.springframework.data.redis.core.script.RedisScript; import org.springframework.stereotype.Service; @Slf4j @Service public class TokenUtilService { @Autowired private StringRedisTemplate redisTemplate; /** * private static final String IDEMPOTENT_TOKEN_PREFIX = "idempotent_token:"; /** * Create Token to store in Redis, Return the Token * * @param Value Value used for authentication * @return Generated Token String */ public String generateToken(String Value) {// Instantiate an ID utility object String token = uuID.randomuuid ().toString(); String Key = IDEMPOTENT_TOKEN_PREFIX + token; / / store Token to Redis, and set the expiration time is 5 minute redisTemplate opsForValue (). The set (key, value, 5, TimeUnit. MINUTES); // Return Token return Token; } /** ** Verify Token correctness ** @param Token Token string * @param value value Auxiliary verification information stored in Redis * @return verification result */ public Boolean ValidToken (String token, String value) {// Set Lua script, where KEYS[1] is key, KEYS[2] is value String script = "if redis. Call ('get', KEYS[1]) == KEYS[2] then return redis. Call ('del', KEYS[2]) KEYS[1]) else return 0 end"; RedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class); String Key = IDEMPOTENT_TOKEN_PREFIX + token; Long result = redistemplate. execute(redisScript, arrays.aslist (key, value)); If (result! = 0) // if (result! = 0) = null && result ! = 0, l) {log. The info (" authentication token = {}, key = {}, value = {} "success, token, key, value); return true; } the info (" authentication token = {}, key = {}, value = {} failure ", token, key, value); return false; }}Copy the code

4. Create the Controller class for the test

Create a Controller class for testing, which contains the interface to obtain Token and idempotency of the test interface, as follows:

import lombok.extern.slf4j.Slf4j; import mydlq.club.example.service.TokenUtilService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.*; @Slf4j @RestController public class TokenController { @Autowired private TokenUtilService tokenService; /** * @return Token String */ @getMapping ("/ Token ") public String getToken() { The content stored here is only an example. It serves as an auxiliary verification to make its verification logic more secure. For example, the user information is stored here for the purpose of: // -1), use "token" to verify whether the corresponding Key exists in Redis // -2), use "user information "to verify whether the Redis Value matches. String userInfo = "mydlq"; / / access Token string, and returns the return tokenService. GenerateToken (the userInfo); } /** * Interface idempotent test interface ** @param Token Idempotent token String * @RETURN Execution result */ @postMapping ("/test") public String Test (@requestheader (value = "token") String token) {// Obtain user information (here using analog data) String userInfo = "mydlq"; Boolean result = tokenservice.validToken (Token, userInfo); Return result? "Normal call" : "repeated call "; }}Copy the code

5. Create the SpringBoot boot class

Create a boot class to start the SpringBoot application. Basic tutorial is not introduced, it is recommended to see the following tutorial, very full.

import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); }}Copy the code

6. Write test classes to test

Write a test class to test whether the same interface can be accessed multiple times.

import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.http.MediaType; import org.springframework.test.context.junit4.SpringRunner; import org.springframework.test.web.servlet.MockMvc; import org.springframework.test.web.servlet.request.MockMvcRequestBuilders; import org.springframework.test.web.servlet.setup.MockMvcBuilders; import org.springframework.web.context.WebApplicationContext; @Slf4j @SpringBootTest @RunWith(SpringRunner.class) public class IdempotenceTest { @Autowired private WebApplicationContext webApplicationContext; @test public void interfaceIdempotenceTest() throws Exception {// Initialize MockMvc MockMvc MockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); / / call the access Token interface String Token = mockMvc. Perform (MockMvcRequestBuilders. Get ("/Token "). The accept (MediaType. TEXT_HTML)) .andReturn() .getResponse().getContentAsString(); Log.info (" Token string: {}", Token); For (int I = 1; i <= 5; I ++) {log.info(" call test interface {} time ", I); / / call the validation interfaces and print the result String result = mockMvc. Perform (MockMvcRequestBuilders. Post ("/test "). The header (" token ", token) .accept(MediaType.TEXT_HTML)) .andReturn().getResponse().getContentAsString(); log.info(result); If (I == 0) {assert.assertequals (result, "normal call "); } else {assert. assertEquals(result, "repeated call "); }}}}Copy the code

The following information is displayed:

[main] IdempotenceTest: Obtained Token string: 980EA707-CE2E-456E-A059-0A03332110b4 [main] IdempotenceTest: Obtained Token string: 980EA707-CE2E-456E-A059-0A03332110b4 The first call to test interface [main] IdempotenceTest: normal call to test interface [main] IdempotenceTest: Second call to test interface [main] IdempotenceTest: Call [main] IdempotenceTest repeatedly: call the test interface for the third time [main] IdempotenceTest: Call test interface [main] IdempotenceTest: repeat call [main] IdempotenceTest: repeat call 5th call test interface [main] IdempotenceTest: repeat callCopy the code

Eight, the final conclusion

Idempotency is a very common and important requirement in development, especially in payment, order and other money-related services, ensuring interface idempotency is especially important. In actual development, we need to flexibly choose the realization mode of idempotency for different business scenarios:

• For orders with unique primary keys, the “unique primary key scheme” can be used.
• For update scenario operations, such as updating order status, it is simpler to implement an “optimistic locking scheme”.
• For upstream and downstream, where downstream requests upstream, the upstream service can use the “downstream pass unique sequence number scheme” more reasonable.
• Similar to the scenario of repeated submission, repeated ordering, and no unique ID number, the “Anti-duplicate Token solution” can be implemented faster through the combination of Token and Redis.

The above is just some suggestions. Once again, to realize idempotence, it is necessary to understand its own business needs first and realize it according to the business logic, so as to deal with every node details and improve the overall business process design, so as to better ensure the normal operation of the system. To conclude this post with a brief summary:

Package name	Applicable method	Implementation complexity	Solution disadvantages
Database unique primary key	Insert operation Delete operation	simple	– Can only be used for insert operations; – Applies only to scenarios with unique primary keys.
Optimistic database locking	The update operation	simple	– Can only be used for update operations; – Additional fields need to be added to the table;
Request serial number	Insert operation Update operation Delete operation	simple	– The need to ensure that the downstream generation of unique serial numbers; – Redis third party is required to store requested serial numbers;
Anti-duplicate Token	Insert operation Update operation Delete operation	moderate	– The Token string generated by Redis third-party storage is required;