01 What is idempotent
Idempotent is a mathematical and computer concept commonly found in abstract algebra.
In program development, an operation is idempotent if any number of executions have the same effect on the system as a single execution. Idempotent functions (idempotent methods) are functions that are executed repeatedly with the same parameters and obtain the same results. These functions do not affect system state, nor do they have to worry about system changes caused by repeated execution.
02 How can I determine whether an operation is idempotent
Idempotent in HTTP/1.1
A request method is considered “idempotent” if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.
Datatracker.ietf.org/doc/html/rf…
In HTTP/1.1, each method has its own specific semantics, and the nature of the method determines its idempotent nature. Such as:
-
GET /resource/page HTTP/1.1 This is an idempotent operation. The results of one and multiple requests are the same.
-
DELETE /resource/ DELETE HTTP/1.1 Is an idempotent operation. Once a resource is deleted, subsequent operations have the same impact on the system, even though the return values of multiple operations may be different.
-
PUT /resource/ PUT HTTP/1.1 Is an idempotent operation. If a resource does not exist, it is added or updated. Even if a resource is added or updated for several times, it is added or updated to a known value.
-
POST /resource/add HTTP/1.1 is a non-idempotent operation. Each request will generate a new record on the server, which will cause side effects on the system.
Methods semantics and their specifications in HTTP/1.1 can be summarized as follows:
-
Common idempotent methods: GET, HEAD, PUT, DELETE, OPTIONS, TRACE
-
Common non-idempotent methods: POST,PATCH, CONNECT
The general conclusion of idempotent
Through the above case, we can know the evaluation basis of idempotent: whether any multiple executions of the current operation will have different impacts on the business system, if they have different impacts, the operation is non-idempotent, and we need to consider how to ensure the safety and idempotent of the interface. For example, order creation, payment callback and refund interfaces in common payment services need to consider their idempotency.
03 Causes of common idempotent problems
Conditions under which idempotent problems arise
-
The operation (interface) itself has idempotent problems;
-
The operation may be performed multiple times because the system may retry.
Common idempotent causes
-
Retry mechanism of network fluctuation: retry mechanism of middleware triggered by network instability;
-
Active/passive client retry: users actively retry or malicious users flush interfaces.
-
Retry mechanism of scheduled task: A scheduled task executes an operation multiple times.
-
Repeated message consumption: There may be multiple duplicate messages in the message queue, resulting in multiple execution of business operations.
04 Idempotent solution
Through the analysis of the cause of idempotent problem, we know that the idea of solving idempotent problem is to re-call link. Based on this, the task of de-weighting can be divided into client de-weighting and server de-weighting.
Client solution
-
The button can be clicked only once. The client disables the button and loading mask after the user clicks the button.
-
Use Post/Redirect/Get: A Web design pattern that prevents forms from resubmitting data.
The interception and deduplication of the client can only handle normal users, but the server still needs to participate in malicious requests.
Server solution
Solution 1: Database unique index feature
For simple business scenarios, such as newly inserted services, you can use the primary key index or unique index feature of a database to add indexes to unique fields of a service. Subsequent requests with the same uniqueId will fail to insert. The diagram below:
However, for some complex business scenarios, the transaction of one request is more complicated and the database needs to be operated for many times, it is necessary to design anti-repeat table, its principle is also to use the unique index feature of the database. The diagram below:
Step description:
-
The caller carries a uniqueId request, and the Service first inserts data into the anti-replay table.
-
If the insert is successful, the service operation is performed.
-
If the insert fails, the request is considered to be repeated and has been processed.
Note: uniqueId can be the unique business id related to the business, such as the order number.
Solution 2: Token mechanism
The Service generates a token. When a client (caller) requests a token, the Service determines whether the token exists and is valid. The diagram below:
Step description:
-
The client obtains the token before invoking it. The server generates a globally unique ID as the token and stores the token in the cache.
-
The client formally invokes with the token obtained in the first step.
-
The server checks whether the token exists in the cache and verifies it. If the token exists and the verification is successful, services are performed. After the service is successfully executed, the token corresponding to the request is deleted from the cache.
-
If the token verification fails, that is, no corresponding token exists in the cache, the operation is repeated and an exception is thrown.
Note: The third step is to ensure atomicity of checking tokens, service operations and clearing tokens in the cache. You can lock it here.
Description: Global unique ID generation policy: UUID; Redis atomic increment command implementation; Snowflake algorithm -Snowflake; Baidu’s UID-generator; Meituan’s Leaf.
Scheme 3: Unique serial number + cache (standalone/distributed cache)
For multiple requests for a transaction, design a globally unique serial number (uniqueSeqNo) (or a business ID that already has similar attributes in the business system). This serial number will be cached by the server when the service is first called. When the subsequent repeated requests (same serial number) come, repeatability verification is performed. The diagram below:
Step description:
-
The caller passes the transaction’s sequence number uniqueSeqNo;
-
The service provider (server) checks whether uniqueSeqNo exists. If yes, it considers that the request is repeated and throws an exception.
-
If no, service operations are performed. Caches information such as uniqueSeqNo after successful service operations.
Note: The judgment, business execution and cache operation of the third step should be atomic. You can lock it.
Note: uniqueSeqNo can be designed as a source + business Id combination. Source Identifies information such as the source of the caller.
Scheme 4: Service status verification + lock (single-node lock/distributed lock)
In business, judgment is made according to the uniqueness of business ID and the result of business processing, but the logic of this part of judgment needs to consider atomicity. Otherwise it will cause idempotent failure due to concurrency issues. One way to solve the concurrency problem is to lock, depending on the current service environment to choose a stand-alone or distributed lock.
Scheme 5: buffer queue
The request is placed in a buffered queue, which is then processed by asynchronous threads and filtered by unique identifiers in the request. The diagram below:
05 summary
The idempotent problem is a relatively simple one. The core operations of idempotent processing are two things: reweighting and concurrent problem solving with reweighting. Select an appropriate solution based on actual service scenarios. It is understood that it can also be realized through database optimistic lock, state machine and other ways, which will not be described here.
I recommend token-based solutions, anti-repeat table designs, and solutions based on business state + locks.
Welcome to wechat public number, find more dry goods: OneByte
Phase to recommend
Rambling on about distributed locks
Redis implementation of distributed lock
ZooKeeper implementation of distributed lock
Deep thought: cache breakdown (concurrent), cache penetration, and cache avalanche
Bloom Filter