Talk about idempotency in development

The concept of idempotence

The mathematical concept of idempotent

Idempotent is derived from a mathematical concept. There are two main definitions

In a unary operation, if x is any number in a set and f(x) = f(f(x)) is satisfied, then the F operation is idempotent. For example, the absolute value operation ABS (a) = ABS (ABS (a)) is idempotent.

If, in a binary operation,x is any number in a set, f(x,x) = x is said to be idempotent if both parameters of the operation are x. For example, the function Max (x,x) = x is said to be idempotent.

Idempotent concepts in development

The concept of idempotence may be abstract in mathematics, but idempotence is extremely important in development. Simply put, for the same system, under the same conditions, a request and repeated multiple requests have the same impact on resources, the operation is said to be idempotent. For example, if an interface is idempotent, the effect must be the same when the same condition is passed in.

In particular, when RPC or Restful interfaces in a distributed system call each other, it is easy to find exceptions during invocation due to network errors and other reasons and retry is needed. In this case, the idemidemality of interfaces must be ensured; otherwise, the retry result will be different from the first call result. If one of the interface’s call chains A->B->C->D->E returns the wrong result after an abnormal retry at D->E, A,B, and C will also be affected, which could be disastrous.

Some common examples of idempotent requirements in life:

  1. If the same user likes the same article, even if the same person has been single for 30 years, they can only give the same article +1 like
  2. In wechat payment, money can only be deducted once for an order, and money should only be deducted once for repayment due to network problems or bugs

Idempotence and concurrency security

While looking through the web, I’ve come across many articles that confuse idempotence with concurrency security. Idempotence is a promise, not an implementation, of the system interface, that the result of the same operation will be the same many times. The problem of concurrency safety is that when multiple threads operate on the same resource at the same time, the result is incorrect because of the operation sequence.

These two problems are actually completely separate. For example, even if you keep submitting and paying for the same order, if you deduct money more than once, it means that the operation is not idempotent. If multiple orders are paid at the same time and the final amount deducted is not the sum of these orders, it indicates that the operation has concurrency security problems. Therefore, idempotence and concurrency security are completely two dimensional problems, which should be discussed and solved separately.

I have seen in some articles on idempotency that the solutions given in “pessimistic locking” and “optimistic locking” are good solutions to concurrency problems, but should not be solutions to idempotency problems, especially pessimistic locking that prevents multiple threads from modifying a resource at the same time. The version number mechanism for optimistic locking can be idempotent with tokens or state tokens as version numbers (token and state tokens are explained below), which makes sense.

Therefore, idempotency is different from concurrency security. In this article, only idempotency is discussed, not concurrency security

Http protocol and idempotency

If operations are classified according to functions, there are four types of operations: add, Delete, modify, and check. In HTTP, there are four types of operations: Get, Post, Put, and Delete.

Query operation (Get)

The Get method is used to Get resources and should not change system resources, so it is idempotent. Notice that the idempotent enhancement here is now the change to the system resource, not the result of the data returned, even if the result is different but the operation itself has no side effects, so idempotent.

Delete operation (Delete)

The Delete method is used to Delete resources. Although the system resources are changed, the first and NTH Delete operations have the same effect on the system, so they are idempotent. Like to delete a resource id is 1234, could be the first call will be deleted, and all calls behind time has not this id due to system resources, but for the first time and the back of the operation of the system function is the same, so this is idempotent, the caller can invoke the interface don’t have to worry about mistakes many times.

Modify Operation (Put)

The modification operation may or may not be idempotent. If the resource is fixed, such as changing the amount in the account to $1000, it is idempotent no matter how many times it is called. If the resource is not fixed, for example, the amount in the account is reduced by 50 yuan, the result of one call and many calls must be different, this time is not idempotent. The idea of idempotent in a modification operation is discussed below.

The 2019-08-13 revision

The definition of the Put protocol is incorrect. The Put operation must be idempotent. If the Put protocol is declared, the interface is declared to be idempotent. Therefore, for example, the operation of reducing the amount of the account by 50 yuan is not allowed in the Put protocol. Only the operation of changing the amount of the account to 1000 yuan can be performed

Reference: restfulapi.net/idempotent-…

New Operation (Post)

Post is not an idempotent operation by nature. It is defined in HTTP as follows:

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line.

(www.w3.org/Protocols/r)…

In its definition, the Post request is used to create a new resource, which means that each call creates a new resource in the system, so the operation is not meant to be idempotent. In this case, idempotent must be implemented in business, as discussed below.

A scheme to achieve idempotency

The idempotency mentioned above is still a comparative theory, and the idempotency design scheme is discussed in combination with some common practical business scenarios.

Go to the heavy table

Use the characteristics of database to achieve idempotent. Typically, a unique index is built on a table, so that once a certain data is built, subsequent operations cannot write it successfully.

A common business is the blog system “like” function, a user to a blog after the “like”, the user ID and blog ID binding, the subsequent user to agree with a blog cannot be inserted. Or in the financial system, to create financial accounts for users, a user can not have multiple accounts, add a unique index to the account table to store the user ID, so that users can only have one account even if repeated operations.

State identification

State identification is a common idempotent design method. The main idea is to ensure that each process in the business can only be executed in the corresponding state through the change of state identification. If the identification has entered the next state, the operation in the previous state is not allowed to change the state, ensuring the idempotency of the business.

Status identification is often used in scenarios where a long service process requires a lot of data to be modified. The most classic example is the order system, if an order goes through the four steps of order creation -> order payment -> cancellation -> account calculation -> notification to the merchant. Then it is possible to deduct the corresponding balance from the account after the payment of an order and consume the corresponding coupon. However, an error message is returned due to network reasons. At this time, the system will try again to calculate the account, causing data errors.

Therefore, in order to ensure the idempotency of the entire order process, a status identifier can be added to the order information, and the corresponding status identifier can be modified once a step is completed. For example, after the order payment is successful, the order will be marked as changed as payment is successful. Now call the order payment or cancel interface again, the order status will be judged first. If the order has been paid or canceled, the order will not be paid again.

Token mechanism

Token mechanism should be the most widely applicable idempotent design scheme, and the specific implementation methods are also very diverse. But the core idea is that each operation generates a unique Token Token that the server uses to ensure that the same operation will not be performed twice. In addition to being a literal unique string, the Token can also be a combination of multiple tokens (such as the status Token mentioned above), or even a time period Token, etc.

For example, when Posting a new Post in a forum, which is a typical Post addition operation, how do you prevent users from clicking submit multiple times to create the same Post? Users can submit with a unique Token, and the server will not allow the submission as long as the Token exists, thus ensuring idempotency.

The above example is easy to understand, but the business is simple. Since the Token mechanism is widely applicable, the requirements to be paid attention to in its design will vary from business to business.

When and how are tokens generated? This is the core of the mechanism. Take the forum system above as an example. If you generate tokens only when the user submits a post, then the user will generate new tokens every time the user clicks on the post and the submission will be successful, which is not idempotent. The Token must be generated before the user submits the content, for example, when entering the editing page. When the user submits the content, the user submits the content with the Token. No matter how many times the user submits the same page, it can succeed at most once. Therefore, the Token generation must be timed to ensure that the operation has the same effect multiple times. Using the Token mechanism requires a good understanding of business processes.

conclusion

Idempotence is a very common and important requirement in development. In particular, finance, payment and other industries are more stringent on its requirements, both good performance and strict idempotency. In addition to mastering its concept, understanding its own business requirements is the key to realizing idempotent functions. Every node detail must be handled well. Once a place is not well designed, the final result may still fail to meet the requirements.