1. Why idempotent

In distributed scenarios, it is extremely difficult to implement strongly consistent protocols among multiple service systems. One of the simplest and most achievable assumptions is to ensure final consistency, which requires that the server respond to a repeated request with the same response without the side effects of persistent data (that is, multiple operations and the results of a single operation need to be consistent from a business perspective).

When an API has idempotent capabilities, the caller can safely retry the call. This is consistent with our common assumption.

Providing idempotent capabilities is what the service provider needs to do, so this article is written from the perspective of the service provider, meaning that “we” below usually refers to the service provider.

2. How to make idempotent judgment

So how do we make idempotent judgments about a request? First, we need a way to distinguish what constitutes a “once” request, and second, we need different ways to determine how data is persisted in the API implementation logic.

2.1 Request Unique Identifier

Typically, our API needs to add attributes such as bizId that identify the uniqueness of the request, which the call originator uses to identify what the business considers to be a unique business requirement. It is important to note that this is a “business unique identifier”, not a technology unique identifier, which is fundamentally different.

Fact 1: Most of the time, however, the originator of the call probably does not know how to generate a valid unique identifier for the request, which requires more communication during the business docking process.

2.2 How long to be idempotent

Being idempotent means that some data must be persisted, but no data can be persisted forever and must have an expiration date.

For most requests, the recommended idempotent validity period is three months. Some special scenes can even last a year. But it hardly needs much longer.

The expiration date is an explicit contract, which means that we can do some data governance work on persistent data on a regular basis, and requests that exceed the expiration date will basically fail idempotent, with the consequence of “doing the same thing again a few months later.”

Type 2.3 to write

If our business logic is to write something to a persistent store, then it is best to add a unique key, which usually consists of user_id, operation type, biz_id. Operation type is the business operation type defined by our system design, plus it is to avoid interference between different operation types. User_id is added because normally we have separate libraries and tables.

For scenarios where data is updated as well as written, insert needs to come first, otherwise idempotent is unlikely to work. For example, in the following inventory operation scenario, if the two SQL statements are written upside down, in the inventory sold out scenario is not idempotent:

Insert inventory reduction flow; Update inventory set inventory available = inventory available -1 where inventory id=? And inventory available >0;Copy the code

In a unique key conflict, we need to catch it, which is most likely idempotent. Why is it a high probability and not a certainty? This is because of the earlier “Fact 1”. The caller may have misused the request unique number and may have changed some of the core parameters when retrying.

So, we need to do the following when a unique key conflict occurs:

1. Query previously written data based on unique keys.
2. Check key information. For example, if the API we provide is to issue coupons, we need to verify at least some of the information: receiving user Id, coupon template, coupon size, expiration date, etc. If the key information is inconsistent, an error code similar to DUPLICATE_BUT_DIFFERENT_REQUEST is returned. Returns the Id of the ticket that was successfully sent when verification passes. This is important, as returning idempotent success without checking key information is the root cause of failure in most scenarios.

2.4 more new

The biggest challenge with newer models is that we don’t have a place to store multiple updates to a single piece of data, so most of the time we need to use a state machine to figure out if an update has occurred. More complex cases may require us to add dedicated idempotent tables.

2.4.1 Using state Machines

Most business data processing is stateful, such as trade orders. The preferred method is to use the sequence of the state machine to determine whether an update has already been done. This is not easy, however, and we need to be very careful to look at the various business rules and restrictions, and in most cases there is additional information that is updated in addition to the status update, and we need to check that the additional information is updated as requested.

2.4.2 Adding an idempotent table

By adding idempotent tables, the newer type is converted to the write type described in Section 2.3. Note that idempotent tables need to be in the same transaction as the currently updated table, or they will be invalid.

Idempotent tables need to add fields indicating when data was written/updated, which makes it easier to clean up expired data on a regular basis.

Type 2.5 to delete

It is recommended to use logical deletes on storage rather than physical deletes so that we have a way to determine if a delete has already been done. Deletion usually means that the data is in its final state, so it is also the best to deal with idempotent; If a business’s design deletion is not the end state of the data, then care needs to be taken, as this violates general design principles.

3. Summary

Idempotence is arguably the cornerstone of distributed applications, and as you can see, implementing it is not as simple as most people think at first. Doing it well requires us to design and think in terms of business semantics.

The text/wood

Pay attention to technology, do the most fashionable technology