Business background

Today, I will talk about the interface idempotence problem of online system, and how to guarantee the interface idempotence through distributed lock. At the same time, I will share with you the accumulation of some production practice experience in realizing interface idempotence based on distributed lock. First of all, what would happen if the core interface of our online system didn’t have idempotence guarantee? Actually very simple, if you have a system, he has an interface, this interface to accept requests when assumption in inserting a data in the database, a normal user to make a request for this interface should be only one data, the result could one day you will find more than the user through the interface insert data, as shown in figure 1:

Figure 1

The first version of the anti-duplicate code

** So why is this happening? * * we generally this kind of system interface, but anyone who wrote a little better, will participate in the interface code, code is there will be a judgment, whether you want to write the data currently exist, if if he does not exist, would be to insert, but if he will not allow you to repeat insert would exist, that heavy code as shown below:

public void business(Request request) {   
    // select * from db; // select * from DB
    Data data = findData(request);   
    
    // if this item already exists in db, then return it directly
    if(data ! =null) {
     return;   
    }   
    
    // if this item does not exist in db, then insert logic will be performed
    insertData(request); 
}
Copy the code

Combined with the anti-repeat logic of the above code, we can see the operation logic shown in Figure 2 below. Before inserting data, the data must be queried according to the request parameters. If the query is found, the data will be directly returned at this time without repeated insertion.

Figure 2

If you use the same request parameters to insert data repeatedly, you should not insert data repeatedly. This is certainly the case, but there is always an exception to the well-known instantaneous retry + multi-threaded concurrency problem.

Analysis of instantaneous retry + multi-thread concurrency

Below we explain to you, in the above code anti-repeat logic, if in a short period of time the user with the same request parameters repeatedly initiated two requests, why will penetrate the anti-repeat logic, insert two same data in the database. Let’s brace up and take a closer look at this process. Firstly, users may initiate two requests with exactly the same parameters in a flash for various reasons, such as over-excitement, hand shaking, or network convulsions, as shown in Figure 3 below:

Figure 3

In fact, each request is handled by an independent thread, regardless of whether you use the Controller interface provided by Tomcat deployment or the RPC interface provided by Dubbo, as shown in Figure 4 below:

Figure 4.

Figure 5

Figure 6.

Database unique index implements idempotent

For this interface problem such as power, relatively simple a solution that is based on we rely on the database to realize power, etc., that is to say, to implement it with the only index database, if we are based on one or more business field in the request form a unique index, then in fact you want to insert the duplicate data into the database with the same parameters, That is impossible, because the database level will prevent you from inserting, and the unique index will ensure this. If you try to insert again, it will throw an exception, as shown in 7 below:

Figure 7.

Distributed locks are idempotent

But one of the things that happens is that you don’t always have to rely on the database’s unique index to achieve this idemidemality, because you can rely on other service interfaces in your business logic besides the database, or on elasticSearch, Redis, etc. You can also rely on data from multiple tables in the database. You can’t create a unique index for each table to ensure idempotency. So for interfaces with complex business logic, to ensure idempotency, a key component is often introduced, and that is distributed lock. Distributed lock means that you rely on some external system to add a lock, and then you can release the lock later. Now the more common distributed lock implementation is mainly rely on Redis and ZooKeeper to achieve these two, we here to redis distributed lock to illustrate.

Redis distributed lock code: Redisson distributed lock code: Redisson distributed lock code

Put simple, we can at the entrance of the interface code, based on the redis plus a distributed lock, this time only one thread can be successful lock, lock, after which a thread can go to query the data exists, if not, you can insert a data into, and then release the lock, in the process, The other thread cannot acquire the Redis distributed lock, so it has to wait, as shown in Figure 8 below:

Figure 8.

Figure 9.

conclusion

This time we mainly give you an analysis of the idempotent problem of the online system interface, when there is no idempotent, how the interface is in the multi-threaded concurrent scenario of data repetition problem. Then we analyzed, if based on a database table with a unique index, can be achieved interface idempotent, but if the business logic is too complex, there are a lot of data storage, or involves many tables, can’t just rely on a unique index at this time, need to rely on at the entrance of the interface and distributed lock, then can solve complex interface idempotence.

END

Scan code for free access to the entire network subscription amount of 10W + core information

Kafka Producer kernel source code analysis

Partial Information Details