What does a distributed transaction look like online? Take Python’s Saga as an example

The inter-bank transfer service is A typical distributed transaction scenario. Suppose A needs to transfer the data of two banks to B, and the ACID of the transfer cannot be guaranteed through the local transaction of one database, but can only be solved through distributed transaction.

Distributed transaction

Distributed transaction In a distributed environment, in order to meet the requirements of availability, performance and degraded services, and reduce the requirements of consistency and isolation, the BASE theory is followed on the one hand:

• Basic Availability
• Soft State
• Eventual consistency

On the other hand, distributed transactions also follow the ACID specification in part:

• Atomicity: strictly follow
• Consistency: the consistency after the completion of the transaction is strictly followed; Consistency within a transaction can be relaxed
• Isolation: no impact between parallel transactions; The visibility of the results in the middle of the transaction allows security to be relaxed
• Persistence: Strict compliance

SAGA

Saga is a distributed transaction scheme mentioned in this database paper, SAGAS. The core idea is to split a long transaction into multiple local short transactions, which are coordinated by the Saga transaction coordinator. If each local transaction completes successfully, it completes normally, and if one step fails, a compensation operation is invoked once in reverse order.

The main open source frameworks available for SAGA are the Java language, exemplified by SeATA. Our example uses the GO language and uses the distributed transaction framework github.com/yedf/dtm, which has elegant support for distributed transactions. Here’s a detailed look at the composition of a SAGA:

In the DTM transaction framework, there are three roles, as in classic XA distributed transactions:

• AP/ application, initiates a global transaction, and defines which transaction branches the global transaction contains
• RM/ Resource manager, responsible for the management of branch transaction resources
• TM/ transaction manager, responsible for coordinating the correct execution of global transactions, including SAGA forward/reverse operations

It is easy to understand SAGA distributed transactions by looking at a successfully completed SAGA sequence diagram:

SAGA practice

For the example of the bank transfer that we’re going to do, we’re going to do in and out in the forward operation, and the reverse adjustment in the compensation operation.

First we create the account balance table:

CREATE TABLE dtm_busi.`user_account` (
  `id` int(11) AUTO_INCREMENT PRIMARY KEY,
  `user_id` int(11) not NULL UNIQUE ,
  `balance` decimal(10.2) NOT NULL DEFAULT '0.00'.`create_time` datetime DEFAULT now(),
  `update_time` datetime DEFAULT now()
);

Copy the code

We first write the core business code to adjust the user’s account balance

def saga_adjust_balance(cursor, uid, amount) :
  affected = utils.sqlexec(cursor, "update dtm_busi.user_account set balance=balance+%d where user_id=%d and balance >= -%d" %(amount, uid, amount))
  if affected == 0:
    raise Exception("update error, balance not enough")
Copy the code

Let’s write a specific forward/compensation handler

@app.post("/api/TransOutSaga")
def trans_out_saga() :
  saga_adjust_balance(c, out_uid, -30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransOutCompensate")
def trans_out_compensate() :
  saga_adjust_balance(c, out_uid, 30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransInSaga")
def trans_in_saga() :
  saga_adjust_balance(c, in_uid, 30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransInCompensate")
def trans_in_compensate() :
  saga_adjust_balance(c, in_uid, -30)
  return {"dtm_result": "SUCCESS"}

Copy the code

At this point the handler for each subtransaction is OK, and then the SAGA transaction is opened and the branch call is made

This is the DTM service address
dtm = "http://localhost:8080/api/dtmsvr"
# This is the business microservice address
svc = "http://localhost:5000/api"

    req = {"amount": 30}
    s = saga.Saga(dtm, utils.gen_gid(dtm))
    s.add(req, svc + "/TransOutSaga", svc + "/TransOutCompensate")
    s.add(req, svc + "/TransInSaga", svc + "/TransInCompensate")
    s.submit()
Copy the code

At this point, a complete SAGA distributed transaction is written.

If you want to run a complete successful example, refer to the example yedf/dtmcli-py-sample, which is very simple to run

# Deploy to start DTM
Docker version 18 or above is required
git clone https://github.com/yedf/dtm
cd dtm
docker-compose up

# Start another command line
git clone https://github.com/yedf/dtmcli-py-sample
cd dtmcli-py-sample
pip3 install flask dtmcli requests
flask run

# Start another command line
curl localhost:5000/api/fireSaga
Copy the code

Handling network Exceptions

What if a transaction committed to DTM fails briefly when the call is transferred into the operation? According to the protocol of SAGA transactions, DTM retries the unfinished operation, so what do we do? The fault may be a network failure after the completion of the transfer operation, or it may be a machine downtime during the completion of the transfer operation. How to deal with to be able to ensure that the adjustment of account balance is correct without problem?

The proper handling of such network exceptions is a big problem in distributed transactions. There are three types of exceptions: repeated requests, empty compensation, and suspension, which all need to be handled correctly

DTM provides a subtransaction barrier feature that ensures that the business logic in the above exception case will only have one successful commit in the correct order. (For details on subtransaction barriers, see the subtransaction barriers section of the seven most classic solutions for distributed transactions.)

Let’s adjust the handler function to:

@app.post("/api/TransOutSaga")
def trans_out_saga() :
  with barrier.AutoCursor(conn_new()) as cursor:
    def busi_callback(c) :
      saga_adjust_balance(c, out_uid, -30)
    barrier_from_req(request).call(cursor, busi_callback)
  return {"dtm_result": "SUCCESS"}
Copy the code

The barrier_from_req(request).call(cursor, busi_callback) call uses subtransaction barrier technology to ensure that the busi_callback callback is committed only once

You can try to call the TransIn service multiple times, with only one balance adjustment.

Deal with rollback

What if when the bank is preparing to transfer the amount to user 2, it finds that the account of user 2 is abnormal and fails to return? We adjust the handler so that the in return fails

@app.post("/api/TransInSaga")
def trans_in_saga() :
  return {"dtm_result": "FAILURE"}
Copy the code

We present a sequence diagram of a transaction failure interaction

At one point, the forward TransIn operation did nothing but return failure, so if you call TransIn’s compensation operation, will the reverse adjustment go wrong?

Don’t worry, the previous subtransaction barrier technology ensures that if a TransIn error occurs before the commit, the compensation is null; If the TransIn error occurs after the commit, the compensation operation commits the data once.

You can change the TransIn that returns an error to:

@app.post("/api/TransInSaga")
def trans_in_saga() :
  with barrier.AutoCursor(conn_new()) as cursor:
    def busi_callback(c) :
      saga_adjust_balance(c, in_uid, 30)
    barrier_from_req(request).call(cursor, busi_callback)
  return {"dtm_result": "FAILURE"}
Copy the code

The resulting balance will still be correct, based on the principle that the seven classic solutions for distributed transactions are subtransaction barriers

summary

In this article, we introduce the theory of SAGA and, with an example, complete the process of writing a SAGA transaction, covering normal successful completion, exception, and successful rollback. I believe that readers have gained a deep understanding of SAGA through this article.

DTM used in this paper is a new open source Golang distributed transaction management framework, powerful, support TCC, SAGA, XA, transaction messages and other transaction modes, support Go, Python, PHP, Node, CSHARP and other languages. It also provides a very simple and easy to use interface.

After reading this article, welcome to visit the project github.com/yedf/dtm, give a star support!