The world grows to be distributed ~

The body of the former

This article is based on the summary and reflection of the past year’s work, and some practices of various distributed solutions and business thinking. Welcome to read, Comment and discussion:

• The core of this paper is the picture, which carries all the core information and cognitive framework. If you don’t understand the picture, you can help read the text.
• Distributed transaction refers to broad transaction, including flexible transaction, rather than narrow db-based transaction;
• For example, the ideas of 2PC and transaction messages are completely different, but the details of 2PC and 3PC are not developed.
• A simple understanding of distributed transaction-related concepts is a good prerequisite for this article.

Cognitive framework

There are many different and related concepts about transaction. At present, my own cognitive framework is divided into three layers: the guiding ideology layer (Idea), the specific implementation layer (Protocol) and some core design elements (elements).

Basic Idea

BASE, ACID and SAGA

Prior to the explicit introduction of BASE, distributed transactions were moving towards the ACID standard for transactions within a database, but SAGA (introduced in 1987) has begun to evolve towards the ultimate consistent and event-driven approach. In 2008, BASE proposed a more specific framework and abstraction in this direction. Finally, BASE became a theoretical model parallel to ACID, while SAGA was more of an implementation model of the next layer. Pessimistic and optimistic assumptions

ACID differs from BASE’s global assumption in its attitude: BASE is optimistic and believes that every local-unit that makes up a distributed transaction will eventually succeed; ACID is pessimistic as a whole, believing that success is only possible if guaranteed success is confirmed during synchronous execution. It’s important to understand this central assumption:

• Whether the check hypothesis is acceptable: It is necessary to consider whether the hypothesis is appropriate in the application scenario when using different ideas. When the local-unit in your scenario is prone to failure, it is very troublesome to use BASE;
• Mechanics after understanding the assumption: understand this idea, it is easier to understand the ACID idea model and BASE idea model in the implementation level of why so designed;

The BASE scheme

ACID. So let’s talk about what BASE does. One of the core Settings of BASE’s scheme is to relax the requirement of achieving a consistent moment, adopting similar quasi-real-time Eventually consistent; The data state before consistency is achieved is defined as Soft state. Soft state is a temporary state, and data will enter a consistent/integrity state after the completion of distributed transactions. When this is achieved, there is a lot of room for manipulation to achieve consistency, such as queue linking in between. The emergence of Queue decouples the data flow between consistency assurance and business request, increasing the space for control. Of course, without a queue, data scan is another simple and crude way to control. Both queue and data scan operations need a marker for soft state: the signal emitted in the queue with the locally TRX is one way, and the transaction messages go in this direction; Another way to do this is to record an init state in db ahead of time. In addition, similar stand-alone database engines also need to be considered: distributed transaction Id (TrxId), parallel processing of multiple messages (bib), idempotence are also needed to ensure the reliability of the BASE model.

Mode and Protocol

The second level is summarized as concrete patterns and protocols that define in detail how a distributed transaction is implemented. Here, it is divided into two groups according to the general direction. The core idea of the first group is 2Phase: including XA and TCC.

2Phase: XA and TCC

Phase 1 ensures that the operation is executable, and Phase 2 identifies the executable operation or rolls back the uncommitted operation. The first phase is more than just a check phase. It can be understood as a pre-execution phase, and the success of this phase usually means that the local task can be executed. For example, this step is to deduct the inventory, which is usually a direct pre-lock of the inventory. Note that this is the meeting Duration of the falling order, rather than a simple check data. Based on this setting, if all steps in the first phase succeed, the transaction can be considered successful even if all restarts, which is also a way of recovering the algorithm.

The core essence of XA and TCC are both two-stage ideas. XA focuses on the implementation of DB layer, while TCC is mainly implemented in the interface layer of microservices. The essence of the two approaches is the same, but the usage scenarios are different depending on their implementation. The use of TCC from a business development perspective can be quite extensive, as microservice invocations tend to manage the DB themselves, preferring to implement the invocation by providing an RPC interface.

The figure above is a common TCC business system architecture scheme for implementing distributed transactions: the advantage of this scheme is that the green independent business module only needs to focus on how to implement the try-confirm-Cancel interface of its own business unit. After investigation and research, XX financial service and our company’s financial transfer and withdrawal have adopted this scheme. If no-2 is a hot data, if No-2 does not need to succeed in real time and does not want to be affected by the current stability of NO-2. Then the business can often have a different choice. By the way, Seata’s AT scheme attempts to record the local transaction SQL statement and generate its own undo-lock, to do secondary development in the upper layer of the database to rollback. It also uses global and local locks to prevent two-phase chaos and other changes in a transaction (this is well worth studying and learning from). The advantage of this solution is that there is no block-DB and the business impact is small through the implementation of the framework, but the handling of Cascading rollback in the case of concurrent updates will be complicated and not explored in detail.

One By One: SAGA, transaction message

For 2Phase, the Base based idea is named One By One (ObO, my own name) to emphasize executing local transactions One By One. This series mode is no longer suitable for scenarios where large quantities of data need to be rolled back. It is better to ensure success logically. Two patterns are listed first, the essence of which is how a heterogeneous infrastructure implements transactions, that is, how local transactions and messages succeed together. The first method relies on change notifications from the DB itself, which relies on binlog. The second approach is the upper-level infrastructure design, which relies on the distributed transaction protocols half-message, commit/ Cancel, and recheck. Are these two methods familiar, similar to the difference between XA and TCC? When there is no transactional messaging middleware available, a compromise is to record a marker in the database itself to implement a double-check scheme.

DB based scheme:

Transaction message-based scheme:

Quick Review

Here we can quickly review the two ideas. The first idea is to prepare each sub-task node one by one at a time, and then commit or rollback. Another way to think about it is to do subtasks one by one, but ensure that notifications between subtasks are reliable, and then eventually complete all of the subtasks. A key point of both approaches is the logic that determines the success of a transaction and the exception handling based on that logic. There are two choices: 1) Check whether it meets the optimistic and pessimistic hypothesis; 2) Check whether the real-time requirements are high.

There are various implementations of these two ideas that form various modes and practices, with TCC and transaction messages being common patterns. Key design points commonly used by different practices are TRX-ID, lock, powers, and so on. This forms the cognitive framework for the graph in the first part of this chapter.

Business practices

Activities of winning

Demand analysis

The requirement scenario of the activity is: judging whether the user wins the prize according to the user’s behavior, and recording and distributing the prize if the user wins. According to the requirements, we disassemble the technical analysis to the key design as follows:

• More: behavior caused by the award can not be repeated: 【must-have】 otherwise it will cause capital loss;
• Less: also can not be less: 【must-have】 otherwise customer complaints;
• User perception and real-time perspective:
  • User real-time perception: perception of the winning result is real-time requirements are very high, there is exposure to the user;
  • The scenario in which the user experience is preferred is that the user can receive the prize in real time, but a little delay is acceptable.

The project design

Based on the above analysis, the part of non-repeated distribution: the business ID of the user’s request for the award behavior is idempotent, and the Unit that calculates the award is locked. In terms of consistency scheme, this scenario is very suitable for one-by-one design: 1) The award process is sure to succeed, and there is no resource limitation in this scenario, which satisfies the optimistic assumption; 2) Users have a certain degree of acceptance of delay;

So let’s see what the key elements correspond to in the one-by-one pattern in this scenario: 1) when the transaction is considered to be completed: when the first local-trX-1 completes successfully. This satisfies the user’s cognition of the award result, and persists the result of this data; Db-trx-1: the transaction id is the record ID of db-trX-1, and the state of the record indicates whether the transaction has completed the final consistency (from to-reward to reaction-sent). Therefore, DB-TRX-1 must persist the transaction when the transaction is confirmed. 3) How to transmit information: the recording and marking of information to reward the soft state database has a persistent mark, which is not exposed to the outside and is an internal consistency guarantee state; This notation is very useful so that there is an identifiable state that can be scanned and retried when no transaction messages are needed; Here I would like to expand on the cost of transaction message access in business and the company does not support it for the time being. Therefore, what needs to be scanned is abnormal data (quantity level is not large). This pattern is more general to some extent. 4) When the user wins the lottery, it will call the lottery in real time. If the call fails, it will rescan the record of this state to retry; Therefore, all downstream implementations record idempotent according to db-TRX-ID; The solution is also flexible enough to try to call TRX-2 when the user requests it, since most requests are successful in real time. But trX-2 failures are not reported back to the user, balancing user perception and system consistency.

Trade order

Demand analysis

Order purchase is a very common question: the order needs to deduct inventory, deduct marketing resources and so on. It is assumed that the one-by-one BASE model does not work at the time of the order: 1) The Optimist assumption is not satisfied, for a start. Because inventory and marketing resources are rare and cannot be guaranteed to be deducted successfully, it is very easy to rollback using the final consistency model, especially when marketing resource coupons are easy to be collected. 2) Secondly, users’ perception requirements for placing orders are very real-time, and users are unlikely to be required to wait in the interaction scene between users and app; The shipping scenario, on the other hand, has some room for ultimate consistency, even with a 30-minute fast delivery in real time;

The project design

In fact, from the previous TCC model, you should be very familiar with this solution, and this system can be reused with THE TCC model (refer to the TCC diagram in part 1). However, in the absence of distributed framework, it is more redundant to strictly follow the TCC framework step by step. Businesses tend to implement the core ideas of TCC in a lightweight way, and the figure above provides an example.

The core of this solution is to combine the global transaction record and the order, and realize the recording and processing of the global transaction through the internal state of the order: The internal states of the order are prepare and innerclosed, which are not visible to the user. Prepare is an internal soft-status state, which is a temporary state. Before normal return of the ordered interface, the state should be changed to inner-closed or wait-to-pay according to the result of Phase-1 to mark the completion of Phase-2 COMMIT /rollback. 3) If the application encounters a problem, the recovery thread will also detect the problem based on the temporary state of Prepared and restore it. Of course, if there is a very convenient TCC microservices framework directly integrated, this scenario can also be directly implemented using TCC solution.

Practice the principles of selection

In the above two examples, one is suitable for using BASE and the other is suitable for using 2Phase, and both solutions are flexible and simplified according to the business situation. So frameworks are actually outputs of ideas that engineers can adapt and apply flexibly. In fact, the standards and principles of judgment have been relatively clear from the previous reading, and we continue to clarify some conditions here. We can think about what kind of thinking should be used to meet or not meet this condition:

• Is there a high probability of success
• What links can users perceive
• Which link has the hot spot
• Which link allows asynchronous slow execution
• Which link is prone to error and time-consuming
• Is there any mixed case