PS: If you don’t understand this article temporarily, it may be because I omitted some knowledge due to the length of my blog, which leads to logical faults in some links. You need to think about it carefully. It could be that I’m misrepresenting it and it’s ambiguous; No matter what the reason, welcome to comment area exchange or through I write this blog reference data and the blog for systematic learning ~ if it is helpful to you or trouble to click like, I like the third level of the blue ah, the second level of blue is not beautiful 😂

“This article has participated in the good article call order activity click to see: back end, big front end double track submission, 20,000 yuan prize pool for you to challenge!”

1. Evolution from local transactions to distributed transactions

For example, traditional monolithic applications that rely on mysql can provide a consistent view of data to upper-layer applications by using the ACID feature of local transactions based on lock +MVCC, without the problems of dirty reads, dirty writes, and update loss caused by concurrent reads and writes of multiple transactions.

This article goes through a relatively simple business scenario for creating an order:

• There is an OrderService -OrderService that passescreateOrder()Provide the function of creating orders.
• Rely on ConsumerServiceconsumerVerify()Deduct the credit limit of the consumer. If the balance is greater than the order amount, the creation is successful; otherwise, the external return quota authentication fails;
• Rely on AccountingServiceaccountAuth()Calculate the amount of all orders on that day. If the agreed limit amount is not exceeded, the creation is successful; otherwise, the creation of the current Order fails;

For individual services, because the OrderService, ConsumerService, and AccoutingService methods depend on different tables in the same database, The @Transactional(rollbackFor = exception.class) annotation decorates the createOrder method to be a Transactional method so that data inconsistency does not occur due to concurrent access by multiple transactions. If an exception is thrown after an order fails to be created, the transaction can be rolled back directly.

The deployment of the preceding services is as follows:

At that time, due to the increasing complexity of business and data volume, single architecture can no longer support the traffic with high concurrency. At this time, two methods can be adopted for the database: vertical sub-table and horizontal sub-table;

• Horizontal sub-table: Split the rows of the table. Because tables with more than a few million rows become slow, you can split the data from a single table into multiple tables. Horizontal split, there are many strategies, for example, take the module table, time dimension table, etc. In this scenario, even though we are dividing tables according to specific rules, we can still use local transactions because we can route reads and writes to the same data to the same database instance;

• Vertical sub-table: Split the columns of a table. Vertical table is the original data column in the table according to the dimensions in the field of broken up into multiple tables, and multiple tables will be deployed in different micro service corresponding to the different database, but it may still exist the requirement of data consistency between multiple tables, and each database in a distributed scenario can only keep its local transaction ACID properties, Besides, microservices can only know the execution state of other transactions through network communication, so additional mechanisms need to be provided to maintain distributed transactions among microservices.

Because the process of single architecture -> microservice architecture is originally local process call -> remote procedure call/message queue communication, so the discussion of service availability and data consistency needs to include the network factor;

CAP theory defines that when network problems occur in network partitions, we can only guarantee one of the two characteristics: service availability and strong consistency of data. Therefore, when considering network problems, we generally choose between availability and consistency.

BASE theory is to ensure the availability of services by weakening the strong consistency of data and transforming it into the final consistency of data.

Several patterns of distributed transactions will be introduced below:

2. 2PC(XA) and TCC modes

XA and TCC are strong data consistency solutions based on two-phase submission and synchronous network communication. Because they guarantee strong data consistency, service may be unavailable due to network partition.

The specific process is shown in the figure below:

• The first stage – preparation: The transaction manager invokes all participants in the current distributed transaction remotelyprepare()The method is used to lock transaction related data and record transaction related logs to facilitate transaction rollback.
• The second phase – commit phase: if all participants returnprepare()Method succeeds, then the transaction manager invokes all participants remotelycommitMethod to commit the transaction; Otherwise the callcancel()Method to roll back the transaction;

2.1 Problems with the two-phase submission mode

• Some databases and message brokers do not support rollback: popular message brokers Kafka, RabbitMQ, etc., do not support XA transaction rollback. This means that in 2PC, even if other services are rolled back, the information that has been sent to the message broker cannot be withdrawn, resulting in data inconsistency.

• Performance problems of synchronous communication: synchronous communication of coordinator node maintains the state of distributed transaction, and local transaction of participant node maintains strong data consistency among nodes; However, during the whole process of the distributed transaction, the local transaction is always in the uncommitted state and the corresponding business data is in the locked state. Therefore, in the case of high concurrency or a large number of participant nodes involved in the distributed transaction, the amount of concurrency that the system can bear will be reduced. Therefore, the most intuitive optimization scheme is synchronous -> asynchronous, and based on the message event implementation; The reliable event pattern described below;

• Synchronization blocking problem: The whole process of the above two PCS is synchronous, and the transaction manager (coordinator) must wait for each resource manager (participant) to return the operation result before proceeding with the next operation, which is very easy to cause synchronization blocking problem; Therefore, 3PC divides the preparation stage into preparation and preparation stage, which is used to discover the hanging node in advance and terminate the transaction in advance in the preparation stage, and introduces the timeout mechanism to solve the problem that the coordinator is always blocked due to network factors.

• Problems with introducing timeouts: Because the network delay is not fixed, there may be a network packet that the coordinator node initiates the call of prepare() because the network delay has not reached the service node within the timeout period, so the coordinator node will initiate the call of Cancel () for the transaction rollback. So there may be a case where the cancel() packet has arrived but the prepare() packet has not, so the service needs to support null rollback; But a more reliable way is to set the expiration time >> network communication time, and set the network packet based on the average latency of the exponential avoidance of resend mechanism.

• Single point of failure of the coordinator: The two-phase submission will also have a single point of failure. Although 3PC introduces the timeout mechanism and divides the preparation process of the two-phase submission into two steps, it still cannot avoid the single point of failure of the coordinator. The reliable event pattern based on service collaboration can effectively avoid single point of failure.

3. Reliable event mode

Reliable event pattern is the final consistent solution for service collaborative communication based on queue. Unlike 2PC, which divides a distributed transaction into two stages: prepare/commit, the reliable event mode divides a distributed long transaction into several transactions and triggers the transaction flow asynchronously by sending an event to the message queue.

For example, after receiving the createOrder request from the client, OrderService will create the Order object and generate the OrderCreated event to post to the message broker. ConsumerService has subscribed to the Message broker with an OrderCreated event and the corresponding event handler is the consumerVerify() method. If the processing is successful, it will post the ConsumeVerified event to the message broker. The AccoutingService subscribed to the ConsumeVerified event to the message broker, and the corresponding event handler was accountAuth()…

The reliable event pattern, in which transaction flows are delivered via message queues, avoids the performance problems associated with 2PC synchronization, but introduces some new problems.

3.1 Compensation Transaction

Unlike 2 PC, reliable event mode of distributed transactions involving multiple service local transactions, are in their own event handler processing is completed and submitted directly, will not wait for the coordinator node commit/commit/rollback of the reentry after the cancel orders, which can not only produce the problem of isolation (in section 4.1 about together), We also need to provide a transaction rollback mechanism for committed transactions – compensation transactions;

We divide the local transactions involved in distributed transactions into compensable transactions, key transactions and repeatable transactions.

• Compensable transaction: means that in a distributed transaction, other local transactions after the local transaction may fail, so if the subsequent transaction fails, the transaction needs to be compensated;
• Critical transactions: means that in a distributed transaction, other local transactions after the local transaction, such as those used for logging, fail without affecting the process.
• Repeatable transactions: means that the transaction always succeeds.

So if the NTH + 1st transaction in the distributed transaction fails, and that transaction is not repeatable, the impact of the first N transactions on the database needs to be rolled back.

If the current order cannot be created when called AccountingService, the AccoutingAuthFailed event is posted to the message queue, and the ConsumerService needs to subscribe to the event. And associate its own corresponding compensation transaction method to roll back changes to the credit limit.

3.2 Problems to be solved by reliable event mode

3.2.1 Sender: Atomicity of executing business code and sending event messages

Generally, in order to avoid the situation where the message is sent successfully and the business code execution fails, we choose to send the event message after the business logic is executed, but sending the event message to the message queue is not always successful. Therefore, you need to provide a mechanism to ensure that the event will eventually be sent to the corresponding message queue, that is, a transactional message;

The realization of the transactional message is generally by sending additional service maintenance events table, before sending the message will be sent message content and type in the corresponding events in the table, make business code and send the event can be covered by local transaction database to ensure consistency, namely the final state meets the demand of atomicity, Finally, additional message relay service is used to obtain the event information of event table and ensure the reliability of event publishing.

The next problem is how the message relay obtains the event message to be sent from the information in the event table;

• Message filtering: so we need to events based on the state of the process modeling, an event may be in not sending, has been sent to the news agency, news agency has sent to the consumer, consumer spending and successfully ACK this a few process, therefore the transaction in the event there are corresponding to several states in the table, and through the status status are identified.

• Message acquisition: The message relay can obtain the unsent messages in the event table corresponding to the service through SQL polling and publish them. But obviously this will make the speed of message release constrained by the efficiency of database query, and polling will increase the burden of database to some extent. We can get information in the corresponding event table, also can be read by the database transaction log to reflect the change of the event table, so we can through asynchronous log trailing model, optimization, read through the message relay service database transaction log and filtering to obtain event messages to be sent, rather than direct access to the database table; In addition, the benefit of this is decoupling, and message relay can work not only on relational model databases but also on NoSQL databases such as Redis by relying on transaction logs rather than tables.

So far, we analyze how to ensure the atomicity of business code execution and message publishing, mainly by writing the event table to make the message sending join the local transaction; However, this will make the database where the service is to bear the additional data cost of the event table, and also reduce the carrying capacity of the business. Therefore, we can differentiate separate event services to maintain the event tables of different topics and publish the events.

3.2.2 Message queue: Event message sequence problem

Because event messages can’t always be concurrent, there may be a causal order between them, so message queues need to ensure that they send messages in the same order as they receive them.

Because message queues such as Kafka generally achieve high availability by means of replication and partitioning, and for the same topic, the messages may be in different partitions, and there is no global order between the messages of different partitions, but the messages within the partition itself is to meet the full order. Therefore, consumers need to specify a partitioning policy based on event type when sending messages, so that message queues can send messages of the same event type to consumers in full order for processing, so as to satisfy the causal relationship between events.

3.2.3 Consumer: the problem of idempotence of consumption

Because message queues such as Kafka only provide a guarantee that a message will be successfully sent at least once, that is, a transaction message may be received multiple times due to a timeout mechanism, consumers need to ensure that receiving multiple event messages has the same impact as receiving one event – idempotent;

• The most intuitive solution is that the event processing method of the consumer realizes interface idempotent. Generally, it determines whether the event message has been processed through the state of the business object, but it will make all the code redundant with the aspect code of idempotent guarantee, so it is better to intercept the message through additional mechanism.
• The optimization approach is to provide additional mechanisms to determine whether the current event can be processed before the event message is processed by the event processing method. Consumers can maintain additionalEvent processing tableUsed to record what has been processedevent_idIf it has been processed, discard it.But consider the scenario where the consumer fails to process the message halfway throughTimeout mechanism, can rely on the message queue, if you do not receive the message queue arrived after the timeout ACK message to consumers, is to resend the message, but at the moment, need to bypass the idempotence logic, so consumers after the restart because downtime, etc to event processing finished processing the message status in the table are rolled back.

4. Choreographed Saga mode

Saga pattern can not only realize the above reliable event pattern based on the collaboration between services, but also realize the choreographer based event pattern.

The choreographer of Saga has similar functions to the coordinator node in 2PC, but the implementation is different. 2PC is based on request/response communication, while Saga is based on command/asynchronous response.

For each invocation method exposed to the client, such as createOrder, the distributed transaction framework implementing choreographed Saga orchestrates the local transaction flow involved in the corresponding distributed transaction for that method and centrally controls it.

The specific process is shown in the figure below, noting the similarities and differences with the reliable event pattern based on collaboration between services:

The createOrder() method relies on the ConsumerService and AccoutingService, and the corresponding CreateOrderSaga sends the OrderCreated event to the ConsumerService via the message queue. ConsumerService will return the reply event to the CreateOrderSaga reply channel for Saga to process; If the response is correct, then the AccoutingAuth event will be sent to the AccoutingService through the message queue and wait for the reply…

How to ensure reliable communication between Saga choreographers, dependent services, and message queues is described in detail in Section 3.2, followed by isolation issues.

4.1 Isolation problems caused by asynchronous message queue-based communication

Choreographed Saga and reliable event patterns are essentially asynchronous communication between services through the introduction of message queues. Unlike synchronous communication protocols such as XA or TCC with two-phase commit, Saga does not have a strict separation between prepare, commit, and rollback phases. Instead, each service directly commits the current local transaction through a transactional messaging mechanism when processing an event, rather than waiting for other related transactions to complete. That is, the local transactions involved in a distributed transaction may be in different states of unstarted, uncommitted, committed, and rolled back, so the data involved in the whole distributed transaction may be in inconsistent state.

Take a dirty chestnut:

In a local transaction, dirty reads are defined as: a transaction reads an update from an uncommitted transaction; Similarly, in a distributed transaction, an update is read from a distributed transaction that has not yet completed.

• The user initiated oneCreateOrderThe amount of the bill is 100, issued by OrderServiceOrderCreatedEvents;
• ConsumerService determines that the current user account has a credit limit of 120>100, so the remaining 20 is deducted and processed through the event table and message relay mechanismConsumerVerifiedThe response to the event is waiting for the AccoutingService to consume;
• If the user initiates anotherCreateOrderRequest, the bill amount is 10, so even if the last distributed transaction hasn’t been committed, the ConsumerService will return 20 instead of 120, which leads to dirty reads.

Of course, the dirty reads mentioned above may seem harmless, but the problem with dirty reads is that they violate the binding requirements between the data, thus breaking the consistency that the data view should meet. For example, the following example describes the write conflict-update loss problem caused by dirty reads:

• In the second distributed transaction, the ConsumerService deducts the remaining 20 credits from the user, where the user has 10 remaining credits;
• The first distributed transaction fails because of the AccoutingService, triggering a transaction rollback mechanism if the distributed transaction is database-basedUndoThe log is rolled back, and when ConsumerService rolls back, the user’s credit limit is back to 120, which causes the second distributed transaction to create an Order for the credit limit- 10theUpdate the lost.

These are the most common problems in distributed scenarios, such as dirty reads and update loss caused by concurrent reads and writes. We can change this by adding semantic locking and custom transaction rollback.

4.4.1 semantic lock

In the single database, in order to avoid the problem of concurrent read and write of multiple transactions, MVCC and locking mechanism are combined. The most intuitive locking mechanism is row lock, table lock, intention lock and so on, which can make concurrent transactions achieve local serial to avoid the effect of conflict;

The two-phase commit method is to realize the isolation by means of the lock feature of the database local transaction. However, the message queue and event based mode is obviously not applicable, because each local transaction is committed independently, so it needs to implement a semantic lock at the application layer, providing lock information so that the second distributed transaction knows that another transaction is operating on the data it cares about, so as to avoid concurrent read and write conflicts.

At the application layer, the state of the root business object can be aggregated as the lock information. For example, with the Order object above, we can model the state machine for the object:

State machine modeling is generally divided into the state of an object and the operations that cause the object state to change. Based on this business process, the above state flow diagram is drawn. Based on the figure above, we can implement semantic locking based on the state. Check whether the current Consumer business object has an order in the creation process. If so, block and wait until the previous order is created or fails and the rollback is complete.

4.1.2 Custom Rollback Mode

We mentioned above that relying on the database Undo log to roll back transactions can cause updates to be lost, so we need a custom rollback method – swap updates;

The easiest way to swap updates is to use the transaction log. If the user requests to create an Order with a credit limit of 100, but the current transaction fails with the AccountingService, the ConsumerService needs to roll back the current user’s credit limit. For the current user’s credit limit +100; The update loss problem for the second transaction in the above scenario does not occur;

The premise of using this policy is that the operation of the Service on the business object has a corresponding interchangeable operation. For example, borrowing 50 yuan can eliminate the effect of borrowing 50 yuan by paying 50 yuan back.

5. Summary and reference

This paper mainly analyzed based on two-phase commit, coordinator of the synchronous communication 2 PC, then through 2 PC performance and lack of fault-tolerant raises the issues that led to the single point of failure of asynchronous communication based on message queue to realize reliable event model, and then introduces the coordinator + message queue based implementation of command/asynchronous response the arrangement type of Saga plan of means of communication.

Main references: DDIA, Distributed Architecture Design Patterns

This article explains how to ensure transaction consistency in micro services

The first Seata Saga mode for distributed transactions and detailed explanation of the three modes

Microservice data consistency solution in the Choerodon Toothfish platform

Meituan group purchase order system optimization record