We need to accept disappointment because it is finite; We cannot lose hope, for it is infinite.An overview,

With the development of time and business, the amount of data in the database table will be more and more large, and accordingly, the cost of data operation, increase, deletion, change and check will be more and more large. Therefore, some of these large tables are split into multiple tables in multiple databases. This article is based on the asynchronous guarantee of non-transactional messages to solve the transaction problem in the sub-database sub-table.

Second, problems need to be solved

2.1 Original Transaction

Since the new table is in another database after the database is divided, how to ensure the transactional between the main database and the sub-database is a problem that must be solved.

Workaround: Map the logic of the operating database to a flow record by creating a flow table in the main library. When the entire large transaction is completed (the stream is inserted into the stream table), the stream is then executed in other ways to ensure final consistency.

2.2 water

A stream can be understood as a transaction message

By creating a flow table in the database above, a flow record is used to represent a business process logic, so a flow must eventually execute correctly. Therefore, when extracting a piece of business code into the pipeline, consideration must be given to:

  • ** Water delays handling. ** Streams are not processed in real time, but are executed asynchronously using stream actuators. Therefore, if in the original logic, you need to pay special attention to whether subsequent processes are real-time dependent on the flow (for example, subsequent business logic will use the flow results to do some calculations).

  • ** Flow processing disorder. ** guarantees that no problem will occur even if the generated stream is executed first.

  • ** ** ultimate success. ** For each inserted stream, the stream must be guaranteed to execute successfully

Therefore, when extracting water:

  • The simpler the flow treatment, the better

  • The less dependency there is on churn processing, the better

  • The extracted flow has no real-time dependence in the business logic

2.4 Completion of flow treatment

Because the flow table is placed in the original database, and after the completion of the flow processing is the operation of the branch database, if the branch operation is completed to update the old table flow message, then it is a library transaction, how to ensure that the update of the flow state and the branch database is also in the same transaction?

The solution is to create a rundown table in the database. When rundown processing is complete, instead of updating the state of the old table, insert the state into the database.

The benefits of this:

  • The transaction will be rolled back if the unique index does not pass the test when the database is inserted into the database table.

  • In this way, we can judge whether a stream is completed by judging whether the main stream is in the branch

    The basic framework of the pipelining processor

The pipeline processor does not actually contain any business-specific processing logic, and its core functions are:

  • Notify the business access party when and what flow will be processed

  • Verify the success of pipeline execution

Note: The stream executor does not know what logic the stream represents, and the specific business system needs to identify and execute the corresponding business logic.

3.1 Executing tasks in Flow

The scheduling task of stream processing is to inform the business system which stream to execute by scanning the stream to be processed.

The schematic diagram is as follows:

3.2 Pipeline-based Verification Tasks

The flow verification task is to compare the flow records in the primary database and the branch database, and notify the service system to process the flow records that fail to be executed. If multiple retries fail, an alarm is generated.

Process diagram:

Why not use transaction messages

Since it is an existing project (Internet finance, so there is absolutely no tolerance for any message loss or message processing failure), there is one reason not to use transactional messages

  • The introduction of additional message queues increases the complexity of the system, and additional logic is needed to ensure that message queue communication fails

  • In spring, the transaction is transitive, so when we commit the transaction message is a big problem. For example, A.a() is already a transaction. But A.a() is called in another transaction B.b(). Is the transaction message commit in A.a() or B.b()?

This article is published by OpenWrite!