An overview,

With the development of time and business, the amount of data in the database table will be more and more large, and accordingly, the cost of data operation, increase, deletion, change and check will be more and more large. Therefore, some of these large tables are split into multiple tables in multiple databases.

This article is based on the asynchronous guarantee of non-transactional messages to solve the transaction problem in the sub-database sub-table.

Second, problems need to be solved

2.1 Original Transaction

Since the new table is in another database after the database is divided, how to ensure the transactional between the main database and the sub-database is a problem that must be solved.

Workaround: Map the logic of the operating database to a flow record by creating a flow table in the main library. When the entire large transaction is completed (the stream is inserted into the stream table), the stream is then executed in other ways to ensure final consistency.

2.2 water

A stream can be understood as a transaction message

By creating a flow table in the database above, a flow record is used to represent a business process logic, so a flow must eventually execute correctly. Therefore, when extracting a piece of business code into the pipeline, consideration must be given to:

  • ** Water delays handling. ** Streams are not processed in real time, but are executed asynchronously using stream actuators. Therefore, if in the original logic, you need to pay special attention to whether subsequent processes are real-time dependent on the flow (for example, subsequent business logic will use the flow results to do some calculations).

  • ** Flow processing disorder. ** guarantees that no problem will occur even if the generated stream is executed first.

  • ** ** ultimate success. ** For each inserted stream, the stream must be guaranteed to execute successfully

Therefore, when extracting water:

  • The simpler the flow treatment, the better

  • The less dependency there is on churn processing, the better

  • The extracted flow has no real-time dependence in the business logic

2.3 Pipelining Processor

A stream processor must ensure that the stream processing is as fast as possible and that the stream can be successfully executed.

Imagine a scenario: when there is a stream processing failure, if the drain executor waits for the current stream to execute successfully before continuing to execute later, the execution of subsequent streams will be affected, and even worse, it will be stuck in the current record, leading to problems in the whole system

Therefore, there are two tasks in the stream executor:

  • The first task, the flow processing task, has been executed as fast as possible. If the flow processing fails, it does not affect the subsequent flow processing

  • The second task is the flow verification task. This task is to check the flow records in sequence to ensure that all the flow records are successfully executed. If the flow records fail, retry.

2.4 Completion of flow treatment

Because the flow table is placed in the original database, and after the completion of the flow processing is the operation of the branch database, if the branch operation is completed to update the old table flow message, then it is a library transaction, how to ensure that the update of the flow state and the branch database is also in the same transaction?

The solution is to create a rundown table in the database. When rundown processing is complete, instead of updating the state of the old table, insert the state into the database.

The benefits of this:

  • The transaction will be rolled back if the unique index does not pass the test when the database is inserted into the database table.

  • In this way, we can judge whether a stream is completed by judging whether the main stream is in the branch

The basic framework of the pipelining processor

The pipeline processor does not actually contain any business-specific processing logic, and its core functions are:

  • Notify the business access party when and what flow will be processed

  • Verify the success of pipeline execution

Note: The stream executor does not know what logic the stream represents, and the specific business system needs to identify and execute the corresponding business logic.

3.1 Executing tasks in Flow

The scheduling task of stream processing is to inform the business system which stream to execute by scanning the stream to be processed.

The schematic diagram is as follows:

3.2 Pipeline-based Verification Tasks

The flow verification task is to compare the flow records in the primary database and the branch database, and notify the service system to process the flow records that fail to be executed. If multiple retries fail, an alarm is generated.

Process diagram:

Why not use transaction messages

There is one reason for not using transaction messages since it is an existing project that is modified (I am engaged in Internet finance, so I absolutely do not tolerate any message loss or message processing failure)

  • The introduction of additional message queues increases the complexity of the system, and additional logic is needed to ensure that message queue communication fails

  • In spring, the transaction is transitive, so when we commit the transaction message is a big problem. For example, A.a() is already a transaction. But A.a() is called in another transaction B.b(). Is the transaction message commit in A.a() or B.b()?

Write in the last

Welcome to pay attention to my public number [calm as code], massive Java related articles, learning materials will be updated in it, sorting out the data will be placed in it.

If you think it’s written well, click a “like” and add a follow! Point attention, do not get lost, continue to update!!