Tew TDSQL | zhangjiagang, I am your “the wall”

Growing popularity, with the continuous development of the Internet and all walks of life have also chose the road of the cloud, tencent cloud database is committed to using advanced technology, power companies on the cloud, a distributed database TDSQL is deployed in the cloud of tencent a strong consistent high availability, global deployment architecture, distributed horizontally, high-performance products, enterprise security features such as database, “Illustrated TDSQL” column will take you to understand all aspects of TDSQL in the form of graphic combination.

This article will give you a detailed interpretation, Tencent cloud TDSQL and Zhangjiagang Rural Commercial Bank of the “road to break the wall”, jointly promote the traditional bank core database localization for the first time.

The traditional core system of banks is monopolized by foreign countries

Although at present, the centralized database in foreign countries has gradually fallen behind the pace of The Times, but for banks, the cost of trial and error is difficult to bear, and the loss caused by any error data is incalculable. Banks’ database systems are more than 99 percent monopolized by foreign companies. The complexity of database is more than that of operating system. As a basic software, database has high requirements for maturity, which means it needs a long research cycle and test before it can enter the market. This is why the domestic commercial database field has been monopolized by foreign enterprises for a long time.

Domestic database has made great progress

In 2002, Tencent’s technical team chose fully open source MySQL to build the database system. In order to solve the problems of high availability of company-level sensitive business such as billing, zero loss of core data and zero error of core transactions, Tencent has developed a database product since 2007, which is also the predecessor of TDSQL. The database supported the open platform wave of 2009 at the time. With the development and expansion of Tencent’s open cooperation, there are more and more industry scenes, and this database cannot provide good services for partners. Therefore, TDSQL, a self-developed database derived from Tencent’s internal business adaptation, was formally born in 2012. With continuous iteration and development, TDSQL has now perfectly solved the problems of high availability, data consistency and horizontal scaling in financial and other systems.

Challenges

The system structure is special and complex, and data migration is demanding. Generally, the banking business in the industry is divided into traditional business and Internet business, and each has a set of core systems, which are relatively independent. However, different from the general situation in the industry, a set of core systems of Zhangjiagang Bank supports the traditional business and Internet business of the whole bank. The data volume of the core system of this migration is in TB level, including accounts, accounts, flow, bills, logs and other data. Fang Changliang, the construction of Zhangjiagang Travel System, said its core system is mainly divided into two parts. One is the transaction subsystem, with a total of more than 70 structures, covering bank cards, fund management and so on. The other is the accounting subsystem, which is mainly the transaction separation of funds and the clearing general ledger. The core system not only has complex system structure, but also has connection with each system, so the database migration is very complicated and difficult.

The solution

1. The two versions are developed simultaneously

In the process of system construction, Zhangjiagang Bank, together with Tencent Cloud and System construction Fang Changliang Technology, fully combined with the difference between traditional banking business and Internet business scenarios, redesigned the business logic, and jointly completed the compatibility transformation of the new generation of core system applications and database upgrade and optimization. The three parties also jointly developed TDSQL and centralized database data quasi-real-time synchronization strategy as a bottom-line solution. Finally, under the guarantee of sufficient technical verification and thorough implementation plan, Zhangjiagang Travel built a database architecture with high availability and strong consistency.

The database is fragmented in “one active node, three standby nodes” mode to ensure that the active node can automatically switch over to the standby node and restore services within 40 seconds when the active node fails. Perfect global distributed transaction design can completely avoid the problems of wrong account, disorderly account and account inequality.

2. Resolve compatibility issues

Compatibility differences include two parts: Oracle ecosystem vs. MySQL ecosystem, centralized architecture vs. distributed architecture. How to solve this problem? TDSQL has made a number of optimizations for Oracle syntax compatibility to address the problem that Oracle supports syntax but MySQL does not. For some features not suitable for distributed scenarios, such as: Stored procedures, views, triggers, etc., the business is to use these features, because is a lot of business logic in the database, this partly led to the lack of extensibility, TDSQL team and Banks, long core system developers on a careful analysis and evaluation of science and technology, will be more suitable in the application layer up part of the logic, A more thorough distributed architecture has been achieved, greatly improving the overall horizontal scalability.

3. Data synchronization mode

Design multi-source synchronization solutions to ensure the import and export of TDSQL to other heterogeneous databases. When TDSQL needs to be replaced for some reasons, data can be easily moved out through the data synchronization solution. TDSQL supports data subscription in standard formats in the industry, facilitating data import and export. At the same time, Oracle disaster recovery scheme is also introduced to keep real-time synchronization relationship between Oracle and TDSQL as backup. In extreme cases, TDSQL can be switched to Oracle to make banks more confident.

4. Improve the distributed transaction mechanism

TDSQL distributed transaction scheme based on the two-phase commit, on the basis of the MySQL native XA transaction done a lot of optimization, make it meet the usage scenarios of distributed transactions, as well as the transaction during the two stages did it give full consideration to all kinds of abnormal scene, to provide a global view of distributed deadlock detection, clear view multiple lock waits relation between the distributed transaction, Convenient development and investigation. In addition, the whole design is completely decentralized and there is no single point of bottleneck. The whole transaction is completely transparent to the business, and the business only needs to be used as a conventional transaction. Therefore, it is very suitable for the banking financial scene.

5. Two systems assist operation

The “Red Rabbit” and “Bian Que” systems provided by Tencent Cloud provide rich tools for the automatic operation and maintenance of databases, which greatly solve the efficiency problems of traditional centralized database operation and maintenance.

“Red Rabbit” platform can provide hundreds of monitoring indicators display, combined with flexible and rich alarm strategies to provide risk warning; “Que” system is TDSQL provide including data acquisition, real-time detection, automatic processing, performance testing and health assessment, SQL performance analysis, a variety of intelligent tool set, such as business diagnosis in all kinds of database, module pluggable seamless docking, can automatically grab the SQL performance problems, and intelligent analysis index optimization advice, Nip database performance problems in the bud. With the help of Bian Que, DBAs can be freed from the daily complex database operation and maintenance work. “Red rabbit” platform from the perspective of the administrator to provide all TDSQL operation and maintenance functions and hundreds of database status monitoring indicators display, so that more than 90% of the daily operations of database administrators can be completed through the interface, at the same time more convenient to locate and troubleshoot problems. The combination of “Red Rabbit” and “Bian Que” not only meets the refined operation and maintenance of high-star business, but also can easily cope with a large number of ordinary database operation and maintenance requirements, so as to better help users reduce operation and maintenance costs.

The wall process

1. Peripheral pilot

At the beginning of 2018, Tencent Cloud first came into contact with Zhangjiagang Bank. At that time, a peripheral system of Zhangjiagang Bank, which paid water and electricity bills, wanted to try domestic distributed database. After several rounds of POC tests, it finally chose TDSQL. But most banks’ attempts at domestic databases are limited to that.

2. One-year validation

Around August 2018, Zhangjiagang Travel prepared to transform the core system. The original plan was to use a foreign commercial database. Zhangjiagang Travel made a bold decision: Simultaneously develop two sets of new core business systems, one based on a foreign commercial database and the other based on TDSQL, and then conduct “internal horse racing”. One year later, conduct comparative tests on the stability and performance of the two systems, and then decide which set to use according to the test results. Finally, under the guarantee of sufficient technical verification and thorough implementation plan, Zhangjiagang Travel built a database architecture with high availability and strong consistency. The database is fragmented in “one active node, three standby nodes” mode to ensure that the active node can automatically switch over to the standby node and restore services within 40 seconds when the active node fails. Perfect global distributed transaction design can completely avoid the problems of wrong account, disorderly account and account inequality.

3. Winners

After a whole year of transformation, the distributed database showed obvious advantages both in terms of performance cost and ease of use, and finally the new core system adopted TDSQL distributed database, while the previous core system using centralized database was reserved for disaster recovery system.

performance

According to the data disclosed by the bank, after the distributed transformation of the traditional core system of the bank, the transaction inquiry will be completed within 100 milliseconds, the high-frequency transaction will be completed within 300 milliseconds, the loan settlement will take 3 minutes, and 10,000 batch issuing and withholding business can be completed within 20 seconds, the final batch running takes 14 minutes, and the deposit settlement will take 11 minutes. At the same time, the database load is kept below 10%, which has fully met the business volume of Zhangjiagang Travel in the next ten years.

The cost of

The new core system uses x86 servers in the hardware level, replacing the mainframe and minicomputer required by the traditional commercial database, with obvious cost advantages. Data show that the hardware cost of Zhangjiagang Travel after adopting Tencent Cloud TDSQL distributed database architecture is only 1/5 or even lower than the cost of traditional architecture.

A new direction

This case proves to us that in the core banking system, the commercial database which has been monopolized by foreign countries for a long time can be replaced by domestic distributed database. Both for the bank and Tencent cloud, this is a representative case, there may be more similar cases in the future, and there are more possibilities for the development of banks and domestic databases.

Click to enjoy Tencent Cloud CynosDB database preferential activities