The author | large data center of xin-jiang Chen wanda network technology group

Wanda network technology group is China’s only industrial + Internet large open platform company, have to fly all information, quick to pay, credit reporting, network credit, big companies such as data, use of big data, cloud computing, artificial intelligence, as well as application scenario for the entity industry to realize digital upgrade, provide consumers with new consumer service life.

The technical team of Wanda Network Technology Group builds and maintains a real-time risk control platform. This real-time risk control platform provides online risk control data writing and query services for key transactions. The back-end database system of the real-time risk control platform has high requirements on high performance, reliability and scalability, and needs to meet the following core functions and business requirements:

  1. Data related to risk control is stored in real time
  2. Real-time risk control rule calculation
  3. Analyze risk control history data through BI tools
  4. ETL is stored in the Hadoop data warehouse
  5. The application development side must be compatible with MySQL to reduce the threshold for application transformation

In order to achieve the above business objectives, the technical team of Wanda Network Technology Group first chose MySQL Galera Cluster as the technical architecture of the database Cluster in the early stage of real-time risk control database selection. This MySQL database architecture adopts a replication mechanism that is different from the mainstream MySQL replication technology to establish a strong synchronization relationship between multiple MySQL nodes and realize data copy and high availability. However, after business practice, it is found that this scheme has many problems, among which the following points are prominent:

  • The strong synchronization mechanism of MySQL Galera Cluster itself comes at the expense of significantly reducing the overall Cluster performance, which is worse than single-node MySQL. Therefore, it cannot well meet the business requirements of “real-time warehousing of risk control-related business data”.

  • At the same time, the JOIN support of MySQL Galera Cluster is very weak and not enough to support the complex analysis related to BI.

  • It is difficult to calculate the risk control rules to achieve high concurrency performance in the business because of the weakness of the overall performance of cluster and the weak support for JOIN.

Wanda’s technical team also investigated the MySQL master-slave replication, which is widely used in the market, and the MySQL Proxy middleware to realize the sub-database sub-table scheme. However, these schemes cannot meet the business requirements of real-time risk control platform in terms of high availability security, strong consistency, complex transaction/JOIN operation and horizontal expansion capability required by business applications. These problems are mainly reflected in the following aspects:

  • High availability solutions based on MySQL master-slave replication are prone to risks such as access layer fragmentation and data inconsistency.

  • The solution based on MySQL Proxy middleware lacks the support of distributed transaction across libraries and tables after database and table partitioning and good support for complex JOIN, so it cannot meet the requirements of real-time calculation and complex query of risk control rules in business and BI requirements of business teams.

  • The solution based on MySQL Proxy middleware needs to compromise the development of business code, explicitly design and specify the segmentation rules and routing configuration of sub-database and sub-table, and significantly increase the cost of development, transformation, operation and maintenance.

  • In the process of external services with high concurrency and high performance of real-time risk control platform, the related work of online flexible capacity expansion cannot be implemented efficiently and reliably in MySQL Proxy middleware architecture.

Finally, wanda’s technical team, through evaluation and verification, chose TiDB to help them achieve a real-time risk control platform background database system with high performance, high reliability and high scalability.

TiDB products and technical solutions support and assist business requirements, which are mainly reflected in:

  1. With the help of TiDB’s distributed computing and storage engine, the processing capacity of external services of the Cluster is greatly enhanced, and high-concurrency real-time risk control rule calculation can be easily processed. Compared with the original MySQL Galera Cluster solution, the unit processing performance is improved several times. And database clusters gain linear scaling and scaling capabilities.

  2. Overall QPS (from ten thousand level) and Latency (millisecond level) guarantee the real-time requirements of risk control.

  3. There is no need to consider the database and table, and the business application is transparent and non-invasive. Application development and maintenance become intuitive and simple. Even with the rapid growth of business-related data volume and requests, there is no need to worry about the risk of complex application adjustment and operation.

  4. TiDB’s perfect design for distributed transactions and strong consistency as well as support for various JOIN modes enable the business applications of real-time risk control and BI analysis classes to run efficiently.

This real-time risk control platform, with the help of TiDB’s reliability architecture and high performance distributed processing capabilities, has been operating stably in business production environments for more than half a year, during which it has experienced failures caused by environmental problems and stringent tests such as “618” high-concurrency, high-traffic events.

Chen Xinjiang, a big data center technology expert at Wanda Network Technology Group, said: “TiDB’s performance has given wanda’s technical team confidence to build on TiDB and expand its application scale according to its business characteristics, adding complex computing components such as TiSpark, “The integration of CDC tools to improve real-time ETL and TiDB operations and management capabilities continue to play an important role in Wanda’s core business architecture.”