Kunlun wanwei is a global leading and cutting-edge comprehensive Internet group, with its business covering GameArk, Opera and other business segments. Among them, Kunlun Games (GameArk) with the core advantages of R & D and operation, for the global game r & D, release and operation, the formation of a diversity of product matrix. By the end of 2020, monthly live users have reached over 100 million.

Facing the challenge

Game businesses generate a lot of data during operation, such as item generation and consumption, currency generation and consumption, player level distribution, mission completion, map player distribution, player combat data, etc. With the deepening of business development and refined operation, more and more data dimensions are required for market release and operation, which are becoming more and more detailed and begin to show explosive growth. How to store and analyze these data has become an urgent problem to be solved. In complex scenarios, Kunlun Games began to use a variety of databases to support business, from the traditional relational database to NoSQL, NewSQL, as well as various data analysis systems and tools, which brought severe challenges to the operation and maintenance and management level.

Upgrade from MySQL Cluster to TiDB

Since Kunlun games expanded its business in 2008, the inheritance of technology not only brings the accumulation of experience, but also leads to the generation of technical debt. Kunlun game data analysis system provides real-time game summary data for business, such as gold consumption, props purchase distribution, etc. The game business will send the generated information to the specified interface in the specified format in real time, and the MySQL database will store the reported data. In the early stage, the database runs stably under the condition of small amount of data. With the rapid development of services, when the data in a single table of MySQL exceeds 50 million, data storage and real-time query become slow and cannot meet service requirements.

After comprehensive consideration, Kunlun Games decided to smooth transition and upgrade the existing MySQL program on the basis of not reconstructing the existing business side code as far as possible. The first step is to optimize SQL queries to find the cause of slow queries. With the increase of single table data volume, Kunlun games found that the improvement of SQL execution efficiency is not obvious. If MySQL is divided into database and table according to time, a series of problems will follow, such as cross-node query Join, cross-node paging, sorting, functions, etc. The scheme is too complex, and the business side code needs to be rewritten. This is essentially a “patching” approach to horizontal scaling.

In order to better solve the single point of failure problem, kunlun games database form from the original single MySQL library, MySQL master slave, one master and many slaves, upgraded to MySQL Cluster. MySQL Cluster is an enterprise database solution officially provided by Oracle. It adopts the architecture design of Shared Nothing. The advantage of this solution is to solve the previous single point of failure of MySQL, provide automatic failover, and have better scalability and higher processing performance.

However, the MySQL Cluster solution also has significant disadvantages. First, the cost is high. The data nodes of the original version of MySQL Cluster store the data in memory, and the memory of a single node usually needs to be configured to 512GB or more. As a result, you can expect hardware costs to rise rapidly as the volume of data increases, requiring more nodes. Secondly, MySQL Cluster has many limitations, such as no support for foreign keys, data rows cannot exceed 8K, etc. Third, the backup and recovery of the scheme is not convenient. Data recovery requires IT personnel to execute in the form of typing instructions on each data node, and cannot be backed up to cloud storage such as S3. In addition, the deployment, management and configuration of the entire system are relatively complex and cannot be perfectly matched with the existing cloud-based IT architecture.

Kunlun games hopes to use the cloud native NewSQL database to match the performance, scalability and cloud deployment and operation and maintenance requirements. After investigating the mainstream NewSQL database in the market and testing the real business scenarios, Kunlun Games decided to choose TiDB to expand the capability map of the data architecture system.

TiDB solution

Based on the public cloud, Kunlun Games has built a set of TiDB distributed database cluster supporting high concurrency, high availability and horizontal expansion to realize the centralized management of game business data and provide data services for reporting, monitoring, operation, user portrait and big data computing.

By using TiDB Data Migration (DM) tool, Kunlun games synchronizes Data from multiple MySQL libraries to TiDB cluster, and provides theoretically unlimited storage capacity with TiDB’s horizontal expansion capability. At present, the scale of TiDB cluster data of Kunlun games has reached dozens of TB. In the future, Kunlun Games will explore more business scenarios combined with its own business needs and the new features of TiDB.

Logical architecture diagram

Why TiDB?

By constructing TiDB data service platform, Kunlun Games solved the problems of the original database system in performance, expansibility, cloud deployment, operation and maintenance, and improved the ability of business-oriented data service. Its main advantages are mainly reflected in the following aspects:

  • TiDB is highly compatible with MySQL, and the original business code basically needs no change to achieve smooth switching of business;

  • In terms of architecture and performance, the architecture design of TiDB storage and computing separation breaks the capacity and performance bottleneck of a stand-alone database

  • In order to achieve elastic expansion of performance and capacity, read and write QPS and average latency fully meet business requirements;

  • In terms of ecology, TiDB has a strong ecosystem that supports the connection of Spark, Flink, Kafka and other data application ecosystems to create a digital service platform for the business in all scenarios.

  • In terms of cost, TiDB supports cloud deployment, operation and maintenance, and seamlessly connects with S3. After moving the entire data analysis system to cloud deployment, the hardware cost is reduced by more than 50%.

“Our goal is to put the data services become more agile: one is through the distributed architecture improve the service efficiency of database, the database to better meet the needs of the business; 2 IT is to let the database application layer and the underlying decouple IT technology, reduce the dependence on the underlying technology, upper application to reduce the overall construction and operational costs. IT ‘ — Liu Han, DIRECTOR of IT Operation and maintenance of Kunlun Games