This article is based on Xiao Bo’s speech at the 2021 Vivo Developer Conference. Public id reply ** [2021VDC] ** To obtain information related to the topics of the Internetworking technology sub-session.

I. Background of database and storage platform construction

Take history as a mirror, we can know the rise and fall, do technology is the same, before introducing the platform, we first review the development of Vivo Internet business in recent years.

Let’s go back three years to see the development of Vivo’s Internet products in recent years. In November 2018, the total number of vivo mobile Internet users exceeded 220 million. In 2019, the daily activity of Internet applications such as app store, browser, video and wallet exceeded 10 million. In 2020, the daily usage of browsers will exceed 100 million, and in 2021, the total number of online users (excluding export sales) will reach 270 million. Dozens of applications will live more than 100 million a month, and the number of database and storage products will also reach 4,000 + servers and 50,000 + database instances.

What did databases and storage platforms look like three years ago?

If there is a word to describe the current situation of database services in 2018, I think “at risk” is the most suitable, mainly for the following points;

• The availability of the online database environment is often affected by inefficient SQL, human misoperation, unreasonable infrastructure, and the robustness of open source products.

• Non-standard change, low efficiency of change and various operation and maintenance operations, no platform support, using command line terminal for change.

• The cost of using databases is extremely high, and many additional costs are added to cope with increasingly complex business scenarios. This is the current state of Vivo’s database in 2018.

• The security ability is not sound enough, and the data classification and classification, password and account permissions are not standardized.

Let’s take a look at some operational data trends in Vivo database over the years.

From the end of 2017 to the beginning of 2018, the scale of database instances increased by nearly 5 times, the scale of database servers maintained increased by 6.8 times, the single-machine deployment density of database instances increased by more than 5 times, and the database instance scale operated and maintained by per DBA increased by 14.9 times.

Through the above figures, we find that Vivo’s Internet business is in a state of rapid development in recent years. In the process of rapid development, it is urgent to solve the problem of data storage, whether from the perspective of service quality experienced by users or internal cost efficiency. Therefore, in 2018, we launched the plan of self-research database and storage platform. After several years of construction, we have preliminarily acquired some capabilities. Now, we will give you a brief introduction of these capabilities.

Ii. Database and storage platform capacity building

First of all, the overall database and storage platform products are introduced, mainly divided into two layers.

• The first layer of our database and storage products, including relational data, non-relational database, storage services three blocks.

• The second layer is mainly tool products, including database and storage unified control of r & D and operation and maintenance platform, data transmission services, operation and maintenance of white screen chemical tools, and some SQL audit, SQL optimization, data backup and other products.

Tool products are mainly self-developed. For database and storage products in the next layer, we will give priority to mature open source products, and also develop some products based on open source products or purely self-developed products to better meet business development. The following is a brief introduction of the capabilities of some products.

DaaS platform is short for Database as a Service. It aims to provide a platform for the use and management of highly self-service, highly intelligent, highly available and low-cost data storage, covering the whole life cycle of Database and storage products from Service application, deployment, maintenance to offline. Mainly from four aspects to provide value for the company and users.

• The first is to improve the availability of database products, through preventive inspection, monitoring, planning, fault tracking and so on to prevent the failure in advance, timely processing, after the review summary of the whole process closed loop.

• The second is to improve research and development efficiency. The research and development self-service database provides change detection, optimization diagnosis and other functions to reduce manual communication process, standardize the project change process, and improve the research and development efficiency.

• The third is to improve data security, through permission control, password control, data encryption, data desensitization, operation audit, backup encryption and a series of means to comprehensively ensure data security.

• The fourth is to reduce the operating costs of database and storage products. Firstly, the automated process can reduce the repetitive work of DBAs and improve human efficiency. Secondly, the resource utilization efficiency of database and storage services can be improved through service orchestration and resource scheduling to continuously reduce operating costs.

After several years of construction, some progress has been made in the above work, among which thousands of demand work orders are issued every month, more than 90% of which can be completed by the r&d students themselves. The service availability has been maintained at more than 4 9 in recent years, and the platform support for 6 database products and storage services has reached more than 85%. In addition, the same capability covers all database scenarios, such as data changes. We support pre-change statement review of MySQL, ElastiSearch, MongoDB, and TiDB, change data backup, one-click rollback of change operations, and change record audit tracking, etc.

Vivo’s DTS service is a data transmission service developed by vivo based on its own business needs, mainly providing data interaction between DATA sources such as RDBMS, NoSQL and OLAP. The integrated service of data migration, subscription, synchronization and backup has three main features in terms of function.

• The first is to ensure the stability of synchronous links and data reliability. By combining the characteristics of each data source product, data can be neither lost nor redistributed, providing 99.99% service availability guarantee for services.

• The second is the functional level to support heterogeneous multiple database types, in addition to synchronization, migration, subscription and other general functions, we also support centralized storage and retrieval of change data.

• The third layer is fault disaster recovery (Dr), which supports node-level fault disaster recovery (NODE-level Dr). It can achieve synchronous link second-level recovery and resumable transmission, which can effectively solve the transmission interruption caused by hardware and network exceptions.

Let’s take a look at some of the projects we’ve been working on in the underlying datastore layer, starting with the MySQL database.

MySQL, as the most popular database, also undertakes the heavy task of relational database service in Vivo. The MySQL2.0 in the figure above is our internal architecture version. In the past few years, our architecture has evolved two versions.

In order to quickly solve the usability problems at that time, version 1.0 was made based on MHA+ proprietary components.

At present, it has evolved to version 2.0, and MHA and other components are no longer dependent. In terms of architecture, the service access layer of version 2.0 supports business access using DNS or name service. In the middle, a self-developed Proxy layer is added, which is 100% compatible with MySQL syntax and protocol. On Proxy, we also realize three-level read/write separation control, traffic control, data transparent encryption, SQL firewall, log audit and other functions.

Proxy layer combined with the high availability components at the bottom realizes the automatic and manual failover of MySQL cluster. RAFT mechanism ensures the availability of the high availability control components themselves. Of course, MySQL still uses master-slave architecture, which can be deployed across IDC in the same region. Multi-live across regions is not currently supported, which is part of the planned 3.0 architecture.

As a very popular and excellent KV storage service, Redis has been widely used in Vivo. In the development of Vivo Internet, both the stand-alone version of Redis and the master-slave version of Redis have been used. So far, we have all upgraded to cluster mode, and the automatic failover, elastic scaling and other features of cluster mode have helped us solve many problems.

However, there are still many problems when the scale of a single cluster is expanded to TB level and the number of nodes in a single cluster is expanded to 500+. In order to solve these problems, we have made some transformation and development on Redis, mainly including three parts:

• The first problem is the availability of multiple computer rooms of Redis Cluster, for which we developed a live version of Redis based on Redis Cluster.

• Second, there are enhancements to Redis data persistence, including the AOF logging revamp and the introduction of AEP hardware, including the planned Forkless RDB.

• The third is the enhancement of Redis synchronization and Cluster mode, including asynchronous replication, file cache, water level control, etc., as well as the optimization of the time complexity of Redis Cluster instruction. The time complexity of Redis Cluster instruction once brought a lot of trouble to our operation and maintenance. The time complexity has been greatly reduced and this piece of code has now been synchronized to the community.

After we made these optimizations on Redis, we found that the memory KV storage alone could not meet the business needs. In the future, there will be a larger storage scale, so it is necessary to have disk-based KV storage products to distribute data and store data in layers. Therefore, we developed disk-based KV storage products.

When we started the disk KV storage service research and development project, we made clear the basic requirements of the business for storage.

The first is compatible with Redis protocol, which can be easily switched from the original project using Redis service.

Second, low storage cost, large storage space, high performance, combined with some basic requirements of O&M, such as automatic failover, rapid capacity expansion and shrinkage.

Finally, we chose TIKV as the bottom storage engine to realize the disk KV storage service, we encapsulated Redis instructions and Redis protocol in the upper layer. Another reason to choose TIKV is that TiDB products are used in our overall storage product system, which can reduce the learning cost of operation and maintenance personnel and enable them to learn quickly.

We also developed a series of peripheral tools, such as Bulk load Bulk import tools, support from the big data import data to the disk in the ecological KV storage, data backup restore tool, Redis to disk KV synchronization tools, etc., these tools greatly reduces the migration of business cost, at present our KV disk storage products have been widely used in internal, Multiple TB-level storage scenarios are supported.

We know that in the process of business operation, in addition to some structured or semi-structured data access requirements, there are also a large number of unstructured data access requirements. Vivo object and file storage service is built under this background.

The object and file storage services use a unified storage base, and the storage space can be expanded to EB level or higher. The upper layer exposes the standard object storage protocol and POSIX file protocol. Services can access files, pictures, videos, and software packages using the object storage protocol. The standard POSIX file protocol allows businesses to expand their access requirements just like using local file systems. For example, HPC and AI training scenarios can support GPU model training of tens of billions of small files.

For image and video files, it also extends some common image and video processing capabilities, such as watermark, thumbnail, cropping, clip, transcoding and so on. Previously, we briefly introduced some product capabilities of Vivo database and storage platform. Now let’s talk about our exploration and thinking on some technical directions in the process of platform construction.

Third, database and storage technology exploration and thinking

In terms of platform construction, it is a commonplace topic to improve the efficiency of O&M RESEARCH and development. There are also many well-built platforms and products in the industry, but there is little talk about how to improve the efficiency of O&M research and development in data storage.

Our understanding is:

• First of all, the delivery of resources should be agile enough, and enough low-level technical details should be shielded. For this purpose, we will carry out unified management of IDC self-built database, cloud database and cloud host self-built database, provide unified operation view, and reduce the use cost of operation and development.

• Secondly, to improve efficiency, we should not only focus on the production environment, but also need effective means to unify the control of research and development, testing, pre-release, production and other environments, so as to achieve unified experience and secure isolation of data and permissions.

• Finally, we applied the idea of DevOps solution to logically divide the whole platform into two domains, one is r&d domain and the other is operations domain:

In the research and development domain, we need to think about how to solve the efficiency problem of the research and development students about the database and storage products. It is not enough to deliver a database instance and support them to build database tables on the platform. Many operations occur in the coding process, such as constructing test data, writing logical code for adding, deleting, changing and checking, and so on.

We want to interact with our platform during these processes to maximize r&d efficiency.

In the field of operation and maintenance, we believe that a good indicator is the number of login server operations in daily operation and maintenance, so that all operations of operation and maintenance can be standardized and automated, and some operations can be intelligent in the future. In the interactive part of r&d and operation and maintenance, our construction goal is to reduce the interaction. The less people involved in the process, the higher the efficiency, so that the system can make decisions, and the solution is to do self-service. Let’s take a look at some of the explorations and reflections on security.

Security is no small matter, so we will database security and data security separately out of the planning and design, the basic principle is clear rights and responsibilities, database system involves account passwords and so on.

We worked with the SDK team to jointly develop a password encryption and transmission scheme. The database password is ciphertext for r&d, operation and maintenance, and is still ciphertext in the project configuration file, which is used to decrypt the company’s key management system.

Joint security team on the part of the data, we made automatic tagging for sensitive data identification and classification of grading, to sensitive data query, export, made a strict control on changes, such as access control, permissions, upgrading, through the digital watermarking technology using the tracing, who through after the audit can be traced back to see what data at any moment. For sensitive data we have also done a transparent encryption and decryption operation, the data dropped to the storage medium is encrypted storage.

Similarly, backup data and logs are also encrypted and stored, which is what we are doing now, and there are still a lot of capabilities to be built in security in the future. Now let’s look at changing this one.

For data change scenarios, we focus on two main points;

• The first is whether the change of data itself will affect the existing business. For this purpose, we have built an unlocked table structure and an unlocked table data change capability, and set up three lines of defense for the three links before, during and after the online operation. To prevent some bad SQL or Query from flowing into the production environment, we also made a one-click rollback scheme for rollback during or after the change.

• The second problem is the change efficiency. For multiple environments and clusters, we provide a one-click synchronization data change solution. At the same time, in order to better improve user experience, we also provide a GUI library table design platform. With these foundations, we opened the whole scene to the r & D students. Now the r & D students can carry out data change operations independently 24 hours a day, greatly improving the change efficiency.

Explore and think

Now let’s talk a little bit about cost.

We mainly manage the cost from four aspects;

• The first is budget management and control. Resources are de-physicalized. Business conducts budget submission based on the granularity of resources.

• The second is the deployment of database service, which has gone through several stages. In the early stage, we used single instance, which wasted a lot of resources. Later, we developed into standardized package deployment, which mixed different packages for the same type of storage resources, and continuously improved the efficiency of resource use through algorithm optimization.

• Third is a mixture of we have done a series of different attributes of resources deployment, such as a database agent layer and the data object storage nodes deployment, this two kinds of resource is the CPU type, one is the store type, can complement each other, again in the future development of the next stage should be a cloud native stored separately calculated, is still in exploration.

• Fourthly, after service deployment, it is necessary to pay constant attention to the status of operation, inspect and warn the capacity, and carry out the lifting and configuration operation of the cluster in time to ensure the orderly operation of the whole operation. Pay attention to service running status and reclaim offline data storage clusters in time to reduce zombie clusters.

The cost aspect is also the iteration of hardware resources, which is also critical and will not be discussed here. And then let’s look at the storage service architecture.

Object and file storage we’re going to focus on two things;

• The first one is cost. In terms of cost, we used EC in data redundancy strategy and made EC across IDC. The failure of a single IDC will not affect our data reliability. We also introduced high density and large capacity storage server, as much as possible to improve the storage density of single rack, it should be noted that the running cost of server procurement can not be ignored, there is still a lot of room for optimization. We also provide data lossless and transparent compression capabilities and lifecycle management capabilities, timely clearing of outdated data and archiving of cold data. Continuously reduce storage costs through multiple means.

• The second is performance. In terms of performance, we provide bucket and object granularity IO isolation for the underlying storage engine. By introducing some open source components such as Alluxio, we provide server-side + client-side caching to improve the performance of hot data reading. Further improve disk I/O throughput.

The above is some exploration and thinking of our current capacity building, and finally, let’s look at our future planning.

5. Future planning

At the storage service layer, we will continue to improve the storage service matrix, improve products, and provide more diversified storage products to better meet business development demands. At the same time, the storage service layer will do some SAAS services based on existing storage products to meet more business demands. In the functional layer, we disassemble it into four parts:

• Basic data services provide basic functions of storage products, including deployment, capacity expansion, migration, alarm monitoring, backup and recovery, and offline recovery.

• Data services and storage products are essentially carriers of stored data. For data itself, we also have some specifications, the most basic such as data query and change performance optimization, data governance and how to go deep into the business coding process.

• Autonomous storage service is initially divided into four parts: performance autonomy, capacity autonomy, intelligent diagnosis and scene service. Autonomous storage service can not only improve the happiness of DBA work, but also greatly improve the robustness and stability of our system itself.

• Data security services, although the construction of some capacity, but not enough system, the future needs to increase investment.

In the future, the entire storage service system will be integrated into the company’s overall hybrid cloud architecture to provide users with one-stop and standardized experience. And that’s all for sharing.

Author: Xiao Bo, Vivo Internet Database Team