“We’ve already used it” is one of our favorite words. Behind these simple words is a lot of trust and commitment. From today, we will share in-depth cases through ** “Believe in the Power of Openness” series to see the business value that a database brings to users in various industries from a business perspective. This article will introduce the application and practice of TiDB in 360 network shield business, intelligent business, advertising material data business and other core scenarios.
Technology as the driving force to make the world safer and better
Founded in 2005, 360 Is a leading Internet and security service provider in China. It has launched 360 Security Guard, 360 mobile Security Guard, 360 security Browser and other security products.
With the deepening of the digitalization of the whole society and the whole industry, the era of “great security” is accelerating. With the mission of “making the world safer and better”, 360 is committed to realizing the vision of “constantly creating black technology and being an all-round guardian”. The business development that follows is also more comprehensive, including: Government and enterprise services, fintech, livestreaming, personal services, smart wear and other businesses.
Business challenges
Net shield business
360 net shield is a free Internet protection software, which can block Trojan horses, fraudulent websites and so on to protect consumers from viruses and fake websites. Its operating mechanism is to analyze each URL, after the identification of the URL detection system, it can accurately identify which category the URL belongs to. When problematic urls are published to the cloud, other security service providers can subscribe to cloud services to detect the security of relevant urls and provide more secure web content services to their users.
At present, the core scenarios of the whole network shield business can be divided into the following four parts:
-
Website threat monitoring: every day into the database of 100 million data, more than 800 million resource link data;
-
Association analysis scenario: Conduct association analysis of large-scale malicious websites, pornography, gambling, drug and other websites;
-
High-speed returns: 89.7 billion tables, 100+ queries per scenario need to be returned within 5 seconds;
-
Manual operation analysis: every day everyone repeatedly query statistics, analysis.
Due to the explosive growth of data, there is a bottleneck in reading and writing MySQL. For example, although the disk space can be separated by database and table, it is not a small amount of work for services and DBAs. The most painful changes to these large tables can be very costly to perform DDL on a large table. In general, MySQL has been unable to meet the requirements of shield business, which puts forward new selection requirements for 360 Cloud Platform technical team responsible for the support of the underlying data platform.
360 Cloud Platform is responsible for providing service support for 360 Group’s major business lines, involving database support schemes such as MySQL, Redis, MongoDB, ElasticSearch, Greenplum and PiKA. After thorough market research, the 360 Cloud platform team decided to introduce TiDB to meet the business needs.
The overall structure is as above, there are two main types of businesses using TiDB:
-
Original MySQL services are migrated. Due to the limitation of single-node disks, single-instance disks cannot support the explosive growth of data. Therefore, data is important and needs to be backed up and restored 24 hours a day. This type of business realizes barrier-free migration through DM suite, and the import time of 1TB is 16 hours. If it is a large data source and TiDB is a new cluster, TiDB Lightning can be used for data import, and the speed can reach 100G/ hour.
-
Brand new business. At present, all new businesses will be put into TiDB, and the data volume of such business is generally large. At present, there are many tables of net shield business with the level of 1 billion, among which one table reaches 10 billion +.
Smart business
360 Smart Business relies on Internet products that cover all scenarios of users to provide enterprises with full life-cycle services. Through intelligent marketing, enterprise service, innovation platform and other diversified business layout, it can meet the multi-dimensional growth demand, comprehensively connect users and enterprises, and create a smart business ecology of coexistence and growth. Among them, Internet advertising is an important way of commercial realization of traffic and an important revenue source of 360 Group. It involves enterprise service platform, advertisers, algorithm strategy, data engineering and other directions. In the process of advertising, real-time/offline reporting and advertising materials are the most important and core business for advertisers.
Advertiser real-time & offline reporting business
Real-time reporting business process for advertisers: Business data into Kafka, one processing link is through Druid to obtain Kafka data for real-time analysis, the other link through Flink for aggregation into the MySQL sub-database sub-table database, and then through the advertiser dimension to provide real-time query requirements.
Offline report business process of advertisers: Every morning, data warehouse extracts all data of the previous day from MySQL sub-database sub-table into Hive, uses Spark or service program to collect statistics and then writes the result data into MySQL result table, and provides it to offline report platform or Console platform for query.
As the core business of advertising platform, report is faced with the following problems:
-
Large amount of data: total data of 100 billion level, single table data volume of 120 ~ 150 million.
-
Low response delay: Real-time feedback is required for users’ queries of arbitrary periods and keywords.
-
Query complex: time dimension, region, industry, keywords and so on, at the same time to meet the diversified display.
-
Complex architecture: MySQL based sub-database sub-table can not carry out global statistics, can only be based on the UID of advertisers to produce detailed reports, global statistics need to introduce Druid to assist processing; Offline reports require Hive data warehouse to extract full data to achieve.
Database selection: MySQL or TiDB?
Before TiDB was deployed, 360 tried single-instance MySQL to meet business requirements. After the test, it was found that single-instance MySQL was under great pressure. In order to disperse the writing pressure, it was necessary to follow the old way of MySQL database and table division. Every rule changes may involve data to move, and the business side also need to invest a lot of manpower to maintain routing rules, and to meet the demand of advertisers report need to introduce other database, offline ETL for extracting MySQL nic load in the morning every day, will also affect the other business operations in the morning.
After TiDB is deployed, TiDB’s good expansibility completely solves the problem of database and table. At the same time, after performance pressure test, data storage of 150 million in 2 hours (TPS: With TiFlash (TiDB real-time analysis engine plug-in), we can conduct global and detailed real-time analysis of all dimensions of the merged single table, and realize online statistics of offline reports, eliminating the T+1 timeliness and synchronization process of offline data warehouse. It also provides a financial level of consistency.
Advertising material data service
For advertisers, in the search marketing, based on a new generation of safe, accurate and reliable search engine search, 360 through keyword matching technology, locate the target users, accurately show enterprise promotion information, material creative role is to help advertisers appeal to dive, and resulting conversion behavior, such as registration, submit orders online, visit, telephone consultation, and so on.
At present, the material platform of 360 advertising carries the information of pictures, texts and videos produced by customers, and supports the uploading, downloading, importing, exporting, adding, editing and deleting of materials at various levels of promotion accounts, promotion plans, promotion groups, keywords, promotion ideas and advanced styles.
Before TiDB is used, 16 sets of sub-database * 4 MySQL architecture are used at the bottom of the point material data, each sub-database MySQL single table has reached 1 billion, single table data size is 370G, for single database QPS more than 10,000 SQL requests, MySQL tables have reached the performance bottleneck. In addition, new services want to add fields to large tables during peak hours. Due to insufficient disk space, new services cannot be launched. If we continue to use MySQL, we need to split the current 16 sets of sub-libraries into 64 sets of sub-libraries (about hundreds of new servers are needed). Besides the new servers, migration, operation and maintenance costs are also very high.
The technical team of 360’s commercial business line finally decided to take TiDB as the database at the bottom of the material platform through research on technology selection. At present, the TiDB cluster scale supporting the material platform is 63 nodes, and the daily SQL requests are more than 7 billion times, and the QPS of the singles’ day just passed is 25 W/s. 99% of the SQL requests on weekdays are within 15ms, which has achieved the expected fast response, stability and scalability.
The benefits from TiDB deployment are as follows:
- Cost savings
-
40% server cost savings compared to the previous MySQL deployment mode.
-
In the past, the routing rules of service maintenance were divided into database and table, but now they are all a table for services, which improves the development efficiency of services and enables r&d to focus more on services.
-
TiDB provides a rich tool chain ecosystem covering data migration, synchronization, backup and other scenarios, improving operation and maintenance efficiency.
- Performance improvement
-
618, Double 11 QPS can be up to 25 W/s, 99% of SQL requests on weekdays are in 15 ms.
-
Compared to the VIP switch-based MySQL master-slave architecture, TiDB has over 99.95% availability.
-
Pure distributed architecture, with good scalability, support flexible capacity expansion, throughput and storage can be smoothly expanded online, database expansion capacity increased by 1 to 2 data levels.
In the future, 360 technical team also plans to put TiDB on 360 HULK cloud platform to provide stable and reliable database services for 360 Group’s security brain, 360 games, core security service platform and other business lines.
Walk with customers, believe in the power of openness
Every time database architecture improvement and implementation, no matter TB or PB level, need to pay efforts, but it is also worth every enterprise to practice. In this era, regardless of the size of the enterprise, we should learn to use the power of open source to avoid repeating the wheel. Behind every seemingly easy, there are unknown efforts, behind every seemingly bright, there are unknown efforts. The road to build distributed database is long and difficult. TiDB is willing to work with 360 and every customer to do things well.