Annual double 11 and double 叒 yi, to technical people the best gift is **** to promote technical guide! But after years of development, promoting has long confined to the electricity industry, now all walks of life actually do operations would adopt a similar manner, car has 818, 618, 11.11, and so on electricity, all kinds of big promote the scene, at the basis of the software, including database proposed many new challenges, but also accumulated a lot of best practices.
\
Before the arrival of Singles’ Day, PingCAP conducted a series of in-depth discussions with autohome, Bitauto, JINGdong, ZTO and other users, hoping to reveal what technical problems are hidden behind the annual soaring sales? What technical architecture can be used to smoothly withstand the flood peak? This article is the second part of “Database topics behind the promotion” : How does TiDB + Easy Car create the Super car Carnival? \
The big Bang carnival in the motor world
Founded in 2000, Bitcar is one of the earliest automobile Internet platform enterprises in China. It provides professional and rich Internet information services for automobile users, and improves the whole experience of users in the process of car selection, car purchase, car use and car exchange.
In this year “818” period, easy car and Zhejiang SATELLITE TV jointly launched a comprehensive car craft show, star song and dance performances and star variety show of the car industry “Spring Festival Gala” – “easy car super 818 car carnival night”. In addition to bringing audio-visual feast and car purchase benefits to car users, the party also offers more than 150 half-price cars with value-added benefits. The audience can grab 50% off cars while watching the party. At the same time, there are multiple discounts such as red envelope for car purchase, deduction vouchers and car price straight down to get real car purchase benefits. By the end of the party, 224 million people watched the live broadcast across the platform, and 43,900 online orders were obtained, with a cumulative turnover (GMV) of 6.42 billion yuan. \
Easy car’s big promotion debut
In the 818 Carnival of Easy Car, there are many application scenarios of database, among which real-time data kanban is one of the main application business. Kanban can display in real time the special topics, activities, traffic, clues, interaction and other data performance of Yiche 818 shopping festival, which is the overall data output of the big data platform.
Since the car carnival night of BITcar is a live event of Tai.com interactive, shaking (red envelope, half-price car, BITCar currency) and voting of the live program in the main venue are the links with the highest user participation and the largest data flow. In the whole process of the activity, the database is not only required to store massive data, but also to cope with high concurrency and low latency scenarios. The database here will not only serve as the data storage medium, but also as the data source of real-time calculation. With traffic data, real-time data broadcasting can be realized at the second level. Database and Flink are two very important components in the whole system. Flink’s data sources include database and business traffic data, so the database should not only meet the real-time push of data in seconds, but also support Flink’s highly concurrent read and write requests. Tian Zhen, the person in charge of easy car database, admitted that easy car network is the first time to promote this year, there is not much experience, the amount is not good to estimate, a lot of demand is put forward at the end. To be on the safe side, the DBA team implemented a downgrade when designing the big push, but no one wanted to actually implement a downgrade because it was too unfriendly to the user experience. So the entire DBA team focused on compression and planned the database compression solution two orders of magnitude (100 times) higher than usual. In the beginning, yi Che’s preferred database was still MySQL. However, in the process of pressure measurement, real-time tasks frequently write large quantities of data to the database to ensure the real-time performance of the calculation results. As a result, the primary/secondary delay of MySQL is high. In extreme cases, the primary/secondary switchover of MySQL takes too long and the database becomes temporarily unavailable. At the same time, real-time tasks write a large amount of data, causing disk overcrowding. That must have been intolerable in the midst of a live broadcast that counted against the clock. In the urgency of the situation, Tian Zhen thought of TiDB. \
“Learn to swim by swimming” TiDB was ordered in the face of crisis
In fact, Tian Zhen has been in touch with TiDB for a long time. At that time, he once thought that TiDB was an overseas product and learned about TiDB mainly from overseas communities. But for reasons of caution, Tian zhen wants to thoroughly research the product before launching it officially. The promotion was a perfect opportunity for the two sides to work together, a process he described as “learning to swim by swimming”.
The technical support of TiDB community has given bitche DBA very important help. The project has been officially established since July.It only took less than a month to complete the selection, scheme design, pressure test and online deployment, and in the “818” will be a big promotion flow smoothly carried over.
818 Auto Carnival Data Kanban business architecture diagram
Throughout the 818 campaign,TiDB is used as the core database of 818 Car Carnival data Kanban. Easy car has prepared two sets of TiDB clusters, corresponding to the master and standby schemes of real-time computing. Service R&D simultaneously writes data to two clusters in dual-write mode. One service query is connected to cluster 1, and the other service query is connected to cluster 2. When a fault occurs in one cluster, the application ends switch to the other cluster. Both TiDB clusters are deployed with 3 TiDB servers, 3 PD Servers, 6 TiKV nodes and 2 TiFlash nodes. In addition, four machines are being prepared for capacity expansion in case the data surge overwhelms the cluster.
In the end, easy car 818 car carnival data volume reached more than 10 times the usual, in the live final CAI Xukun appearance, the database flow is directly increased four times, almost used in advance to prepare the bottom of the one-key expansion plan. In the whole process, THE 818 Auto Carnival data kanban business SQL 999 is always controlled within 8ms, SQL 99 is around 3ms, and QPS reaches 62K.
Shake the business architecture chart
At the same time, TiDB is also used as a disaster recovery (Dr) solution in the red packet shaking service to avoid MySQL being unavailable due to service traffic surge. In the event of unavailability, the business side can switch the database directly to TiDB. TiDB needs to act as data source, real-time computing dimension table and real-time computing result storage engine in the whole business. TiDB pushes data to Kafka in real time through TiCDC. In order to ensure the stability and efficiency of TiCDC, Bitcar creates a TiCDC task for each library in TiDB to push data to the specified Kafka in real time. Then Flink is responsible for parsing the data belonging to different library tables in the same TOPIC and distributing them to the corresponding TOPIC of the library table for real-time computing business use. Real-time computing tasks consume TiDB data in Kafka for business logic calculation, and also need to query the corresponding dimension data from TiDB, and finally output the calculation results to TiDB. \
The challenge of high growth: technology stack unification
In the rapid development of EASY Car, many businesses have introduced many technology stacks in order to quickly iterate and go online, such as big data architecture such as Lambda and Kappa, real-time data warehouse such as Kylin, Druid and Clickhouse, etc. However, the DBA team of BITche was only six people, and managing so many technology stacks was undoubtedly a big challenge.
The unified technology stack became the best choice for bitche’s DBA team. With this opportunity, bitche wanted to replace Kylin, Druid, and Clickhouse with TiDB, simplifying the technology stack and allowing the DBA team to focus on their full-time work. The HTAP architecture of TiDB is a fusion architecture that combines transactional transactions and analytical processing. Because they are all in the same architecture and the same set of data, it solves the problem of delay of data flow of BITcar real-time data warehouse. The data is no longer copied from the OLTP database and entered into the analysis tool through a lengthy ETL cleaning process. TiDB’s perfect compatibility with MySQL means that no changes are required for DBAs and developers, as long as they know SQL. In the past, Hadoop or Spark had high learning costs, which caused some barriers to use. After this battle, easy car business on TiDB add a lot of expectations and trust. In the future, Advertisements, media platforms, websites, release data and advertising effects of BITcar hope to be able to see in real time. Tian Zhen hopes to use TiDB to cover the whole scene of bitcar hybrid technology stack to get through with other data flows. All these need TiDB HTAP to support real-time data warehouse.
For enterprises, the promotion is not only to support business innovation, but also to practice their own technical architecture and full-link exercise. Through the ultimate test of promotion, the enterprise’s IT architecture, organizational process and talent skills have been greatly improved. The experience and thinking in the promotion process will also accelerate the pace of business innovation, improve the efficiency of technology-driven innovation, and create a new engine for growth. \