December 13-14, by the cloud community and Alibaba Technology Association co-hosted the “2017 Alibaba Double 11 technology twelve lectures” successfully ended, focused on sharing the black technology behind the 2017 Double 11. This paper is the compilation of the speech “The Application of the New generation of Database Technology in The Double 11”. This paper mainly starts from the cloud and elastic scheduling of database, and focuses on sharing the new generation of database and its application in the Double 11, including X-DB, X-KV and ESDB. It reads as follows.
Sharing Guests:
Zhang Rui: Researcher of Alibaba, head of database technology team of Ali Group, experienced the technological transformation of Ali database, and participated in the preparation for Double 11 as the general head of database for six consecutive years.
Double 11 is a technical training, is the Internet industry’s super project. Need to support as high as possible zero peak, to give users the best experience; Also to do as low as possible cost, extreme flexibility; But also to achieve overall system stability.
How do databases achieve extreme resiliency
Cloud on database
Database resiliency is difficult, database performance requirements are very high, therefore, must be implemented on the cloud database, but how to cloud?
The cloud in the database faces the following difficulties:
1. How to get the database into the cloud and quickly build a hybrid cloud?
2. How can I reduce performance loss caused by virtualization?
3. The communication between the public cloud environment and the internal network is abnormal.
After several years of exploration, these difficulties have been solved. First, the performance of high-performance ECS can be the same as that of physical machines, mainly using SPDK, DPDK technology and NVMe storage, so that virtualization loss is very small, close to physical machines; Second, the problem of flexible hybrid cloud of database can be solved, so that both the upper and lower cloud environments can be managed at the same time. Users can build the hybrid cloud before Singles’ Day to support the peak value of singles’ Day.
Flexible database scheduling
Only on the cloud is far from enough, but also from online mixing. The two basic conditions of database elastic scheduling are containerization and separation of computing and storage. Container performance needs to be on par with physical machines, storage computing separation depends on hardware development, 25G networks and high-performance distributed storage Pangu make it possible.
The database storage and computing separation architecture is shown in the figure, including storage layer, network layer, and computing layer. The storage uses The Distributed storage system pangu developed by Alipouch, and the database computing node is deployed in the Alipouch container. In addition, there is a storage control system.
To achieve the separation of storage and computation, we have done a lot of work on storage, including:
- Two or three asynchronous: the third copy is completed asynchronously, reducing the average latency by more than 10%, and reducing the latency by 3-4 times in 4 copies.
- QoS flow control: Controls background I/O traffic according to foreground service loads to ensure write performance.
- Quick Failover: Single storage cluster FO optimized to 5s, reaching the industry leading level.
- High availability deployment: A single cluster of four Rack deployment improves data reliability to ten nine.
We also made a number of optimizations on the database side, most importantly reducing network throughput to reduce the impact of network latency on database performance. For example: Redo Sync optimization, increased throughput by 100%; Since Pangu storage supports atomic writes, we turned off the Double Write Buffer to improve throughput by 20% and save bandwidth by 100% under high pressure.
Double 11 database mixing technology
Containerization and storage computing separation, making the database stateless, with scheduling ability. During the double 11 peak, the shared storage is mounted to different computing clusters (offline clusters) to achieve fast database elasticity.
Ali new generation database technology
Ali was a commercial database at the beginning. Then we went to IOE and developed ali MySQL branch AliSQL and distributed middleware TDDL. In 2016, we began to think about the new generation of database technology X-DB, X stands for the pursuit of extreme performance, challenge the meaning of infinite possibilities.
Ali’s business scenario has high requirements for the database:
- Data should be extensible;
- Continuous availability and strong consistency of data;
- Large amount of data, high degree of importance;
- The data have obvious life cycle characteristics, and the cold and hot data have distinct characteristics.
- Transaction, inventory, payment and other businesses, simple operation logic, high performance requirements.
Therefore, defining a new generation of databases includes several important characteristics: strong data consistency and global deployment capability; Built-in distributed, high performance, high availability; Automatic data life cycle management capability.
X – DB architecture diagram
X-db architecture is shown in the figure, and Paxos distributed consistency protocol is introduced to solve the problem. Can be deployed in different places, although the network delay increases, but can maintain high performance (throughput), in the same city three-node deployment mode, the performance is equal to that of a single machine, and has high tolerance of network jitter.
X-paxos is the core of the three-node capability, which can achieve strong data consistency across AZs and regions and achieve a continuous availability rate of more than 9 in 5.
X-db core technology 2: Batching & Pipelining. During transaction submission, X-DB must ensure that logs are received and submitted by the majority of database nodes, which is the basis for ensuring strong consistency of data. Since transactions must cross the network when they are submitted, this will definitely lead to increased latency, and it is very difficult to ensure throughput under high latency. Batching & Pipelining technology ensures as much batch delivery as possible, data can be received and validated out of order, and logs can be delivered sequentially. Can maintain high throughput under high delay conditions.
X-db core technology 3: Asynchronous submission, the database thread pool will wait during submission, in order to maximize performance, we use asynchronous submission technology, maximum possible to ensure that the database thread pool can work efficiently. These technologies ensure high throughput of X-DB in three-node mode.
X-db vs. MySQL Group Replication
Let’s compare with Oracle’s official Group Replication. Sysbench standardized tests in three nodes with IDC deployment mode. In the Insert scenario, we can achieve 2.4 times the official MySQL response time and lower than the official MySQL response time.
Sysbench standardized tests in a three-node, three-place deployment mode. In Insert scenarios, the performance advantage of X-DB (50,400) is particularly obvious, 5.94 times that of MySQL GR (80,500), and the response delay of X-DB (58ms) is 38% that of MySQL GR (150ms).
Typical Application Scenarios
Intra-city cross-AZ deployment replaces the traditional active/standby mode. We changed the original active/standby mode into three nodes to solve cross-AZ data quality and high availability problems. Data is consistent across AZs. No data is lost in a single AZ. No second switchover is available in one AZ. No cost increase compared with active/standby mode.
In cross-region deployment, lower-level database technology is used to solve the problem of multiple live events in different regions. In active/standby mode, six copies of data in three regions are reduced to four copies of data in three regions (five nodes and four data in three regions). For services, cross-region data consistency is ensured, and no data loss occurs in a single Region. High performance in strong cross-region synchronization. The switching policy is flexible. You can switch between the same Region preferentially or customize the switching sequence across regions.
Database in double 11 black technology
Application of X-KV in double eleven
X-kv is an enhancement of the MySQL Memcached plugin. This year we have made significant improvements to support more data types, non-unique indexes, composite indexes, Multi get functionality, Online Schema change, The biggest change is support for SQL transformation through TDDL. For the business side, X-KV has the advantage of ultra-high read performance and strong data consistency. Reduce application response time and cost; At the same time, SQL is supported and applications can be transparently migrated.
TDDL for X-KV is optimized as follows:
- Independent KV connection pool: SQL and KV connection pool are independent from each other; In case of change, the two sets of connection pools must be consistent. Applications can quickly switch between two sets of interfaces.
- Optimized KV communication protocol: no longer need delimiter, protocol implementation.
- Result set automatic type conversion: String is automatically converted to MySQL type.
Transaction vendor library performance bottleneck solution
As the transaction volume of Singles’ Day increases, the synchronization delay of the transaction seller database has been relatively large in the past two years, leading to the failure of merchants to timely process the orders of Singles’ Day. And the seller database has a large number of complex queries, poor performance. We have optimized it by setting up separate queues for large sellers, synchronous link merging operations, and flow limiting of seller libraries, but we still haven’t solved the problem completely.
ESDB is a distributed document database built on THE basis of ES. On the basis of ES, we support SQL interface, and applications can be seamlessly migrated from MySQL to ESDB. For large sellers, dynamic secondary hashing is provided to solve the performance bottleneck of large sellers synchronization. At the same time also do a lot of performance optimization and current limiting protection and other functions.
The database monitors system evolution
There are many technical challenges to database second-level monitoring, including the following four points:
1. Massive data: processing 10 million monitoring indicators per second on average, with a peak of 14 million;
2. Complex aggregation logic: multi-dimensional data aggregation, such as region, machine room, unit, service cluster and database master/standby;
3. High real-time requirements: the monitoring value of the last second needs to be seen immediately when monitoring the screen;
4. Computing resources: Collect and calculate as few resources as possible.
The whole link experiences three generations of architecture: the first generation, Agent + MySQL; The second generation, Agent + DataHub + distributed NoSQL; The third generation, Agent + real-time computing engine + HiTSDB.
HiTSDB is a time-series database developed by Alibaba. It stores second-level performance data and full SQL health in HiTSDB after pre-processing by real-time computing engine. Through the third-generation architecture, the second-level monitoring capability of the double 11 peak is realized, which is very helpful for us to understand the system running status and diagnose problems.
CloudDBA application in Double 11
Ali has the most experienced DBA in the industry and a huge amount of performance diagnostic data. Our goal is to combine the experience of ALI DBA, big data and machine intelligence technology. Our goal is that after three years, we no longer need DBA to do database diagnosis, optimization and other work, but to let machines to complete the intelligent management of database. We believe that self-diagnosis, self-optimization, self-operation and maintenance are the important directions of database technology development in the future.
CloudDBA also made some explorations on Double 11 this year. Through the analysis of full SQL and monitoring data, we implemented automatic SQL optimization (slow SQL tuning), spatial optimization (useless table useless index analysis), access model optimization (SQL and KV) and storage space growth prediction, etc.
Look ahead to next year’s Double 11
Higher: Higher transaction creation peak;
Faster: high-performance database, high-performance storage;
Smarter: CloudDBA delivers more value.