Tencent Cloud CDB/CynosDB technology revealed

The picture below is the full layout of Tencent cloud database products, with PaaS products at the bottom. On top of this, the database team also prepared many SaaS products for the database sector for users, including migrating cloud services, intelligent butler DBBrain and database management services. We will reflect on the iterations of PaaS and SaaS database products based on the needs of various user industries such as e-commerce, finance and retail modules. Tencent cloud product evolution, are the first from its own business burnish began.

Tencent cloud database products in the research and development to follow the six design concepts: to maintain strong consistency of data, financial high availability, Internet high performance, high security, good expansion, healthy operation and maintenance; These design concepts can be reflected from the Tencent cloud CDB product.

The new infrastructure includes 5G base stations, ultra-high voltage, intercity high-speed railways and urban rail transit, charging piles for new energy vehicles, big data centers, artificial intelligence and the industrial Internet. The impact of new infrastructure on database industry has both opportunities and challenges. When the new infrastructure era comes, there will be a data explosion, so there will be a performance explosion in online trading databases and a business explosion in online analytics databases. Secondly, there will be the business demand of multi-mode data, and the business integration will be diversified. Automate database governance.

In order to meet the new era, database products need to be comprehensively upgraded, with the help of cloud capabilities, to the “cloud + database” model transformation. Cater to users’ old habits, and cultivate users’ new habits, fully compatible with MySQL/PG/Orade protocol, to provide users with OLTP product capabilities. In terms of autonomy, with the massive increase of data volume, layout should be carried out in the direction of intelligent diagnosis and brain tuning. New infrastructure needs super-converged product development trends, multi-mode integration, business integration, scene integration and management integration.

Security is a business that Tencent attaches great importance to internally. Since 2013, it has paid great attention to the development of security education and the cultivation of security talents. The Rhinoceros Bird Foundation was established to promote the development of schools and research. TCTF competition held by Tencent is also one of the top CTF competitions in China. In 2017, Tencent signed a memorandum of education cooperation with the Ministry of Education, and established Tencent Security Academy in 2018. From a series of actions can be seen that Tencent attaches importance to the training of security talents. In recent years, Tencent Security has gradually absorbed millions of security experts, among which the most famous is the head of the seven joint laboratories, who are the top white hat hackers in China.

In addition, we also launched Tencent Cloud Network security Operation and Maintenance certification, which is Tencent’s enterprise-level certification for security operation and maintenance positions. It is suitable for students majoring in network engineering, information security, cloud computing and computer application in colleges and universities. The assessment content of this certification is highly matched with the skills and requirements required by the actual network security operation and maintenance positions. Students who have obtained the certification can enter our talent pool and be preferently employed by Tencent and its ecological enterprises.

CDB’s products commonly use such an architecture diagram, which has some pain points: database capacity is limited by the size of a single disk. Capacity expansion is limited. Common replication may lose data, resulting in poor performance of synchronous replication. Limited availability and slow HA and recovery. The read-only standby node has slow deployment speed and redundant storage and I/O functions.

Based on these pain points, we designed the architecture of CynosDB. Under the new architecture design, the database capacity can be expanded to 100T. Storage layer three copies to ensure that data will not be lost. The HA and recovery speed is fast, and the primary/secondary delay is small. The read-only standby deployment speed is fast.

This is an architecture diagram of RIO, a compute-storage separation architecture with lock-free and parallelized technical features. IOI queues are lock-free to improve concurrency. This architecture also makes full use of the I/O parallelism capability of underlying storage nodes.

In the whole architecture of computing and storage separation, logging is the core point, and the efficiency of logging largely determines the throughput of the system. We optimized the logging system to make the log buffer lock-free, increased the concurrency of writing logs, and added a dedicated log writing thread to improve the efficiency of writing logs.

For asynchronous group commits, instead of waiting for the log compass to commit, the transaction enters the commit queue, and the thread can return to the thread pool to continuously acquire new task execution. The log writer completes the log compass, puts a set of transactions into the task queue, and proceeds with the remaining commit operations.

Btree consistency SMO logs are added to identify Btree splitting or merging operations, and index: : sync LSN is updated when SMO logs are applied. If the Btree page is in the SMO, open the cursor again.