Authors: Yang Zhe, Liu Xiang, Wang Weijia
Small program cloud development has easy access, high performance, high availability and other features, and provides complete native cloud capability support, which can effectively reduce back-end and operation and maintenance costs, help developers focus on business, and achieve rapid online and iteration. Among them, the database developed by the small program cloud is a JSON database that can be operated in the front end of the small program and also read and write in the cloud function, which is powerful and stable.
This article will briefly describe how we built a secure, easy-to-use, high-performance, and highly available NOSQL document database for cloud development.
The Serverless concept leads the new development trend
Before introducing the architecture of the cloud development database, LET me give you a brief background on the design of the cloud development database.
For the developers of small programs and other applications, the traditional development mode and the agile development mode require developers to write core business logic, and inevitably need to control and optimize the infrastructure of the back end. For example, the logic of an application may be simple, but when it comes to the release and deployment of the application, developers need to spend a lot of energy on the application and establishment of servers, databases, networks and other infrastructure, and consider the stability, availability and monitoring indicators of these back-end infrastructure. All of this is time-consuming and labor-intensive, and has nothing to do with the core functions of the product. For products requiring rapid development and trial and error, the traditional mode is slow in development and has high deployment, operation and maintenance costs.
However, with serverless architecture, users can directly access cloud functions, cloud databases and cloud storage by one-click through cloud API without caring about back-end infrastructure to obtain basic back-end capabilities such as computing power, database and storage. This on-demand development mode not only enables developers to focus more on their business logic, but also has many advantages such as low cost, fast development speed and no operation and maintenance. For the development of small program cloud, it combines the development concepts of serverless and small program, relies on wechat as the front running of small program, and achieves out-of-the-box access to the back-end infrastructure by accessing cloud functions, cloud databases and cloud storage. These features can largely release the productivity of small program developers, reduce the cost and difficulty of development, let small program developers to the full stack development, it is easier to develop stable and efficient back-end services. Want to learn more can view cloud development product documentation: cloud.tencent.com/product/tcb
Tencent Photo Album, for example, is an official small program built using cloud development.
In the traditional development mode, developers must start from the small program end, send a series of requests to the background for authentication, file and database processing and other tedious operations, but also need to spend a lot of energy to carry out many basic services design and construction and later operation and maintenance work.
But through cloud development, developers can naturally obtain wechat login state, convenient access to storage and database and other capabilities, other load, disaster recovery, monitoring, etc., are responsible for cloud development, and all this is transparent to developers. This greatly increases developer productivity.
Through cloud development, Tencent album small program development time from more than 1 week to only 1 to 2 days.
Cloud development database architecture introduction
As a database component in cloud development, cloud database has the following characteristics:
1, security: for the database, security is the first, the loss of data is intolerable
2, ease of use: cloud database out of the box, run out, no operation and maintenance deployment
3, low cost: cloud database support according to the volume of visits, data charges, cost control is more precise and lower
4, high performance: cloud data as a NoSQL document database, naturally has very high read and write performance. Our underlying hardware also uses SSDS to provide better read and write capabilities
5. Flexibility: When the user’s business volume expands and the database with larger scale and higher performance is needed, the cloud database also supports live migration to the cloud, and the process is not perceived by users
Below, I’ll give you a brief introduction to the basic architecture of a cloud development database.
The overall architecture of cloud data can be divided into the following parts: the uppermost client, the access layer in the middle, the storage layer at the bottom, and the surrounding control, alarm, and backup modules.
Starting from the client side of the top layer, developers can obtain the login state of cloud database in wechat mini program and QQ mini program with one key through the SDK provided by cloud development, and then send data read and write requests to the access layer. After the access layer receives a user’s read/write request, the stateless modules keeper and Agent process the read/write request.
Keeper is the core module of cloud database permission verification, load balancing and accounting functions, which mainly performs authentication, authentication cache and statistics of read and write requests. At the same time, Keeper also optimizes the request of transaction class. Through consistent hashing and caching sessions, Keeper will distribute the request pressure of different transactions to the Agent module, providing better transaction usage performance.
The Agent module has the following functions:
First, it maintains the connection pool from the access layer to the underlying database instance, and reduces the time consuming of request authentication and connection creation by reusing established connections.
Second, the number of concurrent requests is counted, and the QPS of read and write requests are smoothed to avoid short-term burrs affecting database performance and availability. The third is the optimization of the live migration. Through the Agent module, we will suspend the request during the live migration and switch over the database instance, and then resume the request after the switch, so as to realize that the whole process of the live migration is not aware of the user.
Read and write requests pass through the access layer and then go to our storage layer for reading and writing database instances. Our database instance has the following characteristics in design:
To ensure high availability of the database, each of our database instances is serviced by a replica set as a whole.
Second, in terms of data security, we use consistency algorithm to ensure the final consistency of data in replica set. Regular backup allows users to file data back to a specified time.
Iii. We also make full use of the information collected by the modules of cloud data such as logs, monitoring and dial measurement to carry out the relevant design of automatic operation and maintenance of cloud data, and conduct second-level detection and automatic processing of common database anomalies to ensure the smooth operation of users’ business.
Cloud development database design and optimization
So, what feature optimizations have been made to develop databases around the cloud?
1. Access control optimization
The first is access control. Users can access only their own databases. All connections must be authenticated. Users cannot access other users’ databases. However, you can create multiple accounts with different permissions on your own database (such as a read-only account). Of course, with the feature of wechat full link authentication exemption, users do not need to care too much about authentication.
Secondly, connection number control. We have two layers of control:
1) Control client connection at the access layer, set different initialization limits according to instance types (free/paid, etc.), and prompt corresponding users if the limits are exceeded;
2) Connection number control is also implemented between the access layer and the storage layer to pool links to all primary and secondary nodes of the back-end database to avoid database performance problems caused by excessive links.
Finally, machine-level flow control and resource usage restriction are similar to connection number control. All user requests go through the access layer, so QPS can be controlled at the access layer to implement subsequent pay-as-you-go functionality. When the QPS exceeds the threshold, users can be prompted or queued at the access layer.
PS: Some people may question, isn’t this elastic expansion? Why is there a QPS limit? Should not be my QPS is getting higher and higher, the backend database resources also continue to expand?
Yes, there is some flexibility in the default configuration, but there is a limitation. Of course, how much of this restriction is related to product strategy.
2. Data security optimization
Data security is one of the most important features of a database. After all, a database with the risk of losing data cannot survive the fierce competition in the market. So how does a cloud database ensure data security?
1) Distributed multi-copy Dr. The cloud development database base defaults to three copies, meaning that one copy of data is stored on three different machines. The primary node can be written and read, while the secondary node is read-only. Raft-like replica set protocol is adopted between storage nodes to achieve the final consistency of data in three replicas. In addition, all connections to the database must be authenticated and all data is encrypted, compressed and stored to ensure data link security and storage security.
2) High availability. When a machine is faulty, the data nodes in the replica set automatically switch over to provide services as the active node, minimizing the impact on services.
3) Backup/file back capability. Backup is completely transparent to users. The background automatically selects full and incremental backup according to the state of the database. Support file back at any time within 7 days, you can choose to file back only a single library table *, greatly reducing the time required for file back.
In addition, if a faulty node needs to be added to the replica set, you can restore the faulty node from the backup file to reduce the intrusion on the source cluster.
4) Multiple availability zone Dr. By default, the cloud database is deployed across three equipment rooms (AZs) and is transparent to users. The failure of any equipment room does not affect services. At the same time, it can also support multi-region, two-place, three-center and other modes. For example, Beijing, Shanghai, and Shenzhen each have one node. Services are connected to the nearest node to reduce the latency of service metadata access.
For details about the security rules of the cloud database, see here: Technical Description of security Rules.
3, elastic expansion
In many cases, business access shows obvious periodicity or uneven characteristics. For example, take-out business is in the peak period of meal time, and there are few visits in other periods. The game business belongs to the peak of evening and weekend, with less working time; In addition, some e-commerce businesses are in the peak period when they meet special time points (Singles’ Day, 618).
If you follow the traditional database operation and maintenance mode, you need to estimate the level in advance, and then perform the expansion of the operation and maintenance, and then reduce the capacity after the completion of the activities (otherwise the cost is a problem). So in the scenario of small programs, since the user is not aware of the back-end services, then the expansion and contraction of resources should not be felt.
Based on this starting point, we implemented elastic scaling of the cloud database. Depending on the load monitoring module of the control system, we can dynamically adjust the resources of the database and automatically adjust the sensitivity to effectively cope with the sudden increase of the database load. When the load is low, we can also release the resources to other instances that need more. Second, to avoid frequent adjustments caused by a single large query, we set up a sliding window and a “deburring” mechanism to ensure that elastic scaling happens as smoothly as possible.
When the instance state changes (such as free – > paid, cold – > hot), data migration may be required, such as from a poor machine to a better machine.
With the cooperation of access layer, we realize database live migration without user perception. Users’ data can be migrated losslessly from one database to another without ceaseless service.
Smart DBA
In order to further reduce the operation and maintenance operations on the background side, we have implemented an automated operation and maintenance platform. By monitoring the status of running storage nodes, the system detects and detects faults of each node. Then, the decision center automatically performs corresponding O&M operations based on the fault statistics (for example, if the disk is read-only, the active node is forcibly removed). At the same time, an alarm is generated to the O&M personnel to ensure that the automatic O&M results are correct.
For some problems that cannot be covered by automatic operation and maintenance, we have a full set of second-level monitoring at all levels and dimensions (machines, instances, nodes, etc.), with 69+ items of each indicator. The back-end can sense the state of the database in real time. Deal with problems as soon as possible.
Index is a very important concept in database, used to speed up database lookup. In the applets scenario, we want to reduce the user’s knowledge of the back-end database to as little as possible. A series of query optimization features, such as automatic indexing, are implemented.
When our access layer and storage layer find that many queries of users are full table scan at the back end, they will add corresponding indexes according to the specific query fields of users. After the index is established, users can directly enjoy the optimized query results. Importantly, this process is also insensitive to the user.
Cloud development database applications in different scenarios
1. Application of e-commerce applets
Use basic reading and writing ability to query commodity list
Use aggregate search to sort the list of goods
Use the transaction capability of cloud development database to query and deduct the quantity of goods when placing an order.
2. Application of O2O small programs
Use the geographic search capability to find the nearest business
3. The application of social applets
Real-time data push service is used to realize real-time chat for multiple people
conclusion
In the small program scenario, the cloud development database around data security, ease of use, low cost, high performance, flexibility of the five main scene requirements, the corresponding transformation and optimization, so that the database can be more suitable for the use of small program scenarios.
We also have reason to believe that the cloud development database will continue to improve itself under the guidance of the Serverless concept, and develop better and better.
This course video has been uploaded to the cloud development standing column b: www.bilibili.com/video/BV1Rz…
CloudBase is a back-end cloud service integrated with cloud and adopts serverless architecture, which eliminates the tedious server construction, operation and maintenance in mobile application construction. The static hosting, command-line tools (CLI), and Flutter SDK capabilities provided by Cloud development greatly reduce the barrier to application development. Cloud development can quickly build complete small programs/games, H5, Web, mobile apps and other applications.
The product documentation: cloud.tencent.com/product/tcb
Technical documentation: Cloudbase.net
Technical exchange plus Q group: 601134960
Latest news follow wechat official account [Tencent Yunyun Development]
