Breaking the shackles of relational database: Core analysis of cloud native database middleware

1. Data sharding

Traditional solutions that centrally store data to a single data node have been unable to meet the massive data scenarios of the Internet in terms of performance and availability. Because most relational databases use B+ tree indexes, when the amount of data exceeds the threshold, the increase of index depth will increase the I/O times of disk access, resulting in a significant decline in query performance. At the same time, the high concurrent access request makes the centralized database become the biggest bottleneck of the system.

When traditional relational databases cannot meet the needs of Internet scenarios, there are more and more attempts to store data to NoSQL that supports distributed data natively. However, the incompatibility of NoSQL to SQL and the imperfection of the ecosystem make them unable to complete a fatal blow in the game with the relational database, and the status of the relational database is still unshaken.

Data sharding refers to storing data stored in a single database in multiple databases or tables according to a certain dimension to improve performance bottlenecks and availability. The effective method of data sharding is to divide database or table into relational database. Both sub-database and sub-table can effectively avoid the query bottleneck caused by data volume exceeding the tolerable threshold.

In addition, sub-database can also be used to effectively disperse the access to a single database point; Separate tables provide the possibility of converting distributed transactions into local transactions as much as possible. Multi-master and multi-slave sharding can effectively avoid single data points and improve the availability of data architecture.

1. Vertical sharding

Vertical sharding is also called vertical split, and its core idea is dedicated library. Before the split, a database consisted of multiple tables, each corresponding to a different business. After the split, the tables will be classified according to business and distributed to different databases, so as to share the pressure on different databases, as shown in the figure:

2. Horizontal sharding

Horizontal sharding is also called horizontal splitting. In contrast to vertical sharding, horizontal sharding does not classify data according to business logic, but distributes data into multiple libraries or tables according to certain rules of a certain field, with each sharding containing only a portion of the data.

For example, if the last digit of an ID is mod 10, the mantissa of 0 goes into library 0 (table), and the mantissa of 1 goes into library 1 (table). As shown in figure:

In order to solve the performance problem caused by large amount of data in relational database, data fragmentation is an effective solution.

Split the data concentrated on a single node and store it in multiple databases or tables respectively, which is called sub-database sub-table. Repositories can effectively distribute the pressure on database access caused by high concurrency. Although split table can not relieve database pressure, but only across the split table update operation, still can use the database native ACID transaction; When it comes to cross-library update operations, the problem of distributed transactions becomes extremely complex.

Split data by subdatabase and subtable to keep the data volume of each table below the threshold. Vertical sharding often requires architectural and design changes that are often too slow to cope with the rapidly changing business requirements of the Internet, and it does not really solve a single point of bottleneck. In theory, horizontal sharding breaks through the bottleneck of single machine data processing, and expands the relative freedom. It is the standard solution of database and table.

Library separation and read/write separation are common methods to handle high traffic volume. Although split table can solve the performance problem caused by massive data, it cannot solve the problem of slow response caused by too many requests to access the same database. Therefore, horizontal sharding usually adopts the method of sub-library to solve the problem of huge data volume and page view. Read/write separation is another way to channel traffic, but latency between reads and writes is an architectural consideration.

Although separate libraries can solve the above problems, distributed architecture brings new problems as well as gains benefits. In the face of such scattered database and table after the data, application development and operation of the operation of the database becomes unusually heavy is one of the important challenges. They need to know what data needs to be retrieved from which specific database sub-tables.

The new architecture of NewSQL and data sharding middleware handle this functionality differently:

• The new architecture of NewSQL redesigns the database storage engine to store data from the same table in a distributed file system.

• Data sharding middleware is as transparent as possible about the impact of database and table, so that users can use the database after horizontal sharding as much as possible like using a database.

Cross-library transactions are a tricky problem for distributed databases. Reasonable use of separate table can reduce the amount of single table data, as far as possible to use local transactions, good use of different tables with the same library can effectively avoid the trouble caused by distributed transactions. In scenarios where cross-library transactions cannot be avoided, some businesses still need to maintain transaction consistency. However, XA based distributed transaction cannot be adopted by Internet companies due to its low performance. Most of them use flexible transaction with final consistency instead of distributed transaction.

3, read and write separation

In the face of increasing system traffic, database throughput is facing a huge bottleneck. Less for the same time a large number of concurrent read and write operations on the type of application system, mainly to split a single database library and from the library, library is responsible for handling transactional increase deletion operation, from the library is responsible for handling the query operation, can effectively avoid caused by data update row locks, make the whole system of query performance improved significantly.

In the configuration mode of one master and multiple slaves, query requests are evenly distributed to multiple data copies, which further improves the processing capability of the system.

Using the multi-master and multi-slave mode can not only improve the throughput of the system, but also improve the availability of the system. It can achieve the normal operation of the system in the case of any database breakdown or even physical disk damage.

Read/write split is essentially a kind of data sharding. Unlike horizontal shards, where data is split into different data nodes based on shard keys, read/write split routes read and write requests to master and slave libraries based on SQL semantic analysis. Data in read/write split data nodes is consistent, while data content in horizontal sharding data nodes is different. The combination of horizontal sharding and read/write separation improves system performance, but complicates system maintenance.

Although read/write separation can improve system throughput and availability, it also brings data inconsistency, including data consistency between multiple master libraries and data consistency between master and slave libraries. In addition, read/write separation brings about the same problems as data sharding, which also makes the operation and operation of the database more complicated for application developers and o&M personnel.

The main function of read/write separation is to make the impact of read/write separation transparent so that users can use the master/slave database as much as possible as a database.

4. Core processes

The core of data sharding is composed of SQL parsing, SQL routing, SQL rewriting, SQL execution and result merging. To keep the original application low access cost, it must be compatible with database access. Therefore, database protocol adaptation is required.

Protocol adapter

NewSQL is compatible with traditional relational databases. In addition to SQL, protocols compatible with databases can reduce users’ access costs. Open source relational databases can dress up their products as native relational databases by implementing their protocol standards.

Due to the popularity of MySQL and PostgreSQL, many NewSQL implement their transport protocols, allowing users of MySQL and PostgreSQL to automatically access NewSQL products without changing their business code.

The MySQL protocol

MySQL is the most popular open source database. To understand its protocol, we can start with the basic data types, protocol package structure, connection stage and command stage of MySQL.

Basic data types

All contents in the MySQL protocol package are composed of basic data types defined by MySQL. For details, see the following table:

MySQL basic data type

When binary data needs to be converted into data that can be understood by MySQL, the MySQL protocol package will read it according to the pre-defined bits of the data type and convert it to the corresponding number or string. And vice versa, MySQL writes each field to the protocol package at the length specified in the specification.

Protocol package structure

The MySQL protocol consists of one or more MySQL protocol packets. It is composed of three parts: the Payload Length, the Sequence ID, and the message body.

• The message length is of type int<3>. It represents the total number of bytes consumed by subsequent message bodies. Note that the message length does not include placeholders for the sequence primary key.

• Sequence primary key of type int<1>. It represents the serial number of each of the multiple MySQL protocol packages returned after a request. The maximum number of primary keys for a sequence with a 1-byte placeholder is 0xFF, which is 255 in decimal notation, but this does not mean that a maximum of 255 MySQL protocol packages can be contained in each request. Any sequence with a maximum of 255 primary keys will again be counted from 0. For example, a query may return hundreds of thousands of records, so the MySQL protocol package only needs to ensure that its sequence primary key is continuous, reset the sequence primary key greater than 255 to 0, and restart the count.

• The length of the message body is the number of bytes declared for the message length. It is the real business data in the MySQL protocol package, and the content of the message body varies according to different protocol package types.

The connection phase is used to create a communication channel between the client and server of MySQL. This stage mainly performs three tasks: exchanging and matching Capability Negotiation between MySQL client and server, creating SSL communication pipes, and verifying authorization. The following diagram shows the connection creation flow from the perspective of MySQL server:

MySQL connection phase flow chart

The figure does not include the MySQL server/client interaction. In fact, MySQL connection creation is initiated by the client.

After receiving the connection request from the client, the MySQL server first performs Capability Negotiation and exchange between the server and client versions. Then, according to the Negotiation results at both ends, the MySQL server generates the initial handshake protocol package in different formats and writes the modified protocol package to the client. The package includes the connection primary key assigned by the MySQL server, the server’s current version Capabilities, and the ciphertext generated to verify authorization.

After receiving the handshake protocol packet from the server, the MySQL client sends a handshake protocol response packet. The main information contained in the protocol package is the user name used for database access and the encrypted password ciphertext.

After receiving the handshake protocol response package, the MySQL server performs authorization verification and returns the verification result to the client.

The command phase

After the connection phase succeeds, the interaction phase of command execution enters. MySQL has 32 command protocol packages. For details, see the following figure:

The MySQL command package

The command protocol packages of MySQL fall into four categories: text protocol, binary protocol, stored procedure protocol, and data replication protocol.

The first digit in the packet body identifies the command type. There is no need to explain their specific uses here. Here we will parse some key MySQL command packages:

• COM_QUERY

COM_QUERY is an important MySQL command used to query in plaintext format. It corresponds to java.sql.Statement in JDBC. The COM_QUERY command itself is relatively simple, consisting of identifiers and SQL:

1              [03] COM_QUERY

string[EOF]    the query the server shall execute
Copy the code

COM_QUERY’s response protocol package is more complex, as shown below:

Flow chart of MySQL query commands

COM_QUERY may return four types depending on its scenario: query result, update result, file execution result, and error result.

When errors occur, such as network disconnection or INCORRECT SQL syntax, the MySQL protocol requires that the first part of the protocol packet be set to 0xFF and the error information is encapsulated in the ErrPacket.

COM_QUERY execution from a file is unusual and will not be explained here.

For update requests, the MySQL protocol requires that the protocol packet first be set to 0x00 and that the OkPacket protocol packet be returned. The OkPacket protocol packet must contain the number of rows affected by the update operation and the primary key value inserted last.

The query request is the most complex, requiring that the number of result set fields retrieved by reading ints be returned as a separate FIELD_COUNT protocol package. Then, the details of each column of the returned field are successively generated into an independent COLUMN_DEFINITION protocol packet, and the metadata information of the query field ends with an EofPacket. After that, you can start generating the data Protocol package Text Protocol Resultset Row line by line, which itself does not care about the specific type of data, but uniformly converts it to string format. The data protocol packet still ends up with an EofPacket.

Corresponds to the JDBC in Java, SQL PreparedStatement operations, is composed of MySQL binary protocols in the protocol packet, They consist of COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_ CLOSE, COM_STMT_RESET, and COM_ STMT_SEND_LONG_DATA. The most important ones are COM_STMT_PREPARE and COM_STMT_ EXECUTE, Of their corresponding JDBC connection. PrepareStatement () method, and the connection. The execute () & connection. The executeQuery () & connection. The executeUpdate () method .

• COM_STMT_PREPARE

COM_STMT_PREPARE is similar to COM_QUERY in that it consists of command identifiers and SQL:

1              [16] COM_STMT_PREPARE

string[EOF]    the query to prepare
Copy the code

The return value of COM_STMT_PREPARE is not the query result, but the response package consisting of statement_id, number of columns, and number of parameters. Statement_id is a unique identifier assigned by MySQL to the pre-compiled SQL. The statement_id is used to retrieve the SQL from MySQL.

For the SQL registered with the COM_STMT_PREPARE command, you only need to pass the statement_id to the COM_STMT_EXECUTE command. You do not need to pass the SQL to the COM_STMT_EXECUTE command again, saving unnecessary network bandwidth consumption.

In addition, MySQL can precompile the SQL from COM_STMT_PREPARE into an abstract syntax tree for reuse, thus improving the execution efficiency of SQL. To execute SQL using COM_QUERY, you need to recompile each SQL. This is why preparedStatements are more efficient than statements.

• COM_STMT_EXECUTE

The COM_STMT_EXECUTE protocol package consists of statement-id and parameters paired with SQL. It uses a data structure called null-bitmap to identify NULL values in parameters.

The response protocol package of the COM_STMT_EXECUTE command is similar to that of the COM_QUERY command. Both of them are returned in the format of field metadata and query result set. The EofPacket interval is still used.

The COM_STMT_EXECUTE response Protocol package uses Binary Protocol Resultset Row instead of Text Protocol Resultset Row. It does not convert data to strings regardless of the data type. Instead, the corresponding basic MySQL data type is written according to the returned data type to further save network transmission bandwidth.

In addition to MySQL, the PostgreSQL and SQLServer protocols are fully open source and can be implemented in the same way. The Oracle protocol, another popular database, is not open source and cannot be implemented this way.

SQL parsing

SQL is relatively simple compared to other programming languages. However, it is still a full-fledged programming language, so parsing SQL syntax is not fundamentally different from parsing other programming languages such as Java, C, Go, and so on.

The parsing process is divided into lexical parsing and grammatical parsing. The SQL is first broken down into non-separable words through lexical parsing. The SYNTAX parser is then used to convert the SQL into an abstract syntax tree. Finally, the parsing context is extracted by accessing the abstract syntax tree.

Parsing contexts include tables, selections, sort items, group items, aggregation functions, paging information, and query conditions. NewSQL of the shard middleware type also needs to record placeholder markers that may be modified.

SQL: select username, ismale from userinfo where age > 20 and level > 5 and 1 = 1 SQL: select username, ismale from userinfo where age > 20 and level > 5 and 1 = 1

Abstract syntax tree

There are many third-party tools for generating abstract syntax trees, and ANTLR is a good choice. It can generate Java code for an abstract syntax tree through developer-defined rules and provide a visitor interface. Compared to code generation, handwritten abstract syntax trees are more efficient in terms of execution efficiency, but also require a lot of work. In scenarios with high performance requirements, a custom abstract syntax tree can be considered.

Request routing

Data sharding policies are matched based on the parsing context and routing paths are generated. SQL routes with sharding keys can be classified into single-slice routes (the sharding operator is equal), multi-slice routes (the sharding operator is IN), and range routes (the sharding operator is BETWEEN) according to the different sharding keys. SQL that does not carry sharding keys uses broadcast routing.

Sharding policies can usually be configured either built-in to the database or by the user. The built-in database scheme is relatively simple, the built-in sharding strategy can be roughly divided into mantissa module, hash, range, label, time and so on; The sharding policy configured by the user is more flexible. You can customize the compound sharding policy based on the user’s requirements.

Rewrite the SQL

The NewSQL of the new architecture does not require SQL rewriting, which is mainly for the shard middleware type of NewSQL. It is used to rewrite SQL into statements that would execute correctly in a real database. It includes replacing logical table names with real table names, rewriting the start and end values of paging information, adding supplementary columns for sorting, grouping and auto-increment primary keys, and rewriting AVG to SUM/COUNT, etc.

Result merging Merges multiple result sets and outputs them to the application end in a unified manner. Result merging includes both streaming and in-memory merging:

• Streaming merge is used for simple queries, sorted queries, grouped queries, and scenarios where the sorted items and grouped items are exactly the same. The result set of streaming merge is traversed by calling the next method every time without consuming additional memory.
• In memory merging, all data in the result set needs to be loaded into the memory for processing. If the result set contains too much data, a large amount of memory will be occupied.

Distributed transactions

As mentioned above, database transactions need to meet the four characteristics of ACID (atomicity, consistency, isolation, persistence) :

• Atomicity refers to the execution of a transaction as a whole, either all or none.
• Consistency means that a transaction should ensure that data is transferred from one consistent state to another consistent state.
• Isolation means that when multiple transactions are executed concurrently, the execution of one transaction should not affect the execution of other transactions.
• Durability refers to that committed transaction modification data is persisted.

Within a single data node, transactions are limited to access control of a single database resource and are called local transactions. However, in the distributed application environment based on SOA, more and more applications require that access to multiple database resources and multiple services can be included in the same transaction, and distributed transaction arises at the historic moment.

Relational databases, though, provide perfect ACID native support for local transactions. But in distributed scenarios, it becomes a shackle to system performance. How to make the database meet the characteristics of ACID in distributed scenarios or find the corresponding alternative is the key work of distributed transactions.

1. XA protocol

The earliest Distributed Transaction model is the X/Open Distributed Transaction Processing (DTP) model, referred to as XA protocol, which was proposed by the X/Open International Consortium.

The DTP model interacts with multiple resource managers through a single global transaction manager. The global transaction manager is responsible for managing the global transaction state and the resources participating in the transaction, while the resource manager is responsible for the specific resource operations. The relationship between the DTP model and the application is shown below:

The XA protocol uses two-phase commit to ensure atomicity of distributed transactions. It divides the commit process into preparation and commit phases.

• In the prepare phase, the global transaction manager sends a prepare message to each resource manager to confirm the success of the local transaction operation.
• During the commit phase, the global transaction manager sends a commit message to each resource manager if it receives a success message from all resource managers, otherwise it sends a rollback message. The resource manager commits or rolls back local transactions based on received messages.

The following figure shows the XA protocol transaction flow:

XA transaction flow

Two-phase commit is a standard implementation of the XA protocol. It splits the commit of a distributed transaction into two phases: prepare and COMMIT/ROLLBACK.

After the XA global transaction is enabled, all sub-transactions lock resources according to the default isolation level and record undo and redo logs. Then TM initiates a prepare vote to ask all sub-transactions whether they can commit. When the result of all sub-transactions is yes, TM initiates a commit. If the result of any subtransaction is “no”, TM initiates rollback. If the feedback result in the prepare phase is yes but an exception such as downtime occurs during the COMMIT process, commit compensation can be performed based on XA Recover after node services restart to ensure data consistency.

Distributed transaction based on XA protocol has little impact on business, and its biggest advantage is transparency to users. Users can use distributed transaction based on XA protocol just like using local transaction. The XA protocol ensures strict transaction ACID characteristics.

But strict assurance of transaction ACID properties is a double-edged sword.

In the process of transaction execution, all the resources needed are locked, which is more suitable for short transactions with fixed execution time. For long transactions, the exclusive data during the whole transaction will lead to the deterioration of the concurrency performance of the business system that depends on hot data. Therefore, distributed transactions based on THE XA protocol are not the best choice in high-performance scenarios with high concurrency.

2. Flexible transactions

If a transaction that implements ACID transaction elements is called a rigid transaction, then a transaction based on BASE transaction elements is called a flexible transaction. BASE is an acronym for Basically Available, Soft state, and Eventually consistent:

• Basic availability ensures that distributed transaction participants are not necessarily online at the same time;
• Flexible state allows system state update to have a certain delay, which may not be perceptible to customers;
• Final consistency is usually guaranteed by means of message reachability.

Isolation is a high requirement in ACID transactions, where all resources must be locked during transaction execution. The idea behind flexible transactions is to move mutex operations from the resource level to the business level through business logic. Ease the requirement for strong consistency in exchange for improved system throughput.

Because timeout retries are possible in distributed systems, operations in flexible transactions must be idempotent to avoid the problems of multiple requests. Flexible transaction implementation schemes mainly include maximum effort delivery, Saga and TCC.

Best effort delivery

It is the simplest type of flexible transaction and is suitable for scenarios where the operation on the database must eventually succeed. The failed SQL is automatically recorded by NewSQL and repeated attempts are made until the execution succeeds. Flexible transactions using maximum effort delivery have no rollback capability.

This type of flexible transaction implementation is the simplest, but it is very demanding for the scenario. The advantages of this strategy are lock-free resource time and low performance loss. The disadvantage is that there is no rollback after multiple failed attempts to commit, and it only applies to business scenarios where the transaction is bound to succeed eventually. Therefore, it compromises transaction rollback functionality in exchange for performance gains.

Saga

Saga began with a 1987 paper by Hector Garcaa-Molrna and Kenneth Salem.

References:

www.cs.cornell.edu/andru/cs711…

Saga transactions are better suited to scenarios where long transactions are used. It is composed of multiple local transactions, each local transaction has the corresponding execution module and compensation module, any local transaction error, you can call the relevant supplementary method to achieve the final consistency of the transaction.

The Saga model splits a distributed Transaction into multiple local transactions, each with a corresponding Transaction and Compensation module. When any local transaction fails in a Saga transaction, the previous transaction can be restored by calling its related compensation method to achieve the final consistency of the transaction.

When each Saga subtransaction T1,T2… ,Tn has corresponding compensation definition C1,C2… Cn-1, then Saga system can guarantee:

• Subtransaction sequence T1,T2… Tn is completed. This is the best-case scenario for a transaction, where no rollback is required.

• Or sequences T1,T2… , Tx, Cx,… ,C2,C1, where x is less than n, is completed. It ensures that when a rollback occurs, the compensation operations are performed in the reverse order of the forward operations.

The Saga model supports both forward and reverse recovery. Forward recovery refers to retry the current failed transaction, which is implemented on the premise that each sub-transaction can be successfully executed. Backward recovery, as mentioned earlier, compensates for all completed transactions when any subtransaction fails.

Obviously, there is no need to provide compensation transactions for forward recovery, and if subtransactions in the business always succeed, then forward recovery can reduce the complexity of using the Saga model. In addition, if compensation transactions are difficult to implement, forward recovery is also a good option.

In theory, though, compensating transactions never fail. In a distributed world, however, servers can go down, networks can fail, and even data centers can lose power. Therefore, a mechanism, such as human intervention, needs to be provided to roll back after a fault recovery.

The Saga model does not have the preparation phase of the XA protocol, so transactions do not achieve isolation. If two Saga transactions operate on the same resource at the same time, problems such as update loss and dirty data reading can occur. This requires applications that use Saga transactions to incorporate resource locking logic at the application level.

TCC

TCC (try-confirm-Cancel) distributed transaction model implements distributed transaction through decomposition of business logic. As the name implies, the TCC transaction model requires a business system to provide the following three pieces of business logic:

• The Try. Check services and reserve resources required by services. The Try operation is the essence of the entire TCC, allowing flexibility in the granularity of business resource locks.

• Confirm. Execute the service logic and directly use the service resources reserved during the Try phase without checking services again.

• Cancel. Release service resources reserved during the Try phase.

TCC model only provides two-phase atomic commit protocol to ensure atomicity of distributed transactions. The isolation of transactions is left to the business logic. The idea of isolation of TCC model is to release the underlying database lock resources, relax the distributed transaction lock protocol and improve the concurrency of the system by moving the lock from the database resource level to the business level through the service transformation.

Although the TCC transaction model is the most powerful among flexible transactions, it is the responsibility of the application to provide three interfaces that implement the Try, Confirm, and Cancel operations for the transaction manager to call. Therefore, the transformation cost of the business side is high.

Taking account A transferring 100 yuan to account B as an example, the following figure shows the transformation of TCC’s business:

The remittance service and the receipt service need to implement the try-confirm-Cancel interface, respectively, and inject it into the TCC transaction manager during the business initialization phase.

Try

• Check the validity of account A, that is, check whether the status of account A is in transfer or frozen.
• Check whether the balance of account A is sufficient;
• Deduct RMB 100 from account A and set the status to “transferring”;
• Reserve deduction resources to save the event of transferring 100 yuan from account A to account B in message or log.

Confirm

• Do nothing.

Cancel

• Account A increases by 100 yuan;

• Release deduction resources from logs or messages.

Try

• Check whether account B is valid.

Confirm

• To read logs or messages, account B adds 100 yuan;
• Release deduction resources from logs or messages.

Cancel

• Do nothing.

It can be seen that TCC model is highly intrusive to business and difficult to transform.

message-driven

The message consistency scheme ensures the consistency of data operation of upstream and downstream applications through message middleware. The basic idea is to place local operations and send messages in a local transaction, and the downstream application subscribes to the message system and performs the corresponding operations after receiving the message. In essence, it depends on the retry mechanism of the message to achieve final consistency. The following is a message-driven transaction model:

The disadvantage of message-driven is that the coupling degree is high and message-oriented middleware needs to be introduced into the business system, which increases the system complexity.

In general, acid-based strong consistent transactions and Base-based final consistent transactions are not silver bullets and can only be used to their best advantage in the most appropriate scenarios. A detailed comparison of their previous differences will help developers in their technology selection. Due to the high coupling degree of message-driven and business system, it is not included in the comparison table:

The pursuit of consistency is not necessarily the most sensible solution. For distributed systems, it is recommended to use the “soft outside rigid inside” design scheme. Externality refers to the use of flexible transactions across data sharding to ensure that the data is ultimately consistent and in exchange for the best performance. Inner rigor refers to the use of local transactions within the same data shard to achieve ACID effects.

Third, database governance

1, basic management

Much of the service governance described above is common in the basic database governance. It mainly includes configuration center, registry, traffic limiting, fusing, failover, call link tracing, etc. :

• The configuration center is used for centralized configuration and dynamic configuration update and notification delivery.

• The registry is used for service discovery. The service here refers to the database middle layer instance itself. It can realize status monitoring and automatic notification, and then make the database middleware have high availability and self-healing ability.

• Traffic limiting is used for traffic overload protection, which is divided into traffic overload protection of database middleware and traffic overload protection of database.

• Fusing is also one of the protection measures against traffic overload. It is different in that it fuses the entire client’s access to the database to ensure that the database can continue to provide services for other systems with normal traffic. The automatic fusing mechanism can be realized through the fuse mode mentioned above.

• Failover is used in the case of multiple data copies. In the case of multiple data nodes with identical data, when a node is unavailable, the database middleware can access to another valid data node to operate data through the mechanism of failover.

• Call link tracing visualizes the call link, performance and topological relationship of database access.

2, elastic expansion

The key difference between database governance and service governance is that databases are stateful and each data node has its own persistent data, so it is difficult to scale flexibly like servization.

Resharding of the database is often involved when system visits and data volumes exceed the expectations of previous assessments. In most scenarios, the legacy data in the database cannot be directly mapped to the new sharding policy, although the capacity can be expanded without migrating legacy data. Data migration is required to modify the sharding policy.

In traditional systems, stopping the service for data migration and then restarting the service after the migration is complete is an effective solution. However, this scheme makes the data migration cost of the business side very high and requires the engineers of the business side to accurately evaluate the data volume.

In the Internet scenario, system availability is highly required, and the possibility of explosive business growth is more common than in traditional industries. In the cloud native service architecture model, elastic scaling is a common requirement and can be implemented relatively easily. Therefore, the elastic scaling function of data equivalent to service is an important capability of cloud native database.

In addition to system pre-partitioning, another implementation of elastic scaling is online data migration. The biggest challenge of online data migration, which is often likened to “changing the engine of an airplane while in flight,” is how to ensure that the migration process does not affect the service. Online data migration can be modified in the database of subdivision strategy (for example, according to the primary key % 4 can be divided into four libraries shard from 16 of 16 library according to the primary key % fragmentation), through a series of systematic operation, ensure correct data migration to the new data nodes at the same time, let all rely on the database service awareness. It can be divided into the following four steps:

• Double write on sync line. That is, data is written to both the original data node before the sharding policy is modified and the new data node after the sharding policy is modified. Consistency algorithms, such as Paxos or Raft, are used to ensure the consistency of double-writes.

• Historical data migration. The historical stock data that needs to be migrated to the new data node is migrated from the original data node in an offline manner. It can be processed in SQL or binary mode, such as binlog.

• Data source switchover. Switch the read and write requests to the new data source and stop the double-write of the original data node.

• Clear redundant data. In the old data node, clean up related data that has been migrated to the new data node.

Online data migration supports data expansion and online DDL operations in the same way. Since database native DDL operations do not support transactions and DDL operations on tables containing a large number of data will result in a long time table lock, online DDL operations can be supported through online data migration. The online DDL operation is the same as the data migration procedure. You only need to create an empty table after the DDL modification before the migration, and then perform the preceding four steps.

Enterprise IT budget is not high, but there are hard business requirements, how to do? From now on to March 31, JINGdong cloud “open year hi buy season” cloud host, cloud database, cloud storage, cloud hard disk, cloud security and other categories of popular products as low as 1.4 discount! ** There are Beijing Fish C1 smart speaker, Beijing Fish I8 smart speaker, iPhone 12 64G mobile phone gift! Click [Read article] for more details.

Recommended reading

• Cloud on the 3 q | jingdong what are the big contributors to cloud the “different”?

• Ms-level database query? After reading this article, you can also use massive data

• 839 times faster than MySQL! Uncover the mystery of analytical database JCHDB

Welcome to [JINGdong Technology] to learn about the developer community

More wonderful technical practice and exclusive dry goods analysis

Welcome to “JINGdong Technology Developer” public account