Operations automation from the pain points in the work, jingdong database team is facing mall tens of thousands of engineers, the pressure push us constantly change, change is not achieved overnight, however, also went from the manual to scripting, automation, platform, intelligent hard shift, so it needs to drive the construction of the operational system, The essence of operation and maintenance automation lies in the liberation of operation and maintenance personnel, promote the increase of human rate, reduce human failure, to learn to cultivate their own “lazy” this good habit. The construction of jingdong’s automatic operation and maintenance system began in 2012. The following two aspects will be introduced.

  1. Jingdong’s business is growing every year in the form of explosion. There are a large number of database servers and thousands of product lines. To support such a huge business system, a set of perfect automatic operation and maintenance management platform is needed. Currently, JD MySQL database management platform, referred to as DBS, mainly covers the following contents: Perfect asset management system, database, process management system, database, fault monitoring system, database management systems, databases, reporting system, flexible auxiliary operations tool database system and database, involving all aspects of the operational DBA, automate the DBA to MySQL, self-support, visualization, intelligent, service management, Avoid production accidents caused by manual operation errors by DBA, and ensure the safe, stable and efficient operation of JINGdong database. This section focuses on the following core functional components.

1.1 metadata management as the cornerstone of automated operation and maintenance, its accuracy is directly related to the reliability of the entire database management platform. Jingdong database management platform starts from the operation and maintenance habits of database business side and DBA, covering computer room, host, business, cluster, instance, library, table and other dimensions.  machine room and host dimensions: Primarily records hardware information.  Business Dimension: Primarily records the name, level, and information about the business unit of the business.  Cluster Dimension: Records the MySQL cluster architecture information.  instance Dimension: records related parameters of MySQL to provide assurance for subsequent automated operation and maintenance.  library dimension: Primarily records the database name and business person contact information.

1.2. Automatic Deployment In the face of complex operation and maintenance work such as database addition and expansion, automatic installation and deployment platform can completely liberate DBAs. At present, the automated deployment system of JD includes application server, database instance deployment, data synchronization, consistency check, splitting and switching, and other operations. The whole process is streamlined, including the operation approval of all levels of business and DBA, and finally achieves the comprehensive automatic and process deployment of MySQL services, as shown in the figure below.

Create tasks in batches. The selection principle is local before remote, number of connections before QPS, 10.66.66.66:3366. The target primary database is 10.88.88.89:3366. Batch execution switchover

All function modules of JD MySQL database switching system have been componentalized, which simplifies the operation process of DBA and shortens the time of database switching. At the beginning of the design, JINGdong database backup system aims to free DBA from complicated backup management work, realize automatic processing, reduce human intervention, and improve the availability of backup files. With regard to the availability of backup files, the polling recovery policy ensures that each cluster is restored to within a period. The system architecture design is shown in the figure below:

The architecture has the following characteristics:

  1. Multiple scheduling trigger modes: The scheduling center supports interval, crontab, and Date. Interval is a periodic scheduling task. You can specify a task at a fixed interval, including weeks, days, hours, minutes, and seconds. You can set the start time, end time, and time zone. The crontab is the same as the Linux crontab. The crontab supports year, month, day, week, day_of_week, hour, minute, and second. You can set the start time, end time, and time zone. Date is a one-time scheduling function that supports time zone Settings.
  2. Concurrency control: Due to the imbalanced setting of scheduling tasks, there may be many tasks to be scheduled at a certain time, which is easy to cause problems in the scheduling system. Therefore, tasks can be executed more smoothly by controlling the number of concurrent tasks.
  3. Layering of triggering and execution: Task triggering itself is a lightweight set, while task execution is generally heavy, so the triggering and execution are layered to prevent problems with subsequent triggering due to long execution time.
  4. No loss of tasks during maintenance: The Crontab of Linux will not execute the tasks to be run during maintenance after startup, while the scheduling center based on APScheduler will run the tasks that have not been executed within a specified interval after startup to reduce the missed execution of tasks due to maintenance.
  5. Add, delete, change, and check the backup policy: In the previous backup system, you needed to specify a specific IP address, which often resulted in backup failures due to server maintenance. Therefore, you combined the backup policy with high availability at the beginning of the design. The backup policy specified a domain name instead of an IP address. During a failover, DBS switches the domain name on the slave database to another slave database in the cluster, and the corresponding backup is also transferred to the slave database, ensuring that the backup server is available.
  6. Automatic retry after a backup failure: Backup may fail occasionally. Therefore, the backup retry function is added to enable backup retries for backup jobs that fail within 6 hours. The backup retries are performed for a maximum of three times to achieve a higher backup success rate.
  7. Automatic recovery detection: Backup in every step should be strictly verified, but cannot guarantee absolute backup file is available, so the introduction of the automatic recovery detection mechanism, to help the DBA to test the backup file, found in time because of not considering the result of the backup file is unavailable, and restore testing and audit of a rigid requirements, Automatic recovery detection also relieves DBAs of heavy recovery detection work.

1.5.2 Scheduling design

The whole automatic backup and recovery system mainly consists of dispatching system, backup system, recovery system, recovery detection system and automatic repair system. The dispatching system is the core of the whole system, through which to coordinate the operation of other systems. The scheduling system can deploy Standby devices to implement HIGH availability and deploy actuators in clusters to implement high availability and horizontal expansion. During each backup, the backup system checks instance health status and backup running status to prevent invalid database instances from being backed up. Restoring the system is mainly used when data recovery, elastic capacity expansion and so on need to restore from backup files to running database instances. DBA can complete data recovery simply by operation. Recovery detection automatically detects the availability of backup files under the command of the scheduling system to help DBAs find unavailable backup files in time. Some backup failures can be resolved by automatic retry. However, some backup failures cannot be resolved by retry and need to be repaired accordingly. Therefore, an automatic recovery system is developed to automatically recover backup failures caused by environment problems. Scheduling system is the most core system and the brain of the whole backup and recovery system. At that time, I investigated several implementation methods, such as Crontab of Linux, Azkaban and Apscheduler of Python open source framework, and finally concluded that Apscheduler is more flexible and compact with more diversified scheduling methods. The maintenance cost is lower in the later stage of Python development, so Apscheduler is used to develop the scheduling center.

1.5.3 The front end of the system consists of four modules: backup policy management, backup details, backup blacklist management, and recovery details. Backup policy management:

Restore test details:

  1. 2.1. Before ContainerDB, JINGdong’s database service was containerized. Although the database service has realized the basic functions such as fast delivery and automatic failover of database service through Docker container, the stability and efficiency of database service has been improved to some extent. However, the operation and use methods of database services are basically the same as those of traditional methods. Typical problems are as follows: 2.1.1 Excessively Large Granularity of Resource Allocation A database server has too few resource standards because of fixed resource standards. 2.1.2 serious resource waste resource allocation standard have DBA determined according to the experience, there are a lot of subjectivity, not according to the actual situation for accurate assessment of the business, and the DBA in the allocation of resources usually consider don’t need to service within 3 years to migrate or expansion, and allocate more resources at a time, there are serious resource waste. Moreover, due to the fixed database resource standard, the standard is too large, leading to the fragmentation in the host machine, often a host machine can only create one container, and the remaining resources can not meet any resource standards, resulting in the low resource utilization rate of the host machine. 2.1.3 Static Resources and No Scheduling Once the database service is provided, the resources occupied will be fixed and the online dynamic scheduling cannot be carried out according to the load of the database. Once the disk usage of the database is too high, the DBA needs to manually expand the capacity, which is inefficient.

2.2. Based on the above problems, ContainerDB is no longer a simple solution. We need to make the database service smarter, make the database resources move, and provide phased delivery of resources, so ContainerDB was born. ContainerDB’s flexible load-based scheduling gives jd’s database resources the intelligence to truly flow, and it has successfully served multiple 618 and 11.11 campaigns.

ContainerDB implements prioritized merge sort on the Gate layer to provide fast streaming query. When a large volume of data is queried, part of the query results can be returned instantly, greatly improving customer experience. Imperceptionless data migration ContainerDB develops online data migration and access tool JTransfer by implementing partial data copy and incremental data addition algorithm respectively in the Window function. JTransfer migrates dynamic data from the traditional MySQL database to ContainerDB. When the lag between ContainerDB data and source MySQL data is less than 5 seconds, the source MySQL database will stop writing data. When lag becomes 0, the domain name of the source MySQL is migrated to the Gate cluster, and the user AppServer is unaware of the whole migration process. Compatible with the MySQL protocol ContainerDB is fully compatible with the MySQL protocol, supports standard MySQL clients and official drivers, and supports most ANSI SQL syntax.

Transparent routing rules The ContainerDB is connected to the user through a Gate cluster. The Gate obtains all tables involved in the query based on the syntax tree and query execution plan generated by the query statement sent by the user. The fragmented information of each table is obtained based on the metadata in the Topology. Finally, the query or write is routed to the correct shard by combining the association condition in the Join statement with the predicate information in the Where clause. The whole process is automated by Gate and completely transparent to the user. Self-service Service ContainerDB abstracts functions such as database service instantiation, DDL/DML execution, sharding upgrade, and expansion into independent interfaces, and provides streamlined self-service user access service with the help of a process engine. After users successfully apply for database service, ContainerDB automatically pushes the database access password to the user’s email box. 3. The past is gone, the future is here. We’ll be thinking more about the value that databases can generate from a user’s perspective. We believe that the future database service of JD will be: More Smart: Based on the monitoring data of different resources such as CPU, memory and hard disk in each database instance, we will conduct deep learning and cluster analysis to analyze the tendency resources of each database instance, and intelligently adjust the limit of the tendency resources of each database instance and lower the limit of the non-tendency resources. More Quick: We will analyze the corresponding relationship between the host and the container, the restriction parameters of each container and the historical resource growth rate of each container in real time. We will sort out the fragments of the host where the container is located in advance, so as to ensure that each container can be upgraded vertically to achieve capacity expansion, thus greatly speeding up capacity expansion. More Cheap: We will provide a storage engine completely developed by ourselves. We plan to realize the integration of query engine and storage engine, and provide a multi-model database engine, so as to realize the unification of various data models and greatly save resources required by database services and research and development costs. More Friendly: Both ContainerDB and our own multi-model database are fully compatible with the MySQL protocol and syntax, making the migration cost of existing applications close to zero. More Open: ContainerDB will embrace Open source through various scenarios within JD.com and look forward to working with others in the industry to improve ContainerDB. Our subsequent multi-model database will eventually be made available to the open source community, and we look forward to making it available to the industry.