TiDB Operator is an automated deployment operation and maintenance tool of TiDB on Kubernetes platform. Currently, TiDB Operator is officially open source (Pingcap /tidb-operator). With the help of TiDB Operator, TiDB can seamlessly run on Kubernetes platform provided by public Cloud vendors, making TiDB a real cloud-native database.

To understand TiDB Operator, you need to have some understanding of TiDB and Kubernetes first. I believe those who have been paying attention to TiDB for a long time may be familiar with TiDB. This article will start with a brief introduction to TiDB and Kubernetes, talk about why we want to do TiDB Operator, and then talk about how to quickly experience TiDB Operator, And how to participate in the TiDB Operator project to become Contributor.

Introduction to TiDB and Kubernetes

TiDB, as an open source distributed database product, has strong consistency of multiple copies and can be flexibly scaled according to business requirements, and has no awareness of the upper-layer business during expansion and contraction. TiDB consists of three core components: TiDB/TiKV/PD.

  • TiDB Server: The parser and optimizer primarily responsible for SQL, which acts as the computing execution layer, as well as client access and interaction.

  • TiKV Server: it is a set of distributed key-value storage engine, which undertakes the storage layer of the whole database. The horizontal expansion of data and the high availability of multiple copies are realized in this layer.

  • PD Server: equivalent to the brain of distributed database. On the one hand, PD is responsible for collecting and maintaining the distribution of data in each TiKV node; on the other hand, PD plays the role of scheduler and adopts appropriate scheduling strategies according to data distribution and the load of each storage node to maintain the balance and stability of the whole system.

With the three components above, each role is a multi-node cluster, so the final architecture of TiDB looks like this.

Kubernetes was first born as a pure container orchestration system. After deploying Kubernetes cluster, users directly use its built-in functions to deploy application services.

As the PaaS platform is very convenient to use and attracts many users, different users have different requirements. Some features require Kubernetes to implement directly in its core code, but some features are not suitable for merging into the trunk branch.

To meet these needs, Kubernetes has opened up some apis for users to extend and implement their own needs. The API inside Kubernetes is becoming more and more open, making it more like an operating system running on the cloud. Users can use it as an SDK or Framework for the cloud and easily develop components to extend and meet their business needs. Support for stateful services is a good example.

Why are we doing TiDB Operator

First, there are high deployment and operational costs associated with using traditional automation tools. The layered architecture of TiDB is common for distributed systems. Each component can scale independently and horizontally according to business requirements, and TiKV and TiDB can be used independently. For example, a Redis compliant KV database can be built on top of TiKV, and TiDB can also connect to KV storage engines like LevelDB.

However, this multi-component distributed system increases the cost of manual deployment and operation and maintenance. Some traditional automated deployment and operational tools such as Puppet/Chef/SaltStack/Ansible, due to lack of global state management, can’t do automatic failover of various anomalies in time, and it is difficult to play the elastic scalability of distributed system. Some of them require a lot of DSLS or even mixed with Shell scripts, which have poor portability and high maintenance costs.

Secondly, in the cloud era, containers have become the basic unit of application distribution and deployment, and Kubernetes, an open source container orchestration system launched by Google based on the experience of Borg, a container orchestration system that has been used internally for decades, has become the de facto standard of the current container orchestration technology. Now the major cloud manufacturers have begun to provide hosted Kubernetes cluster, deployed in Kubernetes platform applications can not bind to a specific cloud platform, easily realized in a variety of cloud platforms between the migration, its container packaging and publishing method also solves the dependence on the operating system environment.

At the very beginning, Kubernetes project only supports the management of Stateless Service. The Stateless service uses ReplicationController to define multiple copies. The Kubernetes scheduler decides to start multiple PODS on different nodes for load balancing and failover. For stateless services, multiple copies of the corresponding Pod are equivalent, so in the event of a node failure, starting a Pod on a new node is equivalent to a failed Pod, and there is no state migration involved, so management is very simple.

However, for Stateful Service, data needed to be persisted to disks, so that different pods could not be considered equivalent, and could not be randomly scheduled and migrated as stateless Service.

Kubernetes V1.3 introduced the concept of PetSet to manage stateful services and renamed it StatefulSet in V1.5. StatefulSet explicitly defines the identity of each Pod in a set, and startup and upgrade are performed in a specific order. PersistentVolume is also used to store data. When a node fails and a Pod needs to be migrated, the corresponding PVS are remounted. The underlying PVS are based on distributed file systems, so the Pod can still access the previous data. At the same time, when Pod migrates, its network identity, such as IP address, will change. Many distributed systems cannot accept this situation. Therefore, StatefulSet can bind domain names when migrating pods to ensure that Pod network identity does not change in the cluster.

However, due to the particularity of stateful service, Kubernetes will not automatically fail over nodes when they are abnormal for the sake of data security as stateless service does. Although network storage can be mounted to different nodes for use by pods on them, simply mounting network PVS to other nodes is risky in the event of a node failure.

Kubernetes determines node failure based on whether the Kubelet service deployed on each node can report node status normally. Whether Kubelet can work properly is not necessarily related to user applications. In some special cases, the Kubelet service process may not start normally. However, the business container on the node is still running, and mounting the PV to another node may cause double-write problems.

To deploy and manage TiDB as a stateful service on Kubernetes, we need to extend the capabilities of StatefulSet. TiDB Operator is a TiDB cluster management and operation tool developed based on Kubernetes built-in StatefulSet.

Kubernetes didn’t introduce native PVS experimentally until V1.7. Prior to that, only network PVS, TiKV itself had multiple copies for data storage. Multiple copies of network PVS would increase data redundancy and reduce TiDB performance. Before this, we implemented local PV to meet TiKV’s disk IO requirements based on Kubernetes built-in hostPath Volume. The official local PV scheme has not been able to support scheduling functions in a relatively stable manner until recently Kubernetes V1.10 to meet user needs for local PV. In order to reduce user usage and management costs and embrace the Kubernetes open source community, we re-implemented data management based on the official local PV solution.

TiDB Operator principle parsing

Operator is essentially the Controller of Kubernetes. Its core idea is that the user gives a Spec description file, and the Controller creates corresponding resources in the Kubernetes cluster according to the Spec changes. And constantly adjust the state of the resource to match the Spec expected by the user.

The diagram above shows the TiDB Operator workflow diagram, where TidbCluster is a built-in Resource type extended by CRD (Custom Resource Definition) :

  1. Users use Helm to create or update TidbCluster objects to Kubernetes API Server.

  2. The TiDB Operator creates updates or deletes from the TidbCluster object in the Watch API Server and maintains PD/TiKV/TiDB StatefulSet, Service and Deployment object updates.

  3. Kubernetes creates updates or deletes corresponding containers and services based on StatefulSet, Service, and Deployment objects.

In step 2, TiDB Operator updates objects such as StatefulSet by referring to the cluster status given by PD API for operation and maintenance of TiDB cluster. Through the dynamic scheduling of TiDB Operator and Kubernetes, the TiDB cluster that meets the user’s expectation is created.

Experience the TiDB Operator quickly

TiDB Operator must run Kubernetes V1.10 or later. The deployment and management of TiDB Operator and TiDB cluster is implemented by the package management tool Helm on Kubernetes platform. Before running TiDB Operator, ensure that the Helm is correctly installed in the Kubernetes cluster.

If you do not have a Kubernetes cluster, you can quickly start a multi-node Kubernetes cluster locally using the script provided by TiDB Operator:

git clone https://github.com/pingcap/tidb-operator
cd tidb-operator
NUM_NODES=3    # the default node number is 2
KUBE_REPO_PREFIX=uhub.ucloud.cn/pingcap manifests/local- dind/dind - cluster - v1.10. Sh upCopy the code

Once the Kubernetes cluster is ready, you can deploy the TiDB Operator and TiDB cluster through the Helm and Kubectl installation.

  1. Install TiDB Operator

    kubectl apply -f manifests/crd.yaml
helm install charts/tidb-operator --name=tidb-operator --namespace=tidb-	admin
Copy the code

  2. Deploy the TiDB cluster

    helm install charts/tidb-cluster --name=demo-tidb --namespace=tidb --set clusterName=demo
Copy the code

By default, the cluster uses local-storage as the data store for PD and TiKV. To use other persistent storage, modify storageClassName in Charts /tidb-cluster/values.yaml.

Participate in TiDB Operator

TiDB Operator enables TiDB to become a cloud-native database in the true sense. Open source is just a starting point, which requires the joint participation of TiDB community and Kubernetes community.

When you find bugs or missing features, you can directly submit an issue or PR on GitHub and join the discussion together. To become a Contributor, refer to this document.

Author: Deng Shuan