On August 3, TDengine released its v2.0 version. The most important thing about this update is that we open source distributed clustering. After open source, it got a lot of buzz, and it ranked number one on GitHub trending list for several days. Many people who pay attention to TDengine have marveled that you dare to open source the features that users most need. I would say that not only do we have the guts to open source clustering, but we also have the guts to teach you how to use it. Now we will install, manage the cluster documentation published, the whole process is extremely simple, welcome to use.
Multiple running instances of TAOSD can be grouped into a cluster to ensure the reliable running of the TDengine and provide horizontal scaling capabilities. To understand TDengine 2.0 cluster management, you need to understand the basic concepts of clusters. See the TDengine 2.0 Overall Architecture chapter. And before installing the cluster, follow theBegin immediately”Installs and experiences the single-node functionality.
Each node in the cluster is uniquely identified by the End Point. The End Point consists of the Fully Qualified Domain Name (FQDN) plus the Port, for example, h1.taosdata.com:6030. Generally, the FQDN is the hostname of the server. You can run the hostname -f command in Linux to obtain the FQDN. The default port number is 6030. You can use the serverPort parameter in taos. CFG to change the value. A node may be configured with multiple hostnames.tdEngine automatically obtains the first hostname, but this can also be specified using the configuration parameter FQDN in taos.cfg. If an IP address is used for direct access, set FQDN to the IP address of the local node.
The cluster management of TDengine is extremely simple. Except for adding and deleting nodes, all the tasks are done automatically, which minimizes the workload of operation and maintenance. This chapter describes cluster management operations in detail.
The preparatory work
Step 1: If the cluster node has test data, 1.x version, or other versions of TDengine installed, delete it and clear all the data. For details, see the blog “Installing and Uninstalling TDengine Packages”.
Step 2: Disable the firewall and ensure that TCP and UDP ports 6030-6042 are open. It is strongly recommended to disable the firewall and configure the port after the cluster is set up.
Step 3: Install the TDengine of the same version on all nodes, but do not start TAOSD.
Step 4: Check and configure the FQDN of all nodes: Run the hostname -f command on each node to check and confirm that the hostname of all nodes is different. Run the ping host command on each node, where host is the hostname of other nodes, to check whether other nodes can be pinged through. If the ping fails, check the network Settings, /etc/hosts file, or DNS configuration. If the ping fails, the cluster cannot be formed. The FQDN of each node is the output hostname plus the port number, such as h1.taosdata.com:6030
Step 5: Modify the configuration file of the TDengine (the /etc/taos/taos.cfg file on all nodes must be modified). Assuming the End Point of the first node to be started is h1.taosdata.com:6030, the following parameters are cluster related:
// firstEp is the first node connected after each node is started. FirstEp h1.taosdata.com:6030 H1.taosdata.com // Set the port number of the local node. The default port number is 6030. ServerPort 6030 // If the number of copies is even, you need to set this parameter. See part of Arbitrator ha.taosdata.com:6042 in Use of ArbitratorCopy the codeThe one parameter to change is firstEp. Don’t change the other parameters unless you know exactly why you want to change them.
Start the first node
Start the first node h1.taosdata.com as instructed in Start Now, then run taos, start the taOS shell and run the command “show dnodes;” from the shell. , as shown below:
Welcome to the TDengine shell from Linux, Client Version:2.0.0.0 Copyright (c) 2017 by TAOS Data, Inc. All rights reserved. taos> show dnodes; id | end_point | vnodes | cores | status | role | create_time | ===================================================================================== 1 | h1.taos.com:6030 | 0 | 2 | Ready | any 03:49:29. | 2020-07-31 | 202 Query OK, 1 row (s) in the set (0.006385 s) taos >Copy the codeEnd Point: h1.taos.com:6030
Starting subsequent nodes
To add subsequent nodes to the existing cluster, follow the following steps:
-
Start TAOSD on each node as described in the Start Now chapter.
-
On the first node, use the CLI program taos to log in to the TDengine system and run the following command:
The CREATE DNODE "h2.taos.com: 6030";Copy the codeAdd the new node’s End Point (learned in step 4 of the preparation) to the cluster’s EP list. “FQDN :port” must be enclosed in double quotation marks. Otherwise, an error occurs. Note that the example “h2.taos.com:6030” is replaced with the End Point of this new node.
- And then execute the command
SHOW DNODES;Copy the codeCheck whether the new node is added successfully. If the node to be added is offline, perform the following operations:
- Check whether the TAOSD of the node is working properly. If not, check why
- Check the first few lines of the taosdlog file taosdlog.0 (usually in /var/log/taos) to check whether the FQDN and port number of the node in the log file are the newly added End Point. If not, the correct End Point needs to be added.
Follow the steps above to continuously add new nodes to the cluster.
Note: The firstEp parameter is valid only when the node is added to the cluster for the first time. After the node is added to the cluster, the node saves the latest End Point list of mNodes and does not depend on the two parameters. After the two DNodes without firstEp parameters are started, they run independently. At this point, one node cannot be added to another node to form a cluster. Unable to merge two separate clusters into a new cluster.
Node management
Add a node
Run the CLI program taos, log in to the system as user root, and run the following command:
The CREATE DNODE "FQDN: port";Copy the codeAdd the End Point of the new node to the cluster EP list. “FQDN :port” must be enclosed in double quotation marks. Otherwise, an error occurs. The FQDN and port of a node can be configured using the taos. CFG configuration file. By default, the FQDN and port are automatically obtained.
Remove nodes
Run the CLI program taos, log in to the TDengine system as user root, and run the following command:
DROP DNODE "fqdn:port";
Copy the codeFQDN is the FQDN of the node to be deleted, and port is the port number of the external server
Look at the node
Run the CLI program taos, log in to the TDengine system as user root, and run the following command:
SHOW DNODES;
Copy the codeIt lists all dNodes in the cluster, FQDN :port for each Dnode, status (ready, offline, etc.), number of Vnodes, number of unused Vnodes, and so on. You can run this command to view a node after it is added or deleted.
View virtual node groups
In order to make full use of the multi-core technique and achieve scalability, data fragmentation is required. Therefore, the TDengine divides a DB of data into multiple portions and stores them in multiple VNodes. These Vnodes may be distributed among multiple DNodes, thus achieving horizontal scaling. A Vnode belongs to only one DB, but a DB can have multiple Vnodes. Vnodes are automatically allocated by MNodes based on the current system resources without any human intervention.
Run the CLI program taos, log in to the TDengine system as user root, and run the following command:
SHOW VGROUPS;
Copy the codeHigh availability of VNodes
TDengine provides high availability of systems, including vNode and MNode, through the multi-copy mechanism.
The number of vNode copies is associated with the DB. A cluster can have multiple DB copies. The number of vNode copies can be configured for each DB based on operation requirements. When creating a database, the replica number is specified by the parameter Replica (1 by default). If the number of copies is 1, the reliability of the system cannot be guaranteed. As long as the node where data resides breaks down, services cannot be provided. The number of nodes in the cluster must be greater than or equal to the number of copies. Otherwise, the error “More Dnodes are needed” is returned during table creation. For example, the following command will create database demo with 3 copies:
CREATE DATABASE demo replica 3;
Copy the codeThe data in a DB is divided into multiple Vnode groups. The number of Vnodes in a Vnode group is the number of DB copies. The data of vnodes in a Vnode group is completely the same. To ensure high availability, vnodes in a Vnode group must be distributed among different Dnodes (in actual deployment, on different physical machines). As long as more than half of the Vnodes in a Vgroup are working, the Vgroup can provide normal external services.
A DNode may contain multiple DB data. Therefore, when a DNode goes offline, multiple DB data may be affected. If half or more of the Vnodes in a Vnode group are not working, the Vnode group cannot provide external services and cannot insert or read data. As a result, read and write operations on some tables in the DB to which the Vnode group belongs are affected.
Because of the introduction of Vnodes, it is not easy to conclude that “if half of the dnodes in the cluster work, the cluster should work”. But for the simple case, it’s pretty straightforward. For example, if the number of copies is 3 and there are only three Dnodes, the whole cluster can work normally if only one node does not work, but if two nodes do not work, the whole cluster cannot work normally.
High availability of MNodes
The TDengine cluster is managed by mNodes (logical nodes of TAOSD). To ensure high availability of MNodes, you can configure multiple Copies of MNodes. The number of copies is determined by the system configuration parameter numOfMnodes. To ensure strong metadata consistency, data is replicated between mNode replicas through synchronization.
There are multiple DNodes in a cluster, but each Dnode runs at most one MNode instance. In the case of multiple DNodes, which Dnode can be used as mNode? This is automatically specified by the system based on the overall system resources. You can use the CLI program taos to run the following commands on the console of the TDengine:
SHOW MNODES;
Copy the codeTo see the mNode list, which lists the End points and roles (master, slave, unsynced, or offline) of the DNode in which the mnode is located. When the first node in the cluster starts, it must run an mNode instance, otherwise the DNode will not work properly, because a system must have at least one MNode. If numOfMnodes is set to 2, when a second DNode is started, that dnode will also run an mNode instance.
To ensure high availability of mNode services, numOfMnodes must be set to 2 or larger. Because mNode must hold strongly consistent metadata, if numOfMnodes is greater than 2, the quorum parameter is automatically set to 2, that is, at least two copies must be successfully written before the client application is notified of a successful write.
Note: A TDengine high availability system, whether vNode or MNode, must be configured with multiple copies.
Load balancing
There are three scenarios that trigger load balancing, and none of them require human intervention.
- When a new node is added to the cluster, the system automatically triggers load balancing, and data on some nodes is automatically transferred to the new node without any human intervention.
- When a node is removed from the cluster, the system automatically transfers the data on the node to other nodes without any human intervention.
- If a node is overheated (there is too much data), the system automatically balances the load and moves some Vnodes of the node to other nodes.
When these three situations occur, the system initiates a load calculation on each node to determine how to move.
Offline node processing
If a node is offline, the TDengine cluster will automatically detect it. There are two cases:
If a node is offline for more than a certain period (offlineThreshold control period in taos. CFG), the system automatically deletes the node, generates an alarm, and triggers the load balancing process. If the deleted node goes online again, it cannot be added to the cluster and can work only after the system administrator adds it to the cluster again. After being offline, the node goes online again within the offlineThreshold, and the system automatically starts the data recovery process. After the data recovery is complete, the node starts to work normally.
Note: If each node in a virtual node group (including Mnode group) is in offline or unsynced state, the Master can be selected only after all nodes in the virtual node group are online and can exchange status information, and then the virtual node group can provide external services. For example, a cluster has three nodes and the number of copies is three. If all three nodes break down and two nodes are restarted, they cannot work. External services can be provided only after all three nodes are restarted.
The use of the Arbitrator
If the number of copies is even, it is impossible to select a master from a vnode group when half or more of the Vnodes in the group are not working. Similarly, if half or more of the mNodes are not working, the mnode master cannot be selected because of a “split brain” problem. To address this issue, TDengine introduced the concept of arbitrator. Arbitrator Simulates a Vnode or MNode at work, but simply connects to the web and does not handle any data insertion or access. As long as more than half of vnodes or MNodes including arbitrators are working, the Vnode group or MNode group can normally provide data insertion or query services. For example, in the case of 2 copies, if one node A is offline, but the other node B is normal and can be connected to the arbitrator, then node B will work.
The TDengine installation package comes with an executable program called Tarbitrator, which you can run on any Linux server. The program has almost no requirements for system resources, just to ensure that there is a network connection. The -p command line parameter specifies the port number for external services. The default port number is 6042. When configuring each TAOSD instance, set arbitrator to the End Point of the arbitrator parameter in the taos. CFG configuration file. If this parameter is set to arbitrator and the number of copies is even, the system automatically connects to the configured arbitrator.