background
TDengine has been used on a single machine since TDengine opened source its clustering solution in 2020, so let’s take a look at TDengine clustering.
Currently, two VMS are used. (The first one is the standalone version, whose FQDN is HadoOP1. The second machine is a newly created VIRTUAL machine, and FQDN is Hadoop2). An even number of machines may be brain-split. Therefore, TDengine also provides an arbitrator, which is not configured here. For details, please refer to the official documents. Main steps:
- Stop the service without deleting data
- Configure hostname so that the two VMS can ping each other.
- Turn off the firewall;
- Modify the taos.cfg configuration.
- Start the first Taosd service node.
- Start the second Taosd service node.
- Add the second node to the taOS command line of the first node;
- Verify cluster;
1. Stop the service and do not delete data
[root@hadoop1 vnode]# systemctl stop taosd
If the second or subsequent machine does not start taosd, there is no need to stop it
[root@hadoop2 vnode]# systemctl stop taosd
Copy the codeNote: It is officially recommended to delete data, but I did not delete it here. In the process of building the cluster, I also observed the dynamic allocation process of VNodes, for those in the know.
2. Configure hostname so that the two VMS can ping each other
# 虚拟机1
[root@hadoop1 ~]# vi /etc/hosts
192.168.169.130 hadoop1
192.168.169.132 hadoop2
[root@hadoop1 ~]# hostname -f
hadoop1
# 虚拟机2
[root@hadoop2 ~]# vi /etc/hosts
192.168.169.130 hadoop1
192.168.169.132 hadoop2
[root@hadoop2 ~]# hostname -f
hadoop2
Copy the code3. Disable the firewall
# 虚拟机1
[root@hadoop1 ~]# systemctl stop firewalld.service
# 虚拟机2
[root@hadoop2 ~]# systemctl stop firewalld.service
Copy the code4. Modify the taos. CFG configuration
# 虚拟机1
[root@hadoop1 ~]# vi /etc/taos/taos.cfg
# first fully qualified domain name (FQDN) for TDengine system
firstEp hadoop1:6030
# local fully qualified domain name (FQDN)
fqdn hadoop1
# 虚拟机2
[root@hadoop2 ~]# vi /etc/taos/taos.cfg
# first fully qualified domain name (FQDN) for TDengine system
firstEp hadoop1:6030
# local fully qualified domain name (FQDN)
fqdn hadoop2
Copy the code1. FirstEp is set to the address of the first node, I started with firstEp for the second node as hadoop2:6030, that must not get up.
5. Start the first Taosd service node
[root@hadoop1 ~]# systemctl start taosd
Copy the code6. Start the second Taosd service node
[root@hadoop2 ~]# systemctl start taosd
Copy the code7. Add the second node to the taOS command line of the first node
taos> create dnode 'hadoop2:6030';
Copy the codeNotes:
- In fact, if the second or subsequent
taosdIf the cluster is not installed, the system will ask you if you want to join an existing cluster when the installation is complete. - Any data node that has been added to the cluster and is online can be used as the node to be added later
firstEP.
8. Verify the cluster
taos> show dnodes; id | end_point | vnodes | cores | status | role | create_time | offline reason | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1 | hadoop1:6030 | | | 4 ready | any 12:59:24. | 2021-06-15 | 184 | 3 | hadoop2:6030 | | | 4 ready | Any | 2021-07-26 14:16:08. 726 | | Query OK, 2 row (s)in set(0.001604 s)Copy the codeEverything is OK~
Cluster Synchronization process
Before looking at cluster synchronization, take a look at the directory structure of TDengine on the server side. You can see that TDengine’s data and log directories are soft links.
Since my first node already had some data before, and I did not delete the historical database tables when I set up the cluster, some magical changes occurred during this process, which is briefly described below:
- The first node before setting up the cluster
Initially, node 1 owns all vNodes.
[root@hadoop1 vnode]# du -sh *
43M vnode125
43M vnode126
4.3M vnode127
28K vnode163
20K vnode164
308K vnode2
306M vnode4
306M vnode5
204M vnode6
204M vnode7
Copy the code- Synchronizing the cluster
The following procedure shows the synchronization of vNodes (10:0 — >9:2 — >5:5) and the load balancing process.
taos> show dnodes; id | end_point | vnodes | cores | status | role | create_time | offline reason | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1 | hadoop1:6030 | | | 4 10 ready | any 12:59:24. | 2021-06-15 | 184 | Query OK, 1 row (s)in set(0.001095 s) taos > create dnode'hadoop2:6030';
Query OK, 0 of 0 row(s) inDatabase (0.001380s) taos> show dnodes; id | end_point | vnodes | cores | status | role | create_time | offline reason | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1 | hadoop1:6030 | | | 4 10 ready | any 12:59:24. | 2021-06-15 | 184 | 3 | hadoop2:6030 | | | 0 0 offline 726 | | any 14:16:08. | 2021-07-26 status not received | Query OK, 2 row (s)in set(0.000932 s) taos > show dnodes; id | end_point | vnodes | cores | status | role | create_time | offline reason | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1 | hadoop1:6030 | | | 9 4 ready | any 12:59:24. | 2021-06-15 | 184 | 3 | hadoop2:6030 | 2 | | 4 ready | Any | 2021-07-26 14:16:08. 726 | | Query OK, 2 row (s)in set(0.002130 s) taos > show dnodes; id | end_point | vnodes | cores | status | role | create_time | offline reason | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 1 | hadoop1:6030 | | | 4 ready | any 12:59:24. | 2021-06-15 | 184 | 3 | hadoop2:6030 | | | 4 ready | Any | 2021-07-26 14:16:08. 726 | | Query OK, 2 row (s)in set(0.001604 s)Copy the code- After cluster synchronization
Finally, node 1 and node 2 split all vNodes equally.
# 虚拟机1
[root@hadoop1 vnode]# llTotal Usage 0 drwxr-xr-x 4 root root 68 July 19 13:54 vnode125 DRwxr-xr-x 4 root root 68 July 19 13:58 vnode126 drwxr-xr-x 4 root Root 68 July 21 18:05 vnode127 drwxr-xr-x 4 root root 68 July 22 13:56 vnode163 drwxr-xr-x 4 root root 68 July 22 13:56 vnode164# 虚拟机2
[root@hadoop2 vnode]# llTotal amount 0 drwxr-xr-x. 4 root root 68 July 26 14:25 vnode2 drwxr-xr-x. 4 root root 68 July 26 14:25 vnode4 drwxr-xr-x Root 68 July 26 14:26 vnode5 drwxr-xr-x. 4 root root 68 July 26 14:26 vnode6 drwxr-xr-x. 4 root root 68 July 26 14:26 vnode7Copy the codeAs you can see, data is distributed to different TDengine nodes for fragmented storage, achieving horizontal scaling and load balancing.
Viewing management Nodes
Two nodes, one master and one slave.
taos> show mnodes; id | end_point | role | create_time | =================================================================================== 1 | hadoop1:6030 | master | 2021-06-15 12:59:24. 184 | 3 | hadoop2:6030 | slave 14:23:56. | 2021-07-26 | 898 Query OK, 2 row (s)in set(0.003355 s)Copy the codeDeleting a Data Node
taos> drop dnode "fqdn:port";
Copy the codeNote:
- Once a data node is dropped, it cannot be added to the cluster again. This node needs to be redeployed (emptying the data folder). Before dropping a DNode, the cluster migrates data from the DNode.
- Note that dropping a Dnode and stopping a Taosd process are two different concepts. Do not confuse them. Before deleting a Dnode, data must be migrated, so the deleted Dnode must remain online. Stop the Taosd process only after the deletion is complete.
- After a data node is dropped, other nodes will be aware of the dnodeID deletion operation, and no node in the cluster will receive the dnodeID request again.
- The dnodeID is automatically assigned by the cluster and cannot be manually specified. It increments at generation time and does not repeat.
Synchronous log
I specifically looked at the synchronization log when the second node is added, attached below, which is probably the intra-cluster communication, implementing the vNode migration and load balancing process. Although two databases are at the 100 million level, the synchronization takes only a few seconds ୧(̀◡•́๑)૭, thanks to the excellent compression performance of TDengine
new log file
==================================================
07/26 14:25:17.226219 00023008 DND start to initialize TDengine
07/26 14:25:17.226262 00023008 DND step:dnode-tfile is initialized
07/26 14:25:17.226272 00023008 DND step:dnode-rpc is initialized
07/26 14:25:17.226277 00023008 UTL localEp is: hadoop2:6030
07/26 14:25:17.228221 00023008 UTL WARN timezone not configured, set to system default: (CST, +0800)
07/26 14:25:17.228303 00023008 UTL WARN locale not configured, set to system default:en_US.UTF-8
07/26 14:25:17.228315 00023008 UTL WARN charset not configured, set to system default:UTF-8
07/26 14:25:17.228353 00023008 UTL check global cfg completed
07/26 14:25:17.228357 00023008 UTL ==================================
07/26 14:25:17.228361 00023008 UTL taos config & system info:
07/26 14:25:17.228364 00023008 UTL ==================================
07/26 14:25:17.228367 00023008 UTL firstEp: hadoop1:6030
07/26 14:25:17.228370 00023008 UTL secondEp: hadoop2:6030
07/26 14:25:17.228373 00023008 UTL fqdn: hadoop2
07/26 14:25:17.228376 00023008 UTL serverPort: 6030
07/26 14:25:17.228380 00023008 UTL configDir: /etc/taos
07/26 14:25:17.228383 00023008 UTL logDir: /var/log/taos
07/26 14:25:17.228386 00023008 UTL scriptDir: /etc/taos
07/26 14:25:17.228389 00023008 UTL arbitrator:
07/26 14:25:17.228392 00023008 UTL numOfThreadsPerCore: 1.000000
07/26 14:25:17.228403 00023008 UTL numOfCommitThreads: 4
07/26 14:25:17.228407 00023008 UTL ratioOfQueryCores: 1.000000
07/26 14:25:17.228410 00023008 UTL numOfMnodes: 3
07/26 14:25:17.228413 00023008 UTL vnodeBak: 1
07/26 14:25:17.228416 00023008 UTL telemetryReporting: 1
07/26 14:25:17.228419 00023008 UTL balance: 1
07/26 14:25:17.228422 00023008 UTL balanceInterval: 300
07/26 14:25:17.228425 00023008 UTL role: 0
07/26 14:25:17.228428 00023008 UTL maxTmrCtrl: 512
07/26 14:25:17.228431 00023008 UTL monitorInterval: 30(s)
07/26 14:25:17.228434 00023008 UTL offlineThreshold: 864000(s)
07/26 14:25:17.228437 00023008 UTL rpcTimer: 300(ms)
07/26 14:25:17.228440 00023008 UTL rpcForceTcp: 0
07/26 14:25:17.228443 00023008 UTL rpcMaxTime: 600(s)
07/26 14:25:17.228446 00023008 UTL statusInterval: 1(s)
07/26 14:25:17.228449 00023008 UTL shellActivityTimer: 3(s)
07/26 14:25:17.228452 00023008 UTL minSlidingTime: 10(ms)
07/26 14:25:17.228455 00023008 UTL minIntervalTime: 10(ms)
07/26 14:25:17.228458 00023008 UTL maxStreamCompDelay: 20000(ms)
07/26 14:25:17.228461 00023008 UTL maxFirstStreamCompDelay:10000(ms)
07/26 14:25:17.228464 00023008 UTL retryStreamCompDelay: 10000(ms)
07/26 14:25:17.228467 00023008 UTL streamCompDelayRatio: 0.100000
07/26 14:25:17.228470 00023008 UTL maxVgroupsPerDb: 0
07/26 14:25:17.228473 00023008 UTL maxTablesPerVnode: 1000000
07/26 14:25:17.228476 00023008 UTL minTablesPerVnode: 1000
07/26 14:25:17.228479 00023008 UTL tableIncStepPerVnode: 1000
07/26 14:25:17.228482 00023008 UTL cache: 16(Mb)
07/26 14:25:17.228485 00023008 UTL blocks: 6
07/26 14:25:17.228488 00023008 UTL days: 10
07/26 14:25:17.228491 00023008 UTL keep: 3650
07/26 14:25:17.228494 00023008 UTL minRows: 100
07/26 14:25:17.228497 00023008 UTL maxRows: 4096
07/26 14:25:17.228500 00023008 UTL comp: 2
07/26 14:25:17.228503 00023008 UTL walLevel: 1
07/26 14:25:17.228506 00023008 UTL fsync: 3000
07/26 14:25:17.228509 00023008 UTL replica: 1
07/26 14:25:17.228512 00023008 UTL partitions: 4
07/26 14:25:17.228515 00023008 UTL quorum: 1
07/26 14:25:17.228518 00023008 UTL update: 0
07/26 14:25:17.228521 00023008 UTL compressMsgSize: -1
07/26 14:25:17.228524 00023008 UTL maxSQLLength: 1048576(byte)
07/26 14:25:17.228533 00023008 UTL maxNumOfOrderedRes: 100000
07/26 14:25:17.228536 00023008 UTL queryBufferSize: -1(byte)
07/26 14:25:17.228539 00023008 UTL retrieveBlockingModel: 0
07/26 14:25:17.228542 00023008 UTL keepColumnName: 0
07/26 14:25:17.228545 00023008 UTL timezone: (CST, +0800)
07/26 14:25:17.228548 00023008 UTL locale: en_US.UTF-8
07/26 14:25:17.228551 00023008 UTL charset: UTF-8
07/26 14:25:17.228554 00023008 UTL maxShellConns: 50000
07/26 14:25:17.228557 00023008 UTL maxConnections: 5000
07/26 14:25:17.228560 00023008 UTL minimalLogDirGB: 1.000000(GB)
07/26 14:25:17.228564 00023008 UTL minimalTmpDirGB: 1.000000(GB)
07/26 14:25:17.228567 00023008 UTL minimalDataDirGB: 2.000000(GB)
07/26 14:25:17.228570 00023008 UTL mnodeEqualVnodeNum: 4
07/26 14:25:17.228573 00023008 UTL flowctrl: 1
07/26 14:25:17.228576 00023008 UTL slaveQuery: 1
07/26 14:25:17.228579 00023008 UTL adjustMaster: 1
07/26 14:25:17.228582 00023008 UTL http: 1
07/26 14:25:17.228585 00023008 UTL mqtt: 0
07/26 14:25:17.228587 00023008 UTL monitor: 1
07/26 14:25:17.228590 00023008 UTL stream: 1
07/26 14:25:17.228593 00023008 UTL httpEnableRecordSql: 0
07/26 14:25:17.228596 00023008 UTL telegrafUseFieldNum: 0
07/26 14:25:17.228599 00023008 UTL httpMaxThreads: 2
07/26 14:25:17.228602 00023008 UTL restfulRowLimit: 10240
07/26 14:25:17.228605 00023008 UTL numOfLogLines: 10000000
07/26 14:25:17.228608 00023008 UTL logKeepDays: 0
07/26 14:25:17.228611 00023008 UTL asyncLog: 1
07/26 14:25:17.228614 00023008 UTL debugFlag: 0
07/26 14:25:17.228617 00023008 UTL mDebugFlag: 131
07/26 14:25:17.228620 00023008 UTL dDebugFlag: 135
07/26 14:25:17.228623 00023008 UTL sDebugFlag: 135
07/26 14:25:17.228626 00023008 UTL wDebugFlag: 135
07/26 14:25:17.228628 00023008 UTL sdbDebugFlag: 131
07/26 14:25:17.228631 00023008 UTL rpcDebugFlag: 131
07/26 14:25:17.228634 00023008 UTL tmrDebugFlag: 131
07/26 14:25:17.228637 00023008 UTL cDebugFlag: 131
07/26 14:25:17.228640 00023008 UTL jniDebugFlag: 131
07/26 14:25:17.228643 00023008 UTL odbcDebugFlag: 131
07/26 14:25:17.228646 00023008 UTL uDebugFlag: 131
07/26 14:25:17.228649 00023008 UTL httpDebugFlag: 131
07/26 14:25:17.228652 00023008 UTL mqttDebugFlag: 131
07/26 14:25:17.228654 00023008 UTL monDebugFlag: 131
07/26 14:25:17.228657 00023008 UTL qDebugFlag: 131
07/26 14:25:17.228660 00023008 UTL vDebugFlag: 135
07/26 14:25:17.228663 00023008 UTL tsdbDebugFlag: 131
07/26 14:25:17.228666 00023008 UTL cqDebugFlag: 131
07/26 14:25:17.228669 00023008 UTL enableRecordSql: 0
07/26 14:25:17.228672 00023008 UTL enableCoreFile: 0
07/26 14:25:17.228675 00023008 UTL gitinfo: 2019939bcc5567212d6e07af557c2c4ea540c091
07/26 14:25:17.228678 00023008 UTL gitinfoOfInternal: NULL
07/26 14:25:17.228681 00023008 UTL buildinfo: Built at 2021-06-07 14:27
07/26 14:25:17.228684 00023008 UTL version: 2.1.2.0
07/26 14:25:17.228687 00023008 UTL maxBinaryDisplayWidth: 30
07/26 14:25:17.228690 00023008 UTL tempDir: /tmp/
07/26 14:25:17.228693 00023008 UTL os pageSize: 4096(KB)
07/26 14:25:17.228696 00023008 UTL os openMax: 1048576
07/26 14:25:17.228699 00023008 UTL os streamMax: 16
07/26 14:25:17.228702 00023008 UTL os numOfCores: 4
07/26 14:25:17.228705 00023008 UTL os totalDisk: 0.000000(GB)
07/26 14:25:17.228708 00023008 UTL os usedDisk: 0.000000(GB)
07/26 14:25:17.228711 00023008 UTL os availDisk: 0.000000(GB)
07/26 14:25:17.228716 00023008 UTL os totalMemory: 3770(MB)
07/26 14:25:17.228722 00023008 UTL os sysname: Linux
07/26 14:25:17.228725 00023008 UTL os nodename: hadoop2
07/26 14:25:17.228727 00023008 UTL os release: 3.10.0-1127.el7.x86_64
07/26 14:25:17.228730 00023008 UTL os version: #1 SMP Tue Mar 31 23:36:51 UTC 2020
07/26 14:25:17.228733 00023008 UTL os machine: x86_64
07/26 14:25:17.228737 00023008 UTL dataDir: /var/lib/taos
07/26 14:25:17.228740 00023008 UTL ==================================
07/26 14:25:17.228743 00023008 DND step:dnode-globalcfg is initialized
07/26 14:25:17.229123 00023008 TFS disk /var/lib/taos is mounted to tier level 0 id 0
07/26 14:25:17.229446 00023008 DND dnode storage is initialized at /var/lib/taos/dnode
07/26 14:25:17.229455 00023008 DND step:dnode-storage is initialized
07/26 14:25:17.229465 00023008 DND failed to read /var/lib/taos/dnode/dnodeCfg.json, file not exist
07/26 14:25:17.229467 00023008 DND dnode cfg is initialized
07/26 14:25:17.229470 00023008 DND step:dnode-cfg is initialized
07/26 14:25:17.229489 00023008 DND failed to read /var/lib/taos/dnode/dnodeEps.json, file not exist
07/26 14:25:17.229490 00023008 DND dnode eps is initialized
07/26 14:25:17.229494 00023008 DND step:dnode-eps is initialized
07/26 14:25:17.229502 00023008 DND failed to read /var/lib/taos/dnode/mnodeEpSet.json, file not exist
07/26 14:25:17.229503 00023008 DND dnode minfos is initialized
07/26 14:25:17.229506 00023008 DND step:dnode-minfos is initialized
07/26 14:25:17.229565 00023008 WAL wal thread is launched, thread:0x7f1701abd700
07/26 14:25:17.229573 00023008 WAL wal module is initialized, rsetId:3
07/26 14:25:17.229579 00023008 DND step:dnode-wal is initialized
07/26 14:25:17.229777 00023008 SYN 0x557872b4c9a0 TCP pool is created
07/26 14:25:17.229788 00023008 SYN sync module initialized successfully
07/26 14:25:17.229789 00023008 DND step:dnode-sync is initialized
07/26 14:25:17.229871 00023008 DND dnode check is initialized
07/26 14:25:17.229878 00023008 DND step:dnode-check is initialized
07/26 14:25:17.229887 00023008 UTL worker:vquery is initialized, min:4 max:4
07/26 14:25:17.229894 00023008 UTL worker:vfetch is initialized, min:4 max:4
07/26 14:25:17.229897 00023008 DND step:dnode-vread is initialized
07/26 14:25:17.229900 00023008 DND dnode vwrite is initialized, max worker 4
07/26 14:25:17.229903 00023008 DND step:dnode-vwrite is initialized
07/26 14:25:17.229911 00023008 VND vbackup:0 is created
07/26 14:25:17.229913 00023008 VND vbackup is initialized, num:1 qset:0x557872b520e0
07/26 14:25:17.229943 00023008 VND vbackup:0 is launched, total:1
07/26 14:25:17.229945 00023008 VND vbackup queue:0x557872b52170 is allocated
07/26 14:25:17.229947 00023008 DND step:vnode-backup is initialized
07/26 14:25:17.229953 00023008 VND vmworker:0 is created
07/26 14:25:17.229954 00023008 VND vmworker is initialized, num:1 qset:0x557872b52420
07/26 14:25:17.229983 00023008 VND vmworker:0 is launched, total:1
07/26 14:25:17.229985 00023008 VND vmworker queue:0x557872b524b0 is allocated
07/26 14:25:17.229986 00023008 DND step:vnode-worker is initialized
07/26 14:25:17.229992 00023008 DND step:vnode-write is initialized
07/26 14:25:17.229997 00023008 DND step:vnode-read is initialized
07/26 14:25:17.230002 00023008 DND step:vnode-hash is initialized
07/26 14:25:17.230167 00023008 DND step:tsdb-queue is initialized
07/26 14:25:17.230182 00023008 UTL worker:vmgmt is initialized, min:1 max:1
07/26 14:25:17.230229 00023008 DND dnode vmgmt is initialized
07/26 14:25:17.230244 00023008 DND step:dnode-vmgmt is initialized
07/26 14:25:17.230252 00023008 DND dnode mread worker:0 is created
07/26 14:25:17.230253 00023008 DND dnode mread worker:1 is created
07/26 14:25:17.230254 00023008 DND dnode mread is initialized, workers:2 qset:0x557872b53b20
07/26 14:25:17.230255 00023008 DND step:dnode-mread is initialized
07/26 14:25:17.230260 00023008 DND dnode mwrite worker:0 is created
07/26 14:25:17.230261 00023008 DND dnode mwrite is initialized, workers:1 qset:0x557872b53bc0
07/26 14:25:17.230265 00023008 DND step:dnode-mwrite is initialized
07/26 14:25:17.230268 00023008 DND dnode mpeer worker:0 is created
07/26 14:25:17.230269 00023008 DND dnode mpeer is initialized, workers:1 qset:0x557872b53c50
07/26 14:25:17.230270 00023008 DND step:dnode-mpeer is initialized
07/26 14:25:17.232649 00023008 DND dnode inter-dnodes rpc client is initialized
07/26 14:25:17.232669 00023008 DND step:dnode-client is initialized
07/26 14:25:17.233305 00023008 DND dnode inter-dnodes RPC server is initialized
07/26 14:25:17.233317 00023008 DND step:dnode-server is initialized
07/26 14:25:17.233344 00023008 DND start 4 threads to open 0 vnodes
07/26 14:25:17.233349 00023008 DND there are total vnodes:0, opened:0
07/26 14:25:17.233353 00023008 DND step:dnode-vnodes is initialized
07/26 14:25:17.233489 00023008 MND starting to initialize mnode ...
07/26 14:25:17.233528 00023008 DND dnode mwrite worker:0 is launched, total:1
07/26 14:25:17.233531 00023008 DND dnode mwrite queue:0x557872b84f00 is allocated
07/26 14:25:17.233559 00023008 DND dnode mread worker:0 is launched, total:2
07/26 14:25:17.233591 00023008 DND dnode mread worker:1 is launched, total:2
07/26 14:25:17.233597 00023008 DND dnode mread queue:0x557872b85260 is allocated
07/26 14:25:17.233645 00023008 DND dnode mpeer worker:0 is launched, total:1
07/26 14:25:17.233649 00023008 DND dnode mpeer queue:0x557872b85760 is allocated
07/26 14:25:17.233655 00023008 DND step:sdbref is initialized
07/26 14:25:17.233748 00023008 DND step:profile is initialized
07/26 14:25:17.233766 00023008 DND step:cluster is initialized
07/26 14:25:17.233771 00023008 DND step:accts is initialized
07/26 14:25:17.233779 00023008 DND step:users is initialized
07/26 14:25:17.233784 00023008 DND step:dnodes is initialized
07/26 14:25:17.233792 00023008 DND step:dbs is initialized
07/26 14:25:17.233800 00023008 DND step:vgroups is initialized
07/26 14:25:17.234042 00023008 DND step:tables is initialized
07/26 14:25:17.234053 00023008 DND step:mnodes is initialized
07/26 14:25:17.234106 00023008 MND vgId:1, sdb write is opened
07/26 14:25:17.234123 00023008 WAL vgId:1, object is initialized
07/26 14:25:17.234126 00023008 WAL vgId:1, wal:0x557872ba1e10 is opened, level:2 fsyncPeriod:0
07/26 14:25:17.234127 00023008 SDB vgId:1, open sdb wal for restore
07/26 14:25:17.234159 00023008 WAL vgId:1, file:/var/lib/taos/mnode/wal/wal0, will be restored
07/26 14:25:17.234176 00023008 WAL vgId:1, file:/var/lib/taos/mnode/wal/wal0, open for restore
07/26 14:25:17.234205 00023008 MND dnode:1, fqdn:hadoop2 ep:hadoop2:6030 port:6030 is created
07/26 14:25:17.234219 00023008 MND mnode:1, fqdn:hadoop2 ep:hadoop2:6030 port:6030 is created
07/26 14:25:17.234348 00023008 WAL vgId:1, file:/var/lib/taos/mnode/wal/wal0, it is closed after restore
07/26 14:25:17.234352 00023008 WAL vgId:1, file:/var/lib/taos/mnode/wal/wal0, restore success, wver:7
07/26 14:25:17.234368 00023008 WAL vgId:1, file:/var/lib/taos/mnode/wal/wal0, it is created and open while restore
07/26 14:25:17.234370 00023008 SDB vgId:1, sdb wal load success
07/26 14:25:17.234373 00023008 SDB vgId:1, sdb start to check for integrity
07/26 14:25:17.234378 00023008 SDB vgId:1, sdb:cluster is checked, rows:1
07/26 14:25:17.234382 00023008 SDB vgId:1, sdb:dnodes is checked, rows:1
07/26 14:25:17.234385 00023008 MND vgId:1, update mnodes epSet, numOfMnodes:1
07/26 14:25:17.234390 00023008 MND vgId:1, mnodes epSet is set, num:1 inUse:0
07/26 14:25:17.234396 00023008 MND vgId:1, mnode:1, fqdn:hadoop2 shell:6030 peer:6035
07/26 14:25:17.234403 00023008 SDB vgId:1, sdb:mnodes is checked, rows:1
07/26 14:25:17.234406 00023008 SDB vgId:1, sdb:accounts is checked, rows:1
07/26 14:25:17.234410 00023008 SDB vgId:1, sdb:users is checked, rows:3
07/26 14:25:17.234412 00023008 SDB vgId:1, sdb:dbs is checked, rows:0
07/26 14:25:17.234415 00023008 SDB vgId:1, sdb:vgroups is checked, rows:0
07/26 14:25:17.234419 00023008 SDB vgId:1, sdb:stables is checked, rows:0
07/26 14:25:17.234421 00023008 SDB vgId:1, sdb:ctables is checked, rows:0
07/26 14:25:17.234428 00023008 SDB vgId:1, sdb is restored, mver:7 rows:7 tables:9
07/26 14:25:17.234431 00023008 DND step:sdb is initialized
07/26 14:25:17.234503 00023008 MND dnode:1 set access:0 to 0
07/26 14:25:17.234517 00023008 DND step:balance is initialized
07/26 14:25:17.234521 00023008 DND step:grant is initialized
07/26 14:25:17.234646 00023008 DND step:show is initialized
07/26 14:25:17.234659 00023008 MND mnode is initialized successfully
07/26 14:25:17.234786 00023008 DND dnode modules is initialized
07/26 14:25:17.234796 00023008 HTP start http server ...
07/26 14:25:17.234865 00023008 HTP http result queue is opened
07/26 14:25:17.235694 00023008 MON monitor module start
07/26 14:25:17.235730 00023008 DND step:dnode-modules is initialized
07/26 14:25:17.236197 00023042 HTP http server init success at 6041
07/26 14:25:17.236571 00023008 DND dnode shell rpc server is initialized
07/26 14:25:17.236582 00023008 DND step:dnode-shell is initialized
07/26 14:25:17.236588 00023008 DND dnode status timer is initialized
07/26 14:25:17.236591 00023008 DND step:dnode-statustmr is initialized
07/26 14:25:17.236628 00023008 DND dnode telemetry is initialized
07/26 14:25:17.236635 00023008 DND step:dnode-telemetry is initialized
07/26 14:25:17.236638 00023008 DND TDengine is initialized successfully
07/26 14:25:17.803235 00023021 DND dnode:3, in create mnode msg is not equal with saved dnodeId:0
07/26 14:25:17.820577 00023022 DND print minfos, mnodeNum:1 inUse:0
07/26 14:25:17.820626 00023022 DND mnode index:1, hadoop1:6030
07/26 14:25:17.821480 00023022 DND successed to write /var/lib/taos/dnode/mnodeEpSet.json
07/26 14:25:17.821508 00023022 DND dnodeId is set to 3, clusterId is set to d0fcfd0b-16e3-431b-b3cc-c0bba31afafc
07/26 14:25:17.822066 00023022 DND successed to write /var/lib/taos/dnode/dnodeCfg.json
07/26 14:25:17.822091 00023022 DND print dnodeEp, dnodeNum:2
07/26 14:25:17.822095 00023022 DND dnode:1, dnodeFqdn:hadoop1 dnodePort:6030
07/26 14:25:17.822098 00023022 DND dnode:3, dnodeFqdn:hadoop2 dnodePort:6030
07/26 14:25:17.822652 00023022 DND successed to write /var/lib/taos/dnode/dnodeEps.json
07/26 14:25:18.371348 00023021 DND dnode:3, create mnode msg is received from mnodes, numOfMnodes:2
07/26 14:25:18.371392 00023021 DND mnode index:0, mnode:1:hadoop1:6030
07/26 14:25:18.371395 00023021 DND mnode index:1, mnode:3:hadoop2:6030
07/26 14:25:18.371398 00023021 DND start mnode module, module status:6, new status:7
07/26 14:25:18.371436 00023021 DND module status:7 is set, start mnode module
07/26 14:25:18.371445 00023021 MND mnode module already started...
07/26 14:25:18.371455 00023021 MND vgId:1, update mnodes epSet, numOfMinfos:2
07/26 14:25:18.371463 00023021 MND vgId:1, mnodes epSet is set, num:2 inUse:0
07/26 14:25:18.371470 00023021 MND vgId:1, mnode:1, fqdn:hadoop1 shell:6030 peer:6035
07/26 14:25:18.371477 00023021 MND vgId:1, mnode:3, fqdn:hadoop2 shell:6030 peer:6035
07/26 14:25:18.371486 00023021 SDB vgId:1, work as mnode, replica:2
07/26 14:25:18.371493 00023021 SDB vgId:1, mnode:1, hadoop1:6040
07/26 14:25:18.371499 00023021 SDB vgId:1, mnode:3, hadoop2:6040
07/26 14:25:18.371926 00023021 SYN vgId:1, nodeId:1, 0x7f16e0000e40 it is configured, ep:hadoop1:6040 rid:2
07/26 14:25:18.371996 00023021 SYN vgId:1, nodeId:3, 0x7f16e0001010 it is configured, ep:hadoop2:6040 rid:3
07/26 14:25:18.372002 00023021 SYN vgId:1, 2 replicas are configured, quorum:2 role:unsynced
07/26 14:25:18.372107 00023021 SDB vgId:1, mnode role changed from master to unsynced
07/26 14:25:18.372136 00023021 SDB vgId:1, update mnodes role, replica:2
07/26 14:25:18.372148 00023021 SDB vgId:1, mnode:1, role:offline
07/26 14:25:18.372158 00023021 MND vgId:1, update mnodes epSet, numOfMnodes:1
07/26 14:25:18.372167 00023021 MND vgId:1, mnodes epSet not set, num:2 inUse:0
07/26 14:25:18.372173 00023021 MND vgId:1, index:0, ep:hadoop1:6030
07/26 14:25:18.372179 00023021 MND vgId:1, index:1, ep:hadoop2:6030
07/26 14:25:18.373227 00023021 VND vgId:2, not exist
07/26 14:25:18.373246 00023021 DND vgId:2, vnode not exist, can't alter it
07/26 14:25:18.719261 00023021 VND vgId:2, not exist
07/26 14:25:18.719272 00023021 DND vgId:2, create vnode msg is received
07/26 14:25:18.719274 00023021 VND vgId:2, not exist
07/26 14:25:18.719331 00023022 DND print minfos, mnodeNum:2 inUse:0
07/26 14:25:18.719353 00023022 DND mnode index:1, hadoop1:6030
07/26 14:25:18.719357 00023022 DND mnode index:3, hadoop2:6030
07/26 14:25:18.719823 00023022 DND successed to write /var/lib/taos/dnode/mnodeEpSet.json
07/26 14:25:18.719846 00023021 VND vgId:2, successed to write /var/lib/taos/vnode/vnode2/config.json
07/26 14:25:18.719952 00023021 VND vgId:2, vnode dir is created, walLevel:1 fsyncPeriod:3000
07/26 14:25:18.720066 00023021 VND vgId:2, load vnode cfg successfully, replcia:2
07/26 14:25:18.720077 00023021 VND vgId:2, dnode:3, hadoop2:6040
07/26 14:25:18.720094 00023021 VND vgId:2, dnode:1, hadoop1:6040
07/26 14:25:18.720151 00023021 DND dnode vwrite worker:0 is launched
07/26 14:25:18.720155 00023021 DND pVnode:0x7f16e0021250, dnode vwrite queue:0x7f16e0021ca0 is allocated
07/26 14:25:18.720177 00023079 DND dnode vwrite worker:0 is running
07/26 14:25:18.720806 00023021 TDB vgId:2 try to restore meta
07/26 14:25:18.720841 00023021 TDB vgId:2 no meta file is restored
07/26 14:25:18.721568 00023021 WAL vgId:2, object is initialized
07/26 14:25:18.721573 00023021 WAL vgId:2, wal:0x7f16e035d1c0 is opened, level:1 fsyncPeriod:3000
07/26 14:25:18.721586 00023021 WAL vgId:2, file:, it is closed before remove all wals
07/26 14:25:18.721635 00023021 WAL vgId:2, file:/var/lib/taos/vnode/vnode2/wal/wal1, it is created and open while renew
07/26 14:25:18.721726 00023021 VND vgId:2, vnode is opened in /var/lib/taos/vnode/vnode2 - /var/lib/taos/vnode/vnode2, pVnode:0x7f16e0021250
07/26 14:25:18.721817 00023021 SYN vgId:2, nodeId:3, 0x7f16e0375c30 it is configured, ep:hadoop2:6040 rid:4
07/26 14:25:18.721845 00023021 SYN vgId:2, nodeId:1, 0x7f16e0375e00 it is configured, ep:hadoop1:6040 rid:5
07/26 14:25:18.721847 00023021 SYN vgId:2, 2 replicas are configured, quorum:1 role:unsynced
07/26 14:25:18.721906 00023021 VND vgId:2, sync role changed from offline to unsynced
07/26 14:25:18.721916 00023021 DND force send status msg to mnode
07/26 14:25:18.721921 00023021 QRY vgId:2, set querymgmt reopen
07/26 14:25:19.286923 00023014 SYN peer TCP connection from ip:192.168.169.130
07/26 14:25:19.287021 00023014 SYN vgId:2, sync connection is incoming, tranId:9412
07/26 14:25:19.287033 00023014 SYN vgId:2, nodeId:1, TCP connection is up, pfd:8 sfd:-1, old pfd:-1
07/26 14:25:19.287038 00023014 SYN vgId:2, nodeId:1, pfd:-1 sfd:-1 will be closed
07/26 14:25:19.287321 00023014 SYN 0x7f16b80008e0 TCP epoll thread is created
07/26 14:25:19.287365 00023014 SYN 0x7f16b80008e0 fd:8 is added to epoll thread, num:1
07/26 14:25:19.287371 00023014 SYN vgId:2, nodeId:1, ready to exchange data
07/26 14:25:19.287551 00023014 SYN vgId:2, nodeId:1, status is sent, self:unsynced:init:0, peer:offline:init:0, ack:1 tranId:23484 type:exchange-data pfd:8
07/26 14:25:19.288445 00023101 SYN vgId:2, nodeId:1, status is received, self:unsynced:init:0, peer:master:18502, ack:0 tranId:23484 type:exchange-data-rsp pfd:8
07/26 14:25:19.288469 00023101 SYN vgId:2, nodeId:1, peer role:offline change to master
07/26 14:25:19.288474 00023101 SYN vgId:2, peer:vgId:2, nodeId:1 is master, index:1
07/26 14:25:19.288476 00023101 SYN vgId:2, nodeId:1, it is the master, replica:2 sver:18502
07/26 14:25:19.288479 00023101 SYN vgId:2, nodeId:1, is master, sync required, self sver:0
07/26 14:25:19.288483 00023101 SYN vgId:2, nodeId:1, try to sync
07/26 14:25:19.288627 00023101 SYN vgId:2, nodeId:1, sync-req is sent to peer, tranId:15506, sstatus:init
07/26 14:25:19.288635 00023101 SYN vgId:2, roles changed, broadcast status, replica:2
07/26 14:25:19.288646 00023101 SYN vgId:2, nodeId:1, status is sent, self:unsynced:init:0, peer:master:init:18502, ack:1 tranId:15437 type:broadcast pfd:8
07/26 14:25:19.290101 00023101 SYN vgId:2, nodeId:1, status is received, self:unsynced:init:0, peer:master:18502, ack:0 tranId:15437 type:broadcast-rsp pfd:8
07/26 14:25:19.290136 00023101 SYN vgId:2, nodeId:1, peer role:master change to master
07/26 14:25:19.290139 00023101 SYN vgId:2, peer:vgId:2, nodeId:1 is master, index:1
07/26 14:25:19.290143 00023101 SYN vgId:2, nodeId:1, it is the master, replica:2 sver:18502
07/26 14:25:19.290145 00023101 SYN vgId:2, nodeId:1, is master, sync required, self sver:0
07/26 14:25:19.290151 00023101 SYN vgId:2, nodeId:1, try to sync
07/26 14:25:19.290253 00023101 SYN vgId:2, nodeId:1, sync-req is sent to peer, tranId:6619, sstatus:init
07/26 14:25:19.290864 00023014 SYN peer TCP connection from ip:192.168.169.130
07/26 14:25:19.290881 00023014 SYN vgId:2, sync connection is incoming, tranId:45341
07/26 14:25:19.290885 00023014 SYN vgId:2, nodeId:1, sync-data msg from master is received, tranId:45341, set sstatus:start
07/26 14:25:19.290969 00023014 SYN vgId:2, nodeId:1, sync restore thread:0x7f16bf7f6700 create successfully, rid:5
07/26 14:25:19.291054 00023102 SYN vgId:2, nodeId:1, start to restore data, sstatus:start
07/26 14:25:19.291067 00023102 VND vgId:2, sync role changed from unsynced to syncing
07/26 14:25:19.291095 00023102 DND force send status msg to mnode
07/26 14:25:19.291129 00023102 SYN vgId:2, recv buffer:0x7f16b40008c0 is created
07/26 14:25:19.291132 00023102 SYN vgId:2, nodeId:1, start to restore, sstatus:init
07/26 14:25:19.291205 00023102 SYN vgId:2, nodeId:1, send sync rsp to peer, tranId:51924
07/26 14:25:19.291210 00023102 SYN vgId:2, nodeId:1, start to restore file, set sstatus:file
07/26 14:25:19.291213 00023102 VND vgId:2, datafile will be synced
07/26 14:25:19.291223 00023102 VND vgId:2, set to reset status
07/26 14:25:19.291229 00023102 QRY vgId:2, set querymgmt closed, wait for all queries cancelled
07/26 14:25:19.493172 00023014 SYN peer TCP connection from ip:192.168.169.130
07/26 14:25:19.493212 00023014 SYN vgId:1, sync connection is incoming, tranId:33734
07/26 14:25:19.493219 00023014 SYN vgId:1, nodeId:1, TCP connection is up, pfd:35 sfd:-1, old pfd:-1
07/26 14:25:19.493223 00023014 SYN vgId:1, nodeId:1, pfd:-1 sfd:-1 will be closed
07/26 14:25:19.493375 00023014 SYN 0x7f16b8000dd0 TCP epoll thread is created
07/26 14:25:19.493400 00023014 SYN 0x7f16b8000dd0 fd:35 is added to epoll thread, num:1
07/26 14:25:19.493405 00023014 SYN vgId:1, nodeId:1, ready to exchange data
07/26 14:25:19.493589 00023014 SYN vgId:1, nodeId:1, status is sent, self:unsynced:init:7, peer:offline:init:0, ack:1 tranId:14082 type:exchange-data pfd:35
07/26 14:25:19.494726 00023103 SYN vgId:1, nodeId:1, status is received, self:unsynced:init:7, peer:master:81556, ack:0 tranId:14082 type:exchange-data-rsp pfd:35
07/26 14:25:19.494749 00023103 SYN vgId:1, nodeId:1, peer role:offline change to master
07/26 14:25:19.494754 00023103 SYN vgId:1, peer:vgId:1, nodeId:1 is master, index:0
07/26 14:25:19.494757 00023103 SYN vgId:1, nodeId:1, it is the master, replica:2 sver:81556
07/26 14:25:19.494761 00023103 SYN vgId:1, nodeId:1, is master, sync required, self sver:7
07/26 14:25:19.494765 00023103 SYN vgId:1, nodeId:1, try to sync
07/26 14:25:19.494893 00023103 SYN vgId:1, nodeId:1, sync-req is sent to peer, tranId:45740, sstatus:init
07/26 14:25:19.494903 00023103 SYN vgId:1, roles changed, broadcast status, replica:2
07/26 14:25:19.494918 00023103 SYN vgId:1, nodeId:1, status is sent, self:unsynced:init:7, peer:master:init:81556, ack:1 tranId:61374 type:broadcast pfd:35
07/26 14:25:19.496057 00023103 SYN vgId:1, nodeId:1, status is received, self:unsynced:init:7, peer:master:81556, ack:0 tranId:61374 type:broadcast-rsp pfd:35
07/26 14:25:19.496075 00023103 SYN vgId:1, nodeId:1, peer role:master change to master
07/26 14:25:19.496080 00023103 SYN vgId:1, peer:vgId:1, nodeId:1 is master, index:0
07/26 14:25:19.496084 00023103 SYN vgId:1, nodeId:1, it is the master, replica:2 sver:81556
07/26 14:25:19.496088 00023103 SYN vgId:1, nodeId:1, is master, sync required, self sver:7
07/26 14:25:19.496094 00023103 SYN vgId:1, nodeId:1, try to sync
07/26 14:25:19.496219 00023103 SYN vgId:1, nodeId:1, sync-req is sent to peer, tranId:9478, sstatus:init
07/26 14:25:19.496848 00023014 SYN peer TCP connection from ip:192.168.169.130
07/26 14:25:19.496872 00023014 SYN vgId:1, sync connection is incoming, tranId:21097
07/26 14:25:19.496879 00023014 SYN vgId:1, nodeId:1, sync-data msg from master is received, tranId:21097, set sstatus:start
07/26 14:25:19.497034 00023014 SYN vgId:1, nodeId:1, sync restore thread:0x7f16be7f4700 create successfully, rid:2
07/26 14:25:19.497142 00023104 SYN vgId:1, nodeId:1, start to restore data, sstatus:start
07/26 14:25:19.497171 00023104 SDB vgId:1, mnode role changed from unsynced to syncing
07/26 14:25:19.497224 00023104 SDB vgId:1, update mnodes role, replica:2
07/26 14:25:19.497283 00023104 SDB vgId:1, mnode:1, role:master
07/26 14:25:19.497313 00023104 MND vgId:1, update mnodes epSet, numOfMnodes:1
07/26 14:25:19.497327 00023104 MND vgId:1, mnodes epSet not set, num:2 inUse:0
07/26 14:25:19.497336 00023104 MND vgId:1, index:0, ep:hadoop1:6030
07/26 14:25:19.497344 00023104 MND vgId:1, index:1, ep:hadoop2:6030
07/26 14:25:19.497372 00023104 SYN vgId:1, recv buffer:0x7f16a4000910 is created
07/26 14:25:19.497377 00023104 SYN vgId:1, nodeId:1, start to restore, sstatus:init
07/26 14:25:19.497526 00023104 SYN vgId:1, nodeId:1, send sync rsp to peer, tranId:43688
07/26 14:25:19.497537 00023104 SYN vgId:1, nodeId:1, start to restore file, set sstatus:file
07/26 14:25:19.498599 00023104 SYN vgId:1, nodeId:1, all files are restored, fver:0
...
Copy the codeI made a mistake in the beginning
FirstEp is set to the address of the first node, I started with firstEp of the second node set to hadoop2:6030, that must not get up.
Reference
- www.taosdata.com/cn/document…
If you have any questions or any bugs are found, please feel free to contact me.
Your comments and suggestions are welcome!
This article has participated in the activity of “New person creation Ceremony”, and started the road of digging gold creation together.