Abstract: Failures in large-scale distributed systems are unavoidable. When a SINGLE point of failure occurs on a DN, what are the methods to recover the DN and how to recover the DN? This section focuses on how to recover the DN single point of failure by operating gs_ctl build.

This article is shared from huawei Cloud community “Huawei Cloud data warehouse standby MACHINE DN reconstruction, quick repair DN single point of failure!” , by Welblupen.

1. Technical background

The DN of GaussDB(DWS) is in active, standby, and slave standby mode. That is, in a distributed environment, complete cluster data is distributed on multiple DN groups by sharding technology. Each DN group undertakes a data sharding, including a primary DN, a standby DN, and a secondary DN. The active and standby hosts have a complete copy of data. The standby host does not store data and only stores data temporarily when the standby host is faulty. After the standby host recovers from the fault, the standby host connects to the host to copy data and Xlog logs to ensure data consistency in the cluster.

2. The standby DN needs to be rebuilt

2.1. After a single point of failure occurs on the host, the standby host performs failover to become the active host, and the cluster is degraded. After the fault is recovered from the original active node, WAL CRC verification may fail on the active and standby nodes. When detecting the fault, the CM system automatically rebuilds the standby node.

2.2. After a single point of failure occurs on the standby node, the status of the standby node changes to Unknown and the cluster is degraded. After the standby node recovers, you need to rebuild the standby node to synchronize data with the host.

3. Operation classification of secondary DN reconstruction

3.1. Incremental reconstruction: gs_ctl build -b incremental -z datanode

USES:

The incremental Build can be used to resolve the log bifurcations of the secondary server caused by common host or instance faults and the loss of some data files. If a host exception occurs during reconstruction, you can manually roll back the restoration.

Process:

  • Obtain the difference file: Obtain the difference file between the primary and secondary DN by parsing Xlog logs

  • Backup and recovery: Atomized recovery and backup are performed for files that differ from the master and slave files. Errors can be recovered during the process and reentrant can be invoked again after errors are eliminated

  • File transfer: 1 to 16 threads are created on the standby host to pull the differential file from the host

  • The incremental reconstruction is complete and xlog logs fall down

Analysis:

Incremental reconstruction: Based on Xlog logs, the difference files between the primary and secondary DN are calculated and sent to the secondary DN. In this way, the data on the secondary DN is not damaged and the incremental reconstruction is performed quickly with a low cost.

3.2. Full rebuild: gs_ctl build-b full-z datanode

USES:

The full rebuild of the standby machine can repair most scenarios where data and logs are damaged or lost, but the repair time is longer than the incremental build

Process:

  • Obtain the difference file: Obtain the CRC check value of the corresponding file on the main DN using the CRC-32C series algorithm tuned according to the hardware, and perform the corresponding operation locally at the same time, and obtain the difference file list by comparing the two

  • Backup and Restore: No atomization is performed by default, but atomization recovery is attempted, ignoring the success or failure of recovery results

  • File transfer: 1 to 16 threads are created on the standby host to pull the differential file from the host

  • The incremental reconstruction is complete and xlog logs fall down

Analysis:

In full reconstruction, the primary DN file is used as the base, and the secondary DN file is checked against it. If the check of a file block in the secondary DN file is inconsistent, the host sends the file block to the secondary DN. Compared with full cleanup reconstruction, the amount of data copied and WAL logs is less, and the cost is medium.

3.3. Run gs_ctl build-b fullcleanup -z datanode

USES:

The value is different from that in full mode: Before synchronization, clear the data directory on the DN host. Can fix the vast majority of data and log corruption or loss scenarios, but the repair time is longer than other modes

Process:

  • Clearing data files on the standby server: Clear the data directory on the standby server and retain the configuration file

  • Transfer of full mirroring from the host to the standby host: The host sends all its data directories except configuration files to the standby host using a single thread

  • The full reconstruction is complete and xlog logs fall down

Analysis:

In full cleanup and reconstruction mode, the standby host clears the data directory, retains the configuration file, and sends a full reconstruction request to the standby host. Then, the standby host sends all its data directories except the configuration file to the standby host. After the reconstruction, the standby host starts.

4. To summarize

The function of standby DN reconstruction mainly aims at single point of failure repair. According to the implementation, standby DN reconstruction can be divided into full reconstruction, full cleanup reconstruction and incremental reconstruction, all of which interact with the primary DN. When a SINGLE point of failure occurs in the DN, you need to select an appropriate method to rebuild data on the standby node based on the actual damage and resource consumption.

For more information about GuassDB(DWS), please search “GaussDB DWS” on wechat to follow the wechat public account and share with you the latest and most complete PB series warehouse black technology ~

Click to follow, the first time to learn about Huawei cloud fresh technology ~