1. Working mechanism of DataNode



1. A data block is stored on the disk as a file on the DataNode, consisting of two files: the data itself and metadata, including the length and number of the data block

Checksum and timestamp of block data

2. After DataNode starts, it registers with Namenode and periodically reports all block information to Namenode

3. The heartbeat is every 3 seconds. The heartbeat returns the result with the command given by Namenode to Datanode (copy block data to another machine, or delete a data block).

If no heartbeat message is received from a DataNode within 10 minutes, the node is considered unavailable

4. Some machines can be added and exited safely during cluster operation

2. Data integrity



Method used by DataNode to ensure data integrity.

1) When DataNode reads blocks, it calculates the CheckSum.

2) If the calculated CheckSum is different from the value when the Block was created, the Block is damaged.

3) The Client reads blocks on other Datanodes.

4) DataNode periodically verifies CheckSum after its file is created

3. Set the offline time limit

It is important to note HDFS - site. The heartbeat of the XML configuration files. Recheck. The interval of milliseconds, DFS. Heartbeat. The interval unit for seconds. <property> <name>dfs.namenode.heartbeat.recheck-interval</name> <value>300000</value> </property> <property> <name>dfs.heartbeat.interval</name> <value>3</value> </property>Copy the code

4. Service new data nodes

1. Environment preparation

(1) Clone another Hadoop105 host on Hadoop104. (2) Change the IP address and host name. (3) Delete the files (/opt/module/hadoop-2.7.2/data and log) stored in the original HDFS file system [atguigu @ hadoop105 hadoop - 2.7.2] $source/etc/profileCopy the code
  • 1
  • 2
  • 3
  • 4
  • 5

2. Specific steps for commissioning the new node

(1) DataNode is directly started by a single node. [atguigu@hadoop109 hadoop-2.7.2]$sbin/hadoop-daemon.sh start datanode [atguigu@hadoop109 hado-2.7.2]$ Sbin /yarn-daemon.sh start nodeManager (2) Upload files to hadoop109 [atguigu@hadoop109 hadoop-2.7.2]$hadoop fs -put /opt/module/hadoop-2.7.2/ license. TXT/(3) If data is unbalanced, [atguigu@hadoop105 sbin]$./start-balancer.sh Starting balancer, Logging to /opt/module/hadoop-2.7.2/logs/hadoop-atguigu-balancer-hadoop105. Out Time Stamp Iteration# Bytes Already Moved  Bytes Left To Move Bytes Being MovedCopy the code

5. Decommission old data nodes

1. Add a whitelist

Hosts added to the whitelist are allowed to access NameNode. Hosts not on the whitelist are removed. To configure a whitelist, perform the following steps: (1) Create the dfs.hosts file [atguigu@hadoop105 hadoop]$PWD in the /opt/module/hadoop-2.7.2/etc/hadoop directory of NameNode /opt/module/hadoop-2.7.2/etc/hadoop [atguigu@hadoop105 hadoop]$touch dfs.hosts [atguigu@hadoop105 hadoop]$vi dfs.hosts Add the following host names (hadoop109 is not added) : hadoop105 hadoop106 hadoop107 Hadoop108 (2) Add the dfs.hosts property <property> to the HDFS -site. XML configuration file of NameNode < name > DFS. Hosts < / name > < value > / opt/module/hadoop - 2.7.2 / etc/hadoop/DFS hosts < value > / < / property > (3) the configuration file distribution [atguigu@hadoop105 hadoop]$xsync hdfs-site. XML (4) Refresh NameNode [atguigu@hadoop105 hadoop-2.7.2]$HDFS dfsadmin -refreshNodes Refresh Nodes successful (5) Update the ResourceManager node [atguigu@hadoop105 hadoop-2.7.2]$YARN rmadmin -refreshNodes 17/06/24 14:17:11 INFO client.RMProxy: Connecting to the ResourceManager at hadoop103/192.168.1.103:8033 (6) on the web browser to view (7) if the data is not balanced, [atguigu@hadoop102 sbin]$./start-balancer.sh Starting balancer, Logging to /opt/module/hadoop-2.7.2/logs/hadoop-atguigu-balancer-hadoop102. Out Time Stamp Iteration# Bytes Already Moved  Bytes Left To Move Bytes Being MovedCopy the code

2. The blacklist is retired

Hosts on the blacklist will be forced to exit. 1. Create dfs.hosts.exclude file [atguigu@hadoop105 hadoop]$PWD in the /opt/module/hadoop-2.7.2/etc/hadoop directory of NameNode /opt/module/hadoop-2.7.2/etc/hadoop [atguigu@hadoop105 hadoop]$touch dfs.hosts.exclude [atguigu@hadoop105 hadoop]$vi Dfs.hosts. exclude Add the following host name (the node to be decommissioned) hadoop109 2. Exclude <property> <name>dfs.hosts.exclude</name> < value > / opt/module/hadoop - 2.7.2 / etc/hadoop/DFS hosts. Exclude < value > / < / property > 3. Configuration file distribution [atguigu@hadoop105 hadoop]$xsync hdfs-site.xml 3. Refresh NameNode and ResourceManager [atguigu@hadoop105 hadoop-2.7.2]$HDFS dfsadmin -refreshNodes Refresh Nodes successful [atguigu@hadoop105 hadoop-2.7.2]$yarn rmadmin-refreshnodes 17/06/24 14:55:56 INFO client.rmProxy: Connecting to the ResourceManager at hadoop103/192.168.1.103:8033 4. Check the Web browser. The decommission node is in the decommission in Progress state. The decommission node is copying blocks to other nodes. Stop the decommissioned node and the node explorer until the decommissioned node is decommissioned. Note: If the number of copies is 3 and the number of nodes in service is less than or equal to 3, it cannot be successfully retired. [atguigu@hadoop109 hadoop-2.7.2]$sbin/hadoop-daemon.sh stop datanode stopping datanode [atguigu@hadoop109 Hadoop-2.7.2]$sbin/yarn-daemon.sh stop nodeManager stopping NodeManager If the data is uneven, [atguigu@hadoop105 hadoop-2.7.2]$sbin/start-balancer. Sh starting balancer, Logging to /opt/module/hadoop-2.7.2/logs/hadoop-atguigu-balancer-hadoop102. Out Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved Note: The host name in the whitelist and blacklist cannot be the same at the same time.Copy the code

6. Configure multiple directories for Datanode

1. DataNode can also be configured as multiple directories. Each directory stores different data. That is: data is not copy 2. XML <property> <name>dfs.datanode.data.dir</name> <value>file:///${hadoop.tmp.dir}/dfs/data1,file:///${hadoop.tmp.dir}/dfs/data2</value> </property>Copy the code