1. Storage mechanism

1. Basic description

NameNode runtime metadata needs to be stored in memory, and fsImage of metadata should be backed up in disk. When metadata is updated or added, modification of metadata in memory will append operation records to edits log files, excluding query operations. If the NameNode fails, the metadata can be reloaded into memory by merging FsImage and Edits. SecondaryNameNode is specifically used for merging FsImage and Edits.

2. Workflow

The NameNode mechanism

  • After NameNode formatting is started, Fsimage and Edits files are created for the first time.
  • Directly load FsImage image files and Edits logs to memory without first boot.
  • Adding, deleting and modifying metadata performed by the client will be recorded in the Edits file.
  • The requested operations then modify the metadata in memory;

SecondaryNameNode mechanism

  • When asked if NameNode wants to CheckPoint, NameNode returns a message.
  • Perform CheckPoint if the SecondaryNameNode request is required.
  • NameNode cuts existing log files, new records scroll to new Edits files;
  • [Fixed] Scroll before editing log and mirror file copy to SecondaryNameNode;
  • SecondaryNameNode load Edits log and FsImage image file to memory merge;
  • Create a new image file fsimage.chkpoint and copy it to NameNode.
  • NameNode renames fsimage. Chkpoint to fsimage;

3. CheckPoint Settings

You can set some SecondaryNameNode mechanisms by modifying the configuration of hdFS-default. XML, for example, executing the SecondaryNameNode every hour.

<property>
  <name>dfs.namenode.checkpoint.period</name>
  <value>3600</value>
</property>
<property>
  <name>dfs.namenode.checkpoint.txns</name>
  <value>1000000</value>
<description>The number of records in the file reaches 1000000</description>
</property>
<property>
  <name>dfs.namenode.checkpoint.check.period</name>
  <value>60</value>
<description>Number of file records checked once a minute</description>
</property >
Copy the code

Ii. Document information

1. FsImage file

NameNode memory metadata serialized backup information;

Generation path: Based on NameNode nodes

CD/opt/hadoop2.7 / data/TMP/DFS/name/current /Copy the code

Check the file

HDFS oiv -p Convert file type -i image file -o Output file path after conversionCopy the code

Based on the syntax format, operate on the file shown above:

Fsimage_0000000000000000019 -o /data/fsimage.xmlCopy the code

So you can see some metadata information.

2. Edits files

The path for adding, deleting, or modifying HDFS files is recorded in the Edits file.

The basic grammar

HDFS oev -p Converted file type -i log file -o Output file path after conversionCopy the code

Check the file

-o /data/edits. XML # run cat /data/edits.xmlCopy the code

Three, fault recovery

1. Copy SecondaryNameNode data

End the NameNode process first.

Delete data stored by NameNode.

/ root @ hop01 / rm - rf/opt/hadoop2.7 / data/TMP/DFS/name / *Copy the code

Copy SecondaryNameNode data to NameNode data storage directory.

# note SecondaryNameNode service configuration on hop03 / root @ hop01 / SCP -r root @ hop03: / opt/hadoop2.7 / data/TMP/DFS/namesecondary / * / opt/hadoop2.7 / data/TMP/DFS/name /Copy the code

Restart the NameNode process.

2. Checkpoint

Modify the hdFS-site. XML configuration, synchronize the configuration to cluster-related services, and restart the HDFS process.

<property>
  <name>dfs.namenode.checkpoint.period</name>
  <value>120</value>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>/ opt/hadoop2.7 / data/TMP/DFS/name</value>
</property>
Copy the code

End the NameNode process.

Delete data stored by NameNode.

/ root @ hop01 / rm - rf/opt/hadoop2.7 / data/TMP/DFS/name / *Copy the code

SecondaryNameNode(hop03) is on the same host node as NameNode(Hop01). Copy the directory where SecondaryNameNode stores data to the directory where NameNode stores data. Delete the in_use.lock file.

/ root @ hop01 / SCP -r root @ hop03: / opt/hadoop2.7 / data/TMP/DFS/namesecondary/opt/hadoop2.7 / data/TMP/DFS / [root @ hop01 namesecondary/] rm -rf in_use.lock [root@hop01 dfs]$ ls data name namesecondaryCopy the code

Import checkpoint data

[root@hop01 hadoop2.7] bin/ HDFS namenode -importCheckpointCopy the code

Restart NameNode

[root@hop01 hadoop2.7] sbin/hadoop-daemon.sh start namenode
Copy the code

Configure multiple directories

NameNode can be configured with multiple local directories. Each directory stores the same content to improve operation reliability.

1. Add the configuration

# vim/opt/hadoop2.7 / etc/hadoop/HDFS - site. XML # add content is as follows<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///${hadoop.tmp.dir}/dfs/name01,file:///${hadoop.tmp.dir}/dfs/name02</value>
</property>
Copy the code

This configuration needs to synchronize all services in the cluster.

2. Delete the original data

This operation is required for all services in the cluster.

[root@hop01 hadoop2.7]# rm -rf data/ logs/
Copy the code

After formatting the NameNode, restart the cluster service.

5. Safety mode

1. Basic description

When NameNode is first started, it loads an image of file system metadata in memory based on image files and edit logs, and then listens to DataNode requests. During this process, the client is in read-only safe mode, and cannot upload files. In this secure mode, DataNode sends the latest data block list information to NameNode. If the minimum copy condition is met, NameNode exits the secure mode after the specified time.

2. Safe mode

  • Safe mode state
/ opt/hadoop2.7 / bin/HDFS dfsadmin - safemode to getCopy the code
  • Enter safe mode
/ opt/hadoop2.7 / bin/HDFS dfsadmin - safemode enterCopy the code
  • Exit safe Mode
/ opt/hadoop2.7 / bin/HDFS dfsadmin - safemode to leaveCopy the code
  • Wait safe mode
/ opt/hadoop2.7 / bin/HDFS dfsadmin - safemode to waitCopy the code