This is the 7th day of my participation in the Gwen Challenge in November. Check out the details: The last Gwen Challenge in 2021.”

Distributed mode

  • Core-site.xml doesn’t need to be changed

  • hdfs-site.xml

<! Replication </name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/var/bigdata/hadoop/full/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/var/bigdata/hadoop/full/dfs/data</value> </property> <! - modify the secondary work in hadoop2 node - > < property > < name > DFS. The namenode. Secondary. HTTP - address < / name > < value > hadoop2:50090 < / value >  </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>/var/bigdata/hadoop/full/dfs/secondary</value> </property>Copy the code
  • Set the dataNode working node

hadoop2
hadoop3
Copy the code
  • Distribute the configuration to other nodes
			cd /opt
			scp -r ./bigdata/  hadoop2:`pwd`
			scp -r ./bigdata/  hadoop3:`pwd`
Copy the code
  • Initialization & startup
hdfs namenode -format
start-dfs.sh
Copy the code

High availability mode

node NN SNN DN ZKFC ZK
hadoop1 * *
hadoop2 * * * * *
hadoop3 * *
hadoop4 * *

Zookeeper needs to be configured on Hadoop2, hadoop3, and hadoop4

  • core-site.xml
<property> <name>fs.defaultFS</name> <value>**hdfs://cluster**</value> </property> <! - a zookeeper cluster configuration address - > < property > < name > ha. The zookeeper. Quorum < / name > < value > hadoop2:2181, hadoop3:2181, hadoop4:2181 value > < / </property>Copy the code
  • hdfs-site.xml
<configuration> <property> **<! - replicas - > * * < name > DFS. Replication < / name > < value > 2 < / value > < / property > < property > * * <! - the namenode data storage directory - > * * < name > DFS. The namenode. Name. Dir < / name > < value > / var/bigdata/hadoop/ha/DFS/name < value > / < / property > <property> **<! - the datanode data storage directory - > * * < name > DFS. Datanode. Data. Dir < / name > < value > / var/bigdata/hadoop/ha/DFS/data value > < / < / property > <property> <name>dfs.nameservices</name> **<! - cluster for custom - > * * < value > cluster value > < / < / property > < property > * * <! --cluster: custom -->** <name>dfs.ha.namenodes. Cluster </name> <value>nn1,nn2</value> <property> <property> **<! Nn1 </name> <value>hadoop1:8020</value> </property> <property> * * <! Nn2 </name> <value>hadoop2:8020</value> </property> <property> * * <! Nn1 </name> <value>hadoop1:50070</value> </property> <property> **<! --cluster: custom -->** <name>dfs.namenode.http-address.cluster.nn2</name> <value>hadoop2:50070</value> </property> - qjournal work node - > * * < property > < name > DFS. The namenode. Shared. The edits. Dir < / name > < value > qjournal: / / hadoop1:8485; hadoop2:8485; hadoop3:8485/cluster</value> </property> ** <! - journal data storage directory - > * * < property > < name >. DFS journalnode. Edits. Dir < / name > < value > / var/bigdata/hadoop/ha/DFS/Jacqueline Nottingham value > < / </property> ** <! -- Java class used by HDFS client to communicate with active namenode -->** <property> **<! - cluster for custom - > * * < name >. DFS client. Failover. Proxy. The provider. The cluster < / name > <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> **<! -- Remote call mode: **< property> <name>dfs.ha.fencing. Methods </name> <value>sshfence</value> </property> **<! - SSH password secret key free configuration - > * * < property > < name > DFS. Ha. Fencing. SSH. The private key - files < / name > < value > / root /. SSH/id_dsa value > < / </property> **<! -- enable automatic failover -->** <property> <name>dfs.ha.automatic-failover. Enabled </name> <value>true</value> </property> </configuration>Copy the code