Hadoop HA

What is the HA

HA is a High availability cluster, which is an effective solution to ensure service continuity. Generally, HA consists of two or more nodes, including active nodes and standby nodes. The one that is performing business is usually called the active node, and the one that is a backup of the active node is called the standby node. When a problem occurs on the active node that causes the ongoing service (task) to fail, the standby node detects it and immediately takes over from the active node to execute the service. In this way, services are not interrupted or temporarily interrupted.

Hadoop HA mechanism

Hadoop2.0 HA mechanism has two namenode, one is active namenode, the state is active; The other one is standby namenode, and the state is standby. The two states can be switched. However, both states cannot be active at the same time. Only one state can be active at most. Only active Namenode provides external services. Standby Namenode does not provide external services. Data is synchronized between active Namenode and standby Namenode using NFS or JN (JournalNode, QJM mode). The active Namenode writes the record of the most recent operation to a local edits file and transfers it to NFS or JN. The standby Namenode periodically checks to read the latest Edit file from NFS or JN and merges the edits and fsimage files into a new fsimage. When the merge is complete, the active Namenode is notified to acquire the new fsimage. After acquiring the new fsimage file, active Namenode replaces the old fsimage file. In this way, data between active Namenode and standby Namenode is synchronized in real time. Standby Namenode can be switched to active Namenode at any time (for example, active Namenode is down). There is also a secondarynamenode, checkpointNode, buckCupNode function of hadoop1.0: merge edits and fsimage files, so that the fsimage file is always updated. Secondarynamenode, checkpointnode, and buckcupnode are no longer required when hadoop2.0 is enabled.

Set up a Hadoop HA cluster

The environment

Linux: centos-7.5_x64 hadoop: hadoop-3.2.0 Zookeeper: zookeeper-3.4.10

The machine planning

Lead to

The zooKeeper and Java environment must be installed on node-2, Node-3, and Node-4

The cluster structures,

# download$wget HTTP: / / http://mirrors.shu.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.2.0.tar.gz# decompression$tar - ZXVF hadoop - 3.2.0. Tar. GzConfigure system environment variables
$ vim /etc/profile

export JAVA_HOME=/usr/local/ jdk1.8.0 _191export PATH=$PATH:$JAVA_HOME/bin

export HADOOP_HOME=/export/ servers/hadoop - 3.2.0export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin


Go to the unzip directory and configure the environment variables
$ cd $HADOOP_HOME

# configure hadoop-env.sh add the following configuration:
$ vim etc/hadoop/core-site.xml

export JAVA_HOME=/usr/local/ jdk1.8.0 _191export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root



# configuration core - site. XML$ vim etc/hadoop/core-site.xml <? xml version="1.0" encoding="UTF-8"? > <? xml-stylesheettype="text/xsl" href="configuration.xsl"? > <! Licensed under the Apache License, Version 2.0 (the"License");
  you may not use this file except inThe compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License forthe specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <! -- Put site-specific property overridesinthis file. --> <configuration> <! <property> <name>fs.defaultFS</name> <value> HDFS ://ns</value> </property> <! -- HA configuration, Quorum </name> <value>node-2:2181, Node-3:2181,node-4:2181</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/export/data/hadoop/temp</value>
    </property>

    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
    </property>

    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>

</configuration>



# configuration HDFS - site. XML$ vim etc/hadoop/hdfs-site.xml <? xml version="1.0" encoding="UTF-8"? > <? xml-stylesheettype="text/xsl" href="configuration.xsl"? > <! Licensed under the Apache License, Version 2.0 (the"License");
  you may not use this file except inThe compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License forthe specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <! -- Put site-specific property overridesinthis file. --> <configuration> <! -- Set HDFS nameservice to NS. --> <property> <name>dfs.nameservices</name> <value>ns</value> </property> <! Nn1 --> <property> <name>dfs.ha.namenodes. Ns </name> <value>nn1,nn2</value> </property> Nn1 </name> <value>node-1:9000</value> </property> <! Nn1 </name> <value> node-1:5070 </value> </property> <! Nn2 </name> <value>node-2:9000</value> </property> <! Nn2 </name> <value> node-2:5070 </value> </property> <! - specifies the NameNode edits metadata location on JournalNode - > < property > < name > DFS. The NameNode. Shared. The edits. Dir < / name > <value>qjournal://node-3:8485; node-4:8485/ns</value> </property> <! - specifies JournalNode in local disk to store the location of the data - > < property > < name >. DFS JournalNode. Edits. Dir < / name > < value > /export/data/hadoop/journaldata</value> </property> <! -- <property> <name>dfs.ha.automatic-failover. Enabled </name> <value>true</value> </property> <! - the realization of automatic switch configuration failure mode - > < property > < name > DFS. Client. Failover. Proxy. Provider. Bi < / name > <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <! --> <property> <name>dfs.ha.fencing. Methods </name> <value> sshfence shell(/bin/true) </value> </property> <! - use sshfence isolation mechanism require SSH login free -- > < property > < name > DFS. Ha. Fencing. SSH. Private - key - files < / name > <value>/root/.ssh/id_rsa</value> </property> <! - configuration sshfence isolation mechanism timeout - > < property > < name > DFS. Ha. Fencing. SSH. Connect - timeout < / name > < value > 30000 < value > / < / property > <property> <name>dfs.ha.namenodes.jn</name> <value>node-3,node-4</value> </property> </configuration># configuration mapred - site. XML$ vim etc/hadoop/mapred-site.xml <? xml version="1.0"? > <? xml-stylesheettype="text/xsl" href="configuration.xsl"? > <! Licensed under the Apache License, Version 2.0 (the"License");
  you may not use this file except inThe compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License forthe specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <! -- Put site-specific property overridesin this file. -->

<configuration>

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

</configuration>


# configuration yarn - site. XML$ vim etc/hadoop/yarn-site.xml <? xml version="1.0"? > <! Licensed under the Apache License, Version 2.0 (the"License");
  you may not use this file except inThe compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License forthe specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <! -- Site specific YARN configuration properties --> <! - open the RM high availability - > < property > < name > yarn. The resourcemanager. Ha. Enabled < / name > < value >true</value> </property> <! - specifies the RM cluster id - > < property > < name > yarn. The resourcemanager. Cluster - id < / name > < value > yarn - ha < value > / < / property > <! - specify the name of the RM - > < property > < name > yarn. The resourcemanager. Ha. The RM - ids < / name > < value > rm1, rm2 < value > / < / property > <! - specify the address of the RM - > < property > < name > yarn. The resourcemanager. The hostname. Rm1 < / name > < value > node - 1 < value > / < / property > < property > <name>yarn.resourcemanager.hostname.rm2</name> <value>node-2</value> </property> <! Zk-address </name> -- <property> <name>yarn.resourcemanager.zk-address</name> <value>node-2:2181,node-3:2181,node-4:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>Configure the workers node
$ vim $HADOOP_HOME/etc/hadoop/workers

node-3
node-4

# copy Hadoop to other nodes (node-2, Node-3, Node-4)$SCP -r hadoop-3.2.0 root@node-2:/ XXX/XXX /Copy the code

HDFS HA configuration

Start the ZooKeeper cluster
$ $ZOOKEEPER_HOME/bin/zkServer.sh start

# Check the ZooKeeper status
$ $ZOOKEEPER_HOME/bin/zkServer.sh status

Run the following commands on node-3 and Node-4 respectively
$ hdfs --daemon start journalnode

Format the ZooKeeper cluster
$ hdfs zkfc -formatZK

# format cluster NameNode (execute on Node-1)
$ hdfs namenode -format

# start the newly formatted NameNode (on Node-1)
$ hdfs --daemon start namenode

Synchronize NameNode1 metadata to NameNode2 (node-2)
$ hdfs namenode -bootstrapStandby

# start NameNode2 (execute on Node-2)
$ hdfs --daemon start namenode

# Start all datanodes in the cluster (run on Node-1)
$ sbin/start-dfs.sh

# Start the ZKFC process (run the following command on the hosts of Node-1 and Node-2 respectively)
$ hdfs --daemon start zkfc

# Verify ha(stop namenode process on Node-1)
$ hafs --daemon stop namenode


Copy the code

The resourceManager HA configuration

# Start YARN at RM1 (execute on Node-1)
$ yarn --daemon start resourcemanager

# Start YARN at RM2 (execute on Node-2)
$ yarn --daemon start resourcemanager

Obtain the resourceManager status on any node (active)
$ yarn rmadmin -getServiceState rm1

Obtain resourceManager status on any node (Standby)
$ yarn rmadmin -getServiceState rm2

Verify YARN HA (perform this operation on Node Node-1.) The Standby Resourcemanager switches to active
$ yarn --daemon stop resourcemanager

Obtain the resourceManager status on any node (active)
$ yarn rmadmin -getServiceState rm2

Copy the code

conclusion

Hadoop HA setup process encountered a lot of various problems the above steps are verified if you encounter problems during the installation can leave a message, thank you!