1. Plan the cluster environment
IP The host name NN1 NN2 DN Resource Manager NodeManager ZK
172. *. *. 6 master Y Y N Y N Y
172. *. *. 7 slave1 N N Y N Y Y
172. *. *. 8 slave2 N N Y N Y Y
172. *. *. 9 slave2 N N Y N Y Y
2. Create users and user groups
adduser hadoop
passwd hadoop
# Add to hadoop group
usermod -a -G hadoop hadoop
# Grant root permission
vi /etc/sudoers
hadoop  ALL=(ALL)       ALL
Copy the code
3. Modify the master (172..6) the host name
vi /etc/sysconfig/network
HOSTNAME=master
Restart or temporarily run the command to take effect
hostname master
Slave1, Slave2, slave3
hostname slave1
hostname slave2
hostname slave3
Copy the code
4. Configure the mapping between IP addresses and host names
vi /etc/hosts
172.*.*.6 master
172.*.*.7 slave1
172.*.*.8 slave2
172.*.*.9 slave3
Copy the code
4. Configure the encryption-free login
Key generation in three clusters
ssh-keygen -t rsa
Copy the public key to master authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Grant authorized_keys 600 permission
chmod 600 authorized_keys
The final authorized_keys file contains the following contents
[root@localhost .ssh]# cat authorized_keysssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtEvxRj/3xPCtnO38Gy4Y/Y4gj6XX5s+G2hwG5xx19PiDQEKeW3BYUDE616OVdecStBo3X+0Plr2ioirI/3WGlUkm0tod r/irpksy0MTpvsjCNUnCWGUHGFMUmrcw1LSiNLhoOSS02AcIq+hw3QJO0w0Wo0EN8xcOhrYwuAByoVv3CvqWd/2Vce2rNOXxLNSmc9tR0Dl3ZqOAq+2a55GM 7cETj+eiexDeF5zEVJ2vykQdH3+sZ2XLrQu4WXOMn70xFosk7E1lwJ14QLy6lpfRcWnB1JVKJx9mglze6v3U35g59Vu/LP7t3ebW+dJIOD3/Attb5HcvN8MN fQVOX3JD4w== root@master ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuU9KJmlmYCx7D+vfMCl2Fj/kz1mfWBrChco0jmZtbygpYY8MUSjmfnsC/wefWKMnFtEruJb+RrgBLxVY6lNzvVKXh+iV Phrjubzj54FoZjepR+1EEznIvwkKa+Y4fkcSJjmcSq/Wvjvz34j3/wVoa1qZtbQing+GzC8Xt0y5rQ6fD1gzD4Oniu43fHAeQDxpo2cVNnTdO2HEe56ZfhIc tVRP63rc2CoEuD7d0Ea2WhV0Uruqri/ZKFHVAQQqQ7z/jdCgzTdTXJ5t5hpyeaK8+mYhUKEyOF3xrACW1Is6grUjhbjUxTLt2y2Ytw1d5voFxCUJ6MQcy91K FE/9Lfefyw== root@slave1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEArucVUJdZBYXJD0r9WiX6VnR5S3F7BhoR7hB8UTkXs+WRJGEX9E44yjH+BjIJAPn2v/XwOCdqzSZrGPzLL/BG+XRhGN5N Gmdplv8xI3C93hC5kZewRHrHlcAG5Kv4mcHlU+ugcWiyQbIaQvLaFXaq48ZVQHYrzXrz3ZT6QDpsaZtSeW4Z4KWeFmL+AwNyAqxK0nxYXR1zNQJ1r0IdApKm P1WNvbcblB2UKx5G7VMxOs62WY0R9LGdJK6Mmmr5QPlWlpn/g5vXlBvgD80pM6iixFAyz8q19aMQjErTWuULNvX8tdcm+StJV52N8EsiuNMOs+xLVO7L00yx ZRtwrXKGgQ== root@slave2# transfer master authorized_keys to slave1/slave2
scp ~/.ssh/authorized_keys root@slave1:~/.ssh/
scp ~/.ssh/authorized_keys root@slave2:~/.ssh/

# Check remote non-secret login
ssh slave1
ssh slave2
ssh slave3
Copy the code
3. Decompress and configure environment variables
Tar -zxvf hadoop-2.1.7.tar. gz vi /etc/profileexportHADOOP_HOME = / opt/middleware/hadoop - 2.7.7export PATH=$PATH:${HADOOP_HOME}/bin
source /etc/profile
Copy the code
4. Hadoop configuration
The master node creates a folderMkdir -p /opt/middleware/hadoop-2.7.7/ DFS /{name,data} mkdir -p /opt/middleware/hadoop-2.7.7/temp# Other configuration
vi slaves
slave1
slave2
slave3
Copy the code
# modified hadoop - env. Sh
export JAVA_HOME=#{JAVA_HOME}
# change to (depending on the actual JDK installation directory)
export JAVA_HOME=/usr/local/jdk
Copy the code
# configuration core - site. XML<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> < value > file: / opt/middleware/hadoop - 2.7.7 / temp value > < / < / property > < property > < name >. Hadoop proxyuser. Hduser. Hosts < / name > <value>*</value> </property> <property> <name>hadoop.proxyuser.hduser.groups</name> <value>*</value> </property> </configuration>Copy the code
# configuration HDFS - site. XML<configuration> <property> <name>dfs.nameservices</name> <value>rsmshadoop</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:9001</value> </property> <property> < name > DFS. The namenode. Name. Dir < / name > < value > file: / opt/middleware/hadoop - 2.7.7 / DFS/name < value > / < / property > < property > < name > DFS. Datanode. Data. Dir < / name > < value > file: / opt/middleware/hadoop - 2.7.7 / DFS/data value > < / < / property > < property > <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value>
    </property>
</configuration>
Copy the code
# configuration mapred - site. XML
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
</configuration>
Copy the code
# configuration yarn - site. XML
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>
</configuration>
Copy the code
5. Run hadoop
# formatThe/opt/middleware/hadoop - 2.7.7 / bin/HDFS namenode - formatCopy the code
Start the cluster
/ opt/middleware/hadoop - 2.7.7 / sbin/start - all. ShCopy the code
[root@localhost sbin]# sh start-dfs.sh
which: no start-dfs.sh in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/ JDK/bin: / opt/middleware/mongo/bin: / opt/middleware/hadoop - 2.7.7 / bin: / root/bin: / usr /local/ JDK/bin: / opt/middleware/mongo/bin: / opt/middleware/hadoop - 2.7.7 / bin) 19/01/17 18:38:06 WARN util. NativeCodeLoader: Unable to load native-hadoop libraryfor your platform... using builtin-java classes where applicable
Starting namenodes on [master]
The authenticity of host 'master (172.*.*.6)' can't be established.
RSA key fingerprint is a0:47:1b:35:a9:f1:e7:0d:81:6d:8b:f4:47:95:f9:96.
Are you sure you want to continue connecting (yes/no)? yes
master: Warning: Permanently added 'master,172.*.*.6' (RSA) to the list of known hosts. master: Starting namenode, logging to /opt/middleware/hadoop-2.7.7/logs/hadoop-root-namenode-master.out slave2: Starting datanode, logging to /opt/middleware/hadoop-2.7.7/logs/hadoop-root-datanode-slave2.out slave1: starting datanode, Logging to /opt/ Middleware /hadoop-2.7.7/logs/hadoop-root-datanode-slave1.out Starting secondary namenodes [master] master: starting secondarynamenode, Logging to /opt/middleware/hadoop-2.7.7/logs/ hadoop-root-Secondarynamenode-master. out 19/01/17 18:38:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableCopy the code
Validation of the cluster
Hadoop jar/opt/middleware/hadoop - 2.7.7 / share/hadoop/graphs/hadoop - graphs - examples - 2.7.7. Jar PI 10 10Copy the code
Monitoring the page

http://172.*.*.6:50070

Problems encountered

1. Some Namenodes cannot be started
/opt/middleware/hadoop-2.7.7/bin/ HDFS Namenode-format Solution Copy the clusterID of NameNode (Master) to the DataNodeOpen NameNode VersionVi/opt/middleware/hadoop - 2.7.7 / DFS/name/current/VERSION... clusterID=CID-45f7aaaf-424a-472c-9cb5-827a9d18906e# Open DataNode VersionVi/opt/middleware/hadoop - 2.7.7 / DFS/data/current/VERSION...Copy the code
2.NameNode cannot be started
There appears to be a gap inThe edit log. We expected txID 1, but got TXID 37309 Cause: The metadata file is damagedCopy the code