Based on the Settings in the previous sections, we have already configured a machine from network to Hadoop. In order to test the Hadoop cluster, this section describes how to create the same machine using the Virtual Box Clone tool.

1. Use the Virtual Box Manager Clone machine

In the VBox installation directory there is a vboxManage. exe program, using this tool can clone the machine, detailed use as follows:

Exe old machine. vdi new machine. vdiCopy the code

To clone one machine, run the above command to clone the other two machines, Slave1 and Slave2. Vboxmanage.exe also has many other useful functions, disk expansion, etc., please check it out if you need it.

2. Configure the slave machine

2.1 Virtual Creation

In the previous step, we just clone the virtualbox vdi files, so we can use them to create two virtual machines. The process is the same as before, except that when creating virtual disks, we select the existing vdi files, which are the virtual machine files we just clone.




Example Load an existing VDI file

Click Create to create a new VIRTUAL machine. Then click Start.

2.2 Machine Configuration

This step uses Slave1 as an example. The name of the Clone machine is still master, which will be changed later

2.2.1 Changing the Static IP Address of a Machine

Use the ifconfig command to check the machine IP, at which point the machine should have an initial IP, for example




Network IP

What we need to change is the IP address of the second network card,

CD /etc/sysconfig/network-scripts/ sudo vim ifcfg-enp0s8 // Modify the configuration file of the second NICCopy the code

Since the machine is from Master Clone, all the configuration on the machine is the same as master, so only a few changes are required

TYPE=Ethernet BOOTPROTO=static DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME=enp0s8 UUID= 3f19CCD2-06A3-4052-b370-37AD9178a6d4 ONBOOT=yes ADDR=192.168.56.108 // Change the current IP address to DNS=255.255.255.0Copy the code

The IP configuration is complete. Restart the network

service network restartCopy the code

If that fails, restart the machine.

2.2.2 Changing the machine name

Example Change the machine name to Slave1

Sudo hostnamectl -sethostName "slave1" // Set the hostname hostNamectl status -- TRANSIENT // Check the temporary hostname hostnamectl status --staticCopy the code
2.2.3 Adding machine IP addresses to the hosts file

The IP address corresponding to the slave machine has been added to the host during the configuration of the master. Therefore, you do not need to add anything. You only need to check whether the current static IP address is the same as that in the hosts file. Test verification,

SSH localhost // After the first login, the system prompts you whether to record the secret key. Enter yes to log in to exit SSH slave1 without passwordCopy the code

Do the same for Slave2 as you did for Slave1. Test whether password-free SSH login can be performed between the three machines. If any problem occurs, check whether SSH secret keys are added to authorized_keys. Hadoop cluster machines are all set up. Go ahead




3. Hadoop cluster test

3.1 Format master Namenode and Start the test

CD /hadoop-2.7.2./bin/ HDFS namenode-formate./sbin/start-all.sh // Start DFS and YARNCopy the code

If the following information is displayed, the startup is successful:




Hadoop cluster start

View the MASTER’s JPS

8034 ResourceManager
8291 Jps
7880 SecondaryNameNodeCopy the code

View the JPS of the slave

2535 NodeManager
2431 DataNode
2655 JpsCopy the code

View the Hadoop process management page

http://master:8088 //master change to your master IPCopy the code

The following page is displayed:




Hadoop process management page

Now that your Hadoop cluster is up, we’ll start installing Spark and Scala. Reprint please indicate the source: www.jianshu.com/p/e8066c14a…