Reprint please indicate the original address is: https://dongkelun.com/2018/03/23/hadoopConf/

preface

Since I need to use Spark now, and Spark will deal with HDFS and Hive, I can’t use the distributed cluster configured in the company’s server after I leave the company. Later, the cluster of three virtual machines built with Ambari is too congested, so I checked the standalone deployment of Hadoop and Hive online. So that I can carry out simple learning, here is a record, originally wanted to put Hadoop and Hive together to write, because there are too many, divided into two write.

1. Install and configure JDK first (I installed 1.8)

2. Download Hadoop

Download address: mirror.bit.edu.cn/apache/hado… , I downloaded hadoop-27.5.tar. gz (since 2.7.1 was downloaded and saved locally several years ago, now I find that the hadoop download address of spark configuration is slow, so I change it to this address).

3, decompress to /opt directory (according to your own habit)

Tar -zxvf hadoop-27.5.tar. gz -c /opt/Copy the code

4. Configure hadoop environment variables

vim /etc/profile
Copy the code
exportHADOOP_HOME = / opt/hadoop - 2.7.5export PATH=$PATH:$HADOOP_HOME/bin  
Copy the code
source /etc/profile
Copy the code

5. Configure Hadoop

5.1 configure hadoop – env. Sh

Vim/opt/hadoop - 2.7.5 / etc/hadoop/hadoop - env. ShCopy the code

Find # The Java Implementation to Use. Change the line below it to:

exportJAVA_HOME = / opt/jdk1.8.0 _45Copy the code

5.2 Configuring Core-site. XML (The file path and port in the configuration files in 5.2 and 5.3 can be configured according to your own habits)

IP address :192.168.44.128 is the VM IP address, which cannot be set to localhost. If localhost is used, saprk will be used to connect the hive server (VM) on Windows, and an exception will be reported. Localhost = win localhost = win localhost = win localhost = win localhost = win localhost

Vim/opt/hadoop - 2.7.5 / etc/hadoop/core - site. XMLCopy the code
<configuration>
<property>
        <name>hadoop.tmp.dir</name>
        <value>file:///opt/hadoop-2.7.5</value>
        <description>Abase forother temporary directories.</description> </property> <property> <name>fs.defaultFS</name> < value > HDFS: / / 192.168.44.128:8888 < value > / < / property > < / configuration >Copy the code

HDFS – site. 5.3 configuration XML

Vim/opt/hadoop - 2.7.5 / etc/hadoop/HDFS - site. The XMLCopy the code
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> < name > DFS. The namenode. Name. Dir < / name > < value > file:///opt/hadoop-2.7.5/tmp/dfs/name < value > / < / property > < property > < name > DFS. Datanode. Data. Dir < / name > < value > file:///opt/hadoop-2.7.5/tmp/dfs/data < value > / < / property > < / configuration >Copy the code

6. SSH password-free login

For details, see SSH Password-free Login in Linux

7. Start and stop

Format HDFS for the first time:

cd/ opt/hadoop - 2.7.5. / bin/HDFS namenode - formatCopy the code

Re-format filesystem in Storage Directory /opt/hadoop-2.7.5/ TMP/DFS /name? (Y or N) Input: Y (input Y or N, all input Y) start:

./sbin/start-dfs.sh
Copy the code

Stop:

./sbin/stop-dfs.sh
Copy the code

Authentication, the browser input: http://192.168.44.128:50070

Simply verify the Hadoop command:

hadoop fs -mkdir /test
Copy the code

If the following figure is displayed in the browser, it indicates that the browser is successful

8. Configure YARN

8.1 configuration mapred – site. XML

cd/opt/hadoop-2.7.5 /etc/hadoop-/ cp mapred-site.xml.template mapred-site. XML vim mapred-site.xmlCopy the code
<configuration> <! Mapreduce.framework. name</name> <value> YARN </value> </property> </configuration>Copy the code

8.2 configuration yarn – site. XML

vim yarn-site.xml
Copy the code
<configuration> <! Mapreduce_shuffle --> <property> <name> yarn.nodeManager. aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>Copy the code

8.3 Starting and Stopping YARN

Activation:

cd/ opt/hadoop - 2.7.5. / sbin/start - yarn. ShCopy the code
./sbin/stop-yarn.sh 
Copy the code

A browser: http://192.168.44.128:8088

The resources

Blog.csdn.net/cafebar123/…