Hadoop pseudo-distributed configuration process!!

This is the 18th day of my participation in the August Genwen Challenge.More challenges in August

The preparatory work

1, prepare virtual machine, I prepared centOS 6.5 (may be a little old)!!

2. Required software Oracle VM VirtualBox, FZ, Xshell

**3, environment, I got JDK 1.7 / Hadoop 2.6.5 **

Configure SSH encryption exemption

Example Modify VM network configurations

View the Linux IP address

Changing an IP Address

/etc/sysconfig/network-scripts/ifcfg-eth0 sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0

Valid IP address Restart the VM

2. Change the root permission on the /etc/hosts file. Run **vi/etc/hosts ** to add the IP address and host name entered in # to the end

Configure SSH password-free login

1. Run the service SSHH start command to start the SSH service

Check whether SSH has been installed successfully!!

~~If the preceding information is displayed, SSH is started normally~~

** Generates an SSH public key **

Command: ssh-keygen -t rsa

Copy the key to the native authorized_keys list

Disabling the Firewall

temporarily closedPermanent ban Xshell connection test

No password required to indicate success!!

JDK + Hadoop environment installation

The required information

My older version, you can download the new one on the official website!!

Upload tools

~ ~

I can do either. The software doesn’t work and I can’t save it.

~ ~

Set the Linux directory structure

Software directory: store software installation packages App directory: All software installation directories Data directory: store data lib directory: store JAR packages for tasks Shell directory: Store shell scripts for tasks Maven-resp directory: store required dependency packages

Copy resources to the software directory

Use upload tool!!

To view

Configure the JDK

Check the current system JDK, if there is openJDK need to delete!!

See the command: RPM – qa | grep JDKRun the RPM -e –nodeps…… command to delete the node

Unzip the uploaded files to the APP directory

Run the tar -zxvf jdK-8u201-liunx-x64.tar. gz -c ~/app/ command

Configuring environment Variables

sudo vi /etc/porfile

Note: Press capital G at the bottom to go to the bottom ！！！！

Effective Configuration File

Command: source /etc/profile

Check the installation is successful!!

Command: Java -version

Configure Hadoop

Unpack the

Run the tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -c ~/app/ command

Configure hadoop – env. Sh

Position in CD/etc/hadoop

Add environment variables at the end, Java and Hadoop

Configure the core – site. The XML

Modify the HDFS – site. XML

Modify slaves file

Modify the configuration file to append HADOOP_HOME

Command :sudo vi /etc/porfile

File to take effect

source /etc/profile

Format the hadoop

1. Go to the bin directory and run the hadoop namenode-format command

Non-repeatable execution

Appear this for success!!

Start the hadoop

Go to the sbin directory and run start-dfs.sh Check the hadoop If the above four processes appear, they are successful!!

You can enter it in the vm browserhttp://master:50070 Alternatively, in Windows, enter HTTP: VM IP address: 50070

Configuration of Yarn

Modify the yarn – site. XML

Modify the mapred – site. XML

Start the YARN view process

Mysql > install Mysql for Hive

Hive uses the mysql database. Therefore, install the mysql database before installing Hive. When installing the mysql database, you need to enable the network to connect to the Internet

1. Install the mysql service

** Starts && stops the service

** **1, add default-character-set=utf8 vim /etc/my.cnf ** to mysql configuration file /etc/my.cnf **Chkconfig mysqld on chkconfig — mysqld **

**3, mysql service mysqld start **

Mysqladmin -u root password 123456 mysql -u root -p

Sudo service mysqld stop mysqLD_safe –user=root –skip-grant-tables mysql -u root use mysql update user set password=pasword(“new_pass”) where user=”root”; flush privileges;

ERROR 1045 (28000): Access denied for user ‘root’@’localhost’ (using password: Mysql server run mysqld stop mysqld server run mysqld_safe Mysqld_safe –user=mysql –skip-grant-tables –skip-networking & : mysqLD_safe –user=mysql –skip-grant-tables –skip-networking Skip permission control by not starting grant-tables. — Skip-networking: skip TCP/IP and access only on the local computer. (This option is not required, according to some sources on the web. After executing the command above, the session window will appear unresponsive state, you need to use CTRL+C to interrupt the session

Mysql -u root mysql mysql> update user set password= password (‘12345′) -> where user=’root’ and host=’root’ or host=’localhost’; flush privileges

Run the following command to start the mysql service :service mysqld start Note: If an error occurs, restart the mysql service sudo service mysqld restart

Install the hive

You can install Hive after installing mysql

Download the Hive package, available at ！！！！

Decompress the Hive compressed file

The tar ZXVF ~ / app/hive – 1.10 – cdh5.7.0. Tar. Gz

Example Modify the hive-site. XML file

Use hadoop01 to create hive-site. XML cp hive-site.xml.tmplate hive-site. XML in the conf directory of the hive decompressed file

Add the following configuration file


      

      
<configuration>
    <property>
    	<name>javax.jdo.option.ConnectionURL</name>
      	<value>jdbc:mysql://master:3306/hive? createDatabaseIfNotExist=true</value>
    </property>
<property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>Native user name</value>
    </property>
<property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>This machine is the password</value>
    </property>
</configuration>
Copy the code

Example Modify the hive-env.sh file Add mysql JDBC driver JAR package!Add the mysql driver package to the lib folder in the hive decompressed directory. Note that you can use the upload tool.

Copy the code

Configure environment variables!!

Optimize hive environment variables at the end of /etc/profile

Verify installation is successful!!

Run hive to create a table named test and query the number of records in the table create table test(id int); select count(*) from test;

Observe the result, appear below for success!!

Configuration zookeeper

downloadarchive-primary.cloudera.com/cdh5/cdh/5/

Unpack the

Copy the code

Gz Copy the zoo_sample. CFG configuration file of ZooKeeper. Go to the conf directory of ZooKeeper and find the zoo_sample. CFG file. Backup the zoo_sample. CFG file and rename it zoo.cfg

View/modify the zoo. CFG file Pay attention to

The default dataDir is/TMP /zookeeper. Since/TMP is a temporary directory of Ubuntu, data stored in this directory cannot be stored for a long time. Therefore, you need to specify another directory.Start the ZookeeperGo to the bin directory and run sudo./ zkserver. sh start

Use the status parameter to view the ZooKeeper status Zookeeper can be used in HBase in either of two ways: 1. Zookeeper is used for management; 2. An external Zookeeper cluster is used for management.

Hease installation

Download the hbase archive-primary.cloudera.com/cdh5/cdh/5/ Upload decompression

The tar – ZXVF hbase – 1.2.0 – cdh5.7.0. Tar. GzConfiguring environment Variables sudo vi /etc/profile Enable environment variables source /etc/profile Modify the hbase – env. Sh

sudo vi hbase-env.shModify the hbase – env. Sh file, add the following configuration: export JAVA_HOME = / home/hadoop01 / app/jdk1.7.0 _79Export HBASE_CLASSPATH = / home/hadoop01 / app/hadoop – server – cdh5.7.0 / etc/hadoop export HBASE_MANAGES_ZK=false

The default value of HBASE_MANAGES_ZK in hbase-env.sh is true, which indicates that hbase uses its own Zookeeper instance. True if you use the hbase native ZK, false if you use the external ZK

😂 Modify the RegionServers file In fully distributed mode, regionServers is: slave1 slave2… . In pseudo-distributed mode, regionServers is: masterWe try pseudo distributed so use the following ah!!

Modify the hbase – site. XML

sudo vi hbase-site.xml
Copy the code

Hbase. rootDir HDFS ://master:8020/hbase This parameter specifies the location of the HRegion server. The data location hbase. Cluster. Distributed true hbase. Master master: 60000 hbase. Zookeeper. Quorum master

Copy zookeeper conf/zoo. CFG to hbase conf/ Give script execution authorityChmod + x ~ / app/hbase – 1.2.0 – cdh5.7.0 / bin/start – hbase. ShStart the HbaseRun the./start-hbase.sh command in the bin directory

validation The HBase cli is displayed

performHbase (main) : 001-0 > create “testtable”, “colfam1”If no error is reported, the installation succeeds

conclusion

1. Environment and software requirements
2. Configure the SSH password-free service
3, JDK + Hadoop environment installation
Mysql + Hive installation
5. Install ZooKeeper and hbase