1. Reference Documents:

    1. Ambari2.7.3 installing HDP3.1.0+CentOS7 offline
    2. Ambari Documentation Version 2.7.3.0
    3. Official document PDF download: bk_ambari-installation.pdf

  2. Installation package download address:

    1. MySQL Product Archives
    2. HDP 3.1.0 Repositories
    3. Ambari Repositories
    4. RPM resource yum

Installation instructions

  1. Software version

    software version instructions
    ambari 2.7.3.0 HDP management software
    HDP 3.1.0.0
    JDK 1.8.0 comes with _181 The version must be greater than 1.8
    mysql 5.7.30 The version must be greater than 5.7

    The software compatibility table is available on the HPD official website. Different HDP versions have compatibility requirements for other software:

  2. Cluster planning: Currently, five nodes are planned and HA is set up for cluster installation. The plan is as follows:

    component node1 node2 node3 node4 node5
    ambari-server Square root
    ambari-agent Square root Square root Square root Square root Square root
    zookeeper Square root Square root Square root
    namenode Square root Square root
    datanode Square root Square root Square root Square root
    resourcemanager Square root Square root
    nodemanager Square root Square root Square root Square root
    hiveserver Square root
    hiveserver2 Square root Square root
    hiveserver2 interactive Square root
    livy Square root
    HMaster Square root Square root
    HRegionServer Square root Square root Square root Square root
    kafka Square root Square root Square root
    Client (Hive, Spark, ZooKeeper…) Square root Square root Square root Square root Square root
    mysql Square root

Environmental audit

  1. Close the selinux:

    #Check the selinux statusGetenforce Disabled #enforceing means enabled, permissive means warning, and Disabled means Disabled
#If so, skip the next step. If not, modify /etc/sysconfig/selinux
vim /etc/sysconfig/selinux
#Change the following content to Disabled
SELINUX=disabled

#Restarting the Server
reboot
Copy the code

  2. To turn off the firewall, each node needs to:

    #Checking the Firewall StatusService Firewalld status Redirecting to /bin/systemctl status Firewalld. Service ● Firewalld. Service Loaded: masked (/dev/null; bad) Active: inactive (dead)#Disabling the Firewall
service firewalld stop
#Disable automatic firewall startup
chkconfig firewalld off
Copy the code

  3. JDK installation, each node needs:

    #Check which Java is used. If the Java is native to the system, uninstall and install it. If the Java is already installed, skip the subsequent stepsWhich Java/usr/Java/jdk1.8.0 _181 / bin/Java
#View the Java installed on the system
yum list installed | grep java
#Uninstall the Java installed by the system
yum remove -y java*

#Before installing Java, upload the tar package to the server
mkdir /usr/local/java
tar -zxvf jdk-8u188-linux-x64.tar.gz -C /usr/local/java

#Configure the environment variables and add them to /etc/profile
#JAVA_HOMEExport JAVA_HOME=/usr/local/java/jdk1.8.0_181 export CLASSPATH=.:$JAVA_HOME/lib:/lib export PATH=$JAVA_HOME/bin:$PATH
#Put environment variables into effect
source /etc/profile

#Check whether it takes EffectJava -version Java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-B13) Java HotSpot(TM) 64-bit Server VM (Build 25.181-B13, Mixed mode)Copy the code

  4. Hostname change, enter hostname of each node under /etc/hostname:

    noed1
Copy the code

  5. SSH configuration:

    #Log in remotely to other nodes on the primary node node1. Change the hostname of the remote node to that of the environmentSSH node2 Last login: Fri Aug 28 09:37:53 2020 from 192.168.0.3
## If not set, set secret free login:
#Key pair generation on master node (all the way back) :
ssh-keygen -t rsa
#Copy the public key
cd ~/.ssh; cp id_rsa.pub authorized_keys

#Distribute the public key to other nodes
#Directories are created for all other nodes first
mkdir ~/.ssh;

#Method 1: Know the root password:
scp authorized_keys root@node2:~/.ssh/authorized_keys
#Method two, if you don't know the password, you have to copy the key manuallyCat ~/.ssh/id_rsa.pub # master vim ~/.ssh/authorized_keys # copy the public key to other nodes
#After the public key is copied, other nodes change directory permissions
chmod -R 600 ~/.ssh 

#Try again on the primary node to log in remotely to the other nodes
ssh node2
Copy the code

  6. Hosts Configuration (/etc/hosts) :

    vim /etc/hosts
#Add the following content and modify it according to the actual environment:
192.168.0.81 node1
192.168.0.82 node2
192.168.0.83 node3
192.168.0.84 node4

#Distribute hosts to other nodes
scp /etc/hosts root@node2:/etc/hosts
Copy the code

Create a centos local source

Centos local source is created only when the system is minimal and has no external network (or slow external network). Skip this step if the network is good

  1. Select the corresponding centos-DVD or everything version: Download centos Linux ISO Images

  2. HTTPD service installed and self-started:

    RPM RPM -ivh mailcap-2.1.48-2.mga7.noarch. RPM RPM -ivh apr-1.4.8-5.el7.x86_64. RPM RPM -ivh apr-util-1.5.2-6.el7.x86_64. RPM RPM RPM -ivh httpd-tools-2.4.6-93.el7.centos.x86_64. service HTTPD start chkconfig httpd onCopy the code

  3. Upload the downloaded centos-7-x86_64-DVD-1810. Iso file to the /opt/ OS directory on the server and mount the file.

    #-o loop: specifies the mounting mode. Loop indicates that a file is mounted to the system as a disk partition
#-t ISO9660: specifies the file system type. The ISO file type is ISO9660. You can query other file types by yourselfMount -o loop -t iso9660 /opt/ OS/centos-7-x86_64-DVD-1810. iso /var/ WWW/HTML /CentOS mount: /dev/loop0 for write protection, mount it in read-only modeCopy the code

  4. Delete existing yum libraries:

    cd /etc/yum.repos.d/ mkdir .. /etc/yum.repos.d.bak mv ./* .. /etc/yum.repos.d.bakCopy the code

  5. In the/etc/yum. Repos. D/base. Under the repo to fill in the following content:

    [centos]
name=centos
baseurl=http://192.168.0.251/centos
enabled=1
gpgcheck=0
# Parameter meanings:
# name: indicates the name of the yum source
# baseurl: indicates the software directory of the yum source. "http://" indicates that the yum source is a remote HTTP source. Of course, this can also be FTP protocol or other file transfer protocol;
# enabled: indicates whether the yum configuration segment takes effect. 1 indicates yes, and 0 indicates no
# gpgCheck: indicates whether security check is performed on the packages specified by the yum source. 0 indicates no security check.
Copy the code

  6. Yum yum yum yum yum yum yum yum yum yum yum yum

    yum clean all
yum update
Copy the code

Dependency library preinstallation

  1. If the system is minimal, the following libraries are missing and need to be installed. If not, it is better to confirm whether there is any error to avoid installation.

  2. System library installation:

    yum install -y redhat-lsb yum-utils createrepo net-tools
Copy the code

  3. Mysql, HDP, etc.

    yum install -y libaio libaio-devel postgresql-devel

#If the centos DVD is mounted only, the libtirpc-devel file is not available. If the network is not available, download the corresponding RPM package and upload it to manually install the centos DVD. For details about the download address of the RPM package, see installation instructionsRPM -ivh libtirpc-devel-0.2.4-0.16.el7.x86_64. RPM -ivh libtirpc-devel-0.2.4-0.16.el7.x86_64. RPMCopy the code

Mysql installation

  1. Uninstall mysql on Linux

    RPM - qa | grep mysql RPM -e -- nodeps mysql connector - Java - 5.1.25-3. El7. NoarchCopy the code

  2. Uninstall mariadb that comes with Linux

    The RPM - qa | grep mariadb RPM -e -- nodeps mariadb - libs - 5.5.65-1. El7. X86_64
#Delete the mysql configuration file.
rm -f /etc/my.cnf
Copy the code

  3. Mysql > select * from user;

    cat /etc/passwd | grep mysql
cat /etc/group | grep mysql

#If no, create users and groups that do not log in
useradd -s /sbin/nologin mysql
Copy the code

  4. Mysql installation

    1. Upload the mysql installation package to the corresponding server node

    2. Mysql > install mysql

      RPM RPM -ivh mysql-community-common-5.7.30-1.el7.x86_64. RPM RPM -ivh mysql-community-libs-5.7.30-1.el7.x86_64. RPM Mysql - community - the client - 5.7.30-1. El7. X86_64. RPM RPM - the ivh mysql - community - server - 5.7.30-1. El7. X86_64. RPMCopy the code

    3. Start the mysql service: service mysqld start

    4. Mysql service startup: chkconfig mysqld on

    5. Run the service mysqld status command to check the running status of the mysql service

    6. To view the automatically initialized root password, run the grep password /var/log/mysqld.log command

    7. Initial password for logging in to mysql: mysql -u root -p

    8. Reset password: set password = password(“ABcd_123456”). The password must be longer than 8 characters and contains special characters and digits for large messages. If you need to set a simple password, Your password does not satisfy the current policy requirements.

    9. Change the character set to UTf8mb4 and add the following content to /etc/my.cnf:

      [client] 
default-character-set=utf8mb4 

[mysql] 
default-character-set=utf8mb4 

[mysqld] 

character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci      
init_connect='SET NAMES utf8mb4'
Copy the code

    10. Set pid, data, log to mysql:

    chown -R mysql:mysql /var/lib/mysql
chown -R mysql:mysql /var/run/mysqld/
chown mysql:mysql /var/log/mysqld.log
Copy the code

    1. Run the following command to restart mysql: service mysqld restart

    2. To check character set information: SHOW VARIABLES WHERE Variable_name LIKE ‘character_set_%’ OR Variable_name LIKE ‘collation%’;

  5. Log in to the Hive and Ambari databases as user root and create the hive and Ambari databases and their corresponding users. Change the passwords to your own passwords.

    CREATE DATABASE hive CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE ambari CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE USER 'hive'@The '%'IDENTIFIED BY 'Hive_123';
CREATE USER 'ambari'@The '%'IDENTIFIED BY 'Ambari_123';

GRANT ALL PRIVILEGES ON hive.* TO 'hive'@The '%' identified by 'Hive_123';
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@The '%' identified by 'Ambari_123';

flush privileges;
Copy the code

HDP installation

  1. Before installing HDP, ensure that the JDK8 has been installed on all nodes, SSH password-free login for the master node to other nodes has been configured, and the firewall and SELinux have been disabled.

  2. Upload the installation file to the master node, including the following files:

    Ambari 2.7.3.0 - centos7. Tar. Gz HDP - 3.1.0.0 - centos7 - RPM tar. Gz HDP - the GPL - 3.1.0.0 - centos7 - the GPL. Tar. Gz HDP - UTILS - 1.1.0.22 - centos7. Tar. GzCopy the code

  3. Install the yum tool library:

    yum install -y yum-utils
yum install -y createrepo
yum repolist

#The following commands are used for offline installation:RPM -ivh./yum-utils-1.1.31-54.el7_8.noarch. RPM RPM -ivh./createrepo-0.9.9-28.el7.noarch. RPMCopy the code

  4. Install apache HTTPD on the master node.

    yum install -y httpd

#The following commands are used for offline installation:RPM RPM -ivh mailcap-2.1.48-2.mga7.noarch. RPM RPM -ivh apr-1.4.8-5.el7.x86_64. RPM RPM -ivh apr-util-1.5.2-6.el7.x86_64. RPM RPM -ivh httpd-tools-2.4.6-93.el7.centos.x86_64. RPM -ivh httpd-2.4.6-93.el7.centos.x86_64. RPMCopy the code

  5. Create ambari and HDP directories in /var/www/html generated by HTTPD

    mkdir /var/www/html/ambari
mkdir /var/www/html/hdp
Copy the code

  6. Unzip the ambari and HDP installation files into their respective HTML directories:

    Tar -zxvf ambari-2.7.3.0-centos7.tar.gz -c /var/ WWW/HTML /ambari/ tar -zxvf hdP-3.1.0.0-centos7-rpm.tar.gz -c /var/ WWW/HTML /ambari/ tar -zxvf hdP-3.1.0.0-centos7-rpm /var/ WWW/HTML/HDDP/tar -zxvf HDP-gPL-3.1.0.0-centos7-gpl.tar. gz -c /var/www/ HTML/hdp-tar -zxvf HDP - UTILS - 1.1.0.22 - centos7. Tar. Gz - C/var/WWW/HTML/HDP /Copy the code

  7. Configure ambari local repo to the/etc/yum repos. D/ambari. Write the content in repo, baseurl and gpgkey modification for the local path:

    # VERSION_NUMBER = 2.7.3.0-139
[ambari - 2.7.3.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=Ambari Version - ambari - 2.7.3.0
# baseurl=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0
baseurl=http://192.168.0.251/ambari/ambari/centos7/2.7.3.0-139
gpgcheck=1
# gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://192.168.0.251/ambari/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
Copy the code

  8. Configure local repo HDP * * (don’t do, ambari according to the configuration page automatically generated) * *, in the/etc/yum repos. D/HDP. Fill in repo with the following contents:

    # VERSION_NUMBER = 3.1.0.0-78

[HDP - 3.1.0.0]
name=HDP Version - HDP - 3.1.0.0
baseurl=http://192.168.0.251/hdp/HDP/centos7/
gpgcheck=1
gpgkey=http://192.168.0.251/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP GPL - 3.1.0.0]
name=HDP - the GPL Version - HDP - the GPL - 3.1.0.0
baseurl=http://192.168.0.251/hdp/HDP-GPL/centos7/
gpgcheck=1
gpgkey=http://192.168.0.251/hdp/HDP-GPL/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1


[HDP UTILS - 1.1.0.22]
name=HDP - UTILS Version - HDP - UTILS - 1.1.0.22
baseurl=http://192.168.0.251/hdp/HDP-UTILS/centos7/
gpgcheck=1
gpgkey=http://192.168.0.251/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
Copy the code

  9. Generate local source (HDP directory) :

    createrepo /var/www/html/ambari/ambari/centos7
createrepo /var/www/html/hdp/HDP/centos7
createrepo /var/www/html/hdp/HDP-UTILS/centos7
createrepo /var/www/html/hdp/HDP-GPL/centos7
Copy the code

    Browser to open the http://192.168.0.251/ambari/ and http://192.168.0.251/hdp/ to see the following contents:

  10. More line yum

    yum clean all
yum update
yum repolist
Copy the code

  11. Ambari-server is installed on the master node

    1. To upload the mysql driver to any directory on the server, I put it in /usr/share/java

    2. Install ambari-server: yum install -y ambari-server

    3. Ambari database initialization:

      mysql -u ambari -p
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
show tables;

# 111 tables in total
Copy the code

    4. Create user ambari for all nodes ** (root is used for HDP installation, all boot-ups will use ambari) ** :

      useradd ambari
passwd ambari
Copy the code

    5. Add ambari sudo to /etc/sudoers;

      ambari  ALL=(ALL) NOPASSWD:ALL
Copy the code

    6. Ambari-server setup: Ambari-server setup: set user (ambari, root by default), JavaHome, mysql connection information, etc.

  12. Start the Ambari service

    ambari-server start
Copy the code

  13. HDP installation:

    1. Log in to ambari UI: 192.168.0.251:8080. The default user name and password are admin:admin

    2. Set the cluster name:

    3. Add HDP 3.1.0.0-78. XML file path for HTTPD service HDP Here for http://192.168.0.251/hdp/HDP/centos7/3.1.0.0-78/HDP-3.1.0.0-78.xml, add to the version information for HDP after installation:

    4. Select the HDP version, check Use Local Repository, delete other OS, and fill in the corresponding HTTP path of Redhat7:

      http://192.168.0.251/hdp/HDP/centos7/
http://192.168.0.251/hdp/HDP-GPL/centos7/
http://192.168.0.251/hdp/HDP-UTILS/centos7/
Copy the code

    5. The key is the private key of user root on the master node. The default path is ~/. Ssh/id_rsa. If the SSH port is changed, perform the following operations:

    6. Click here to see the Wamings after installing the wamings.

    7. Select required services. HDFS, YARN, TEZ, Hive, ZooKeeper, Ambari Metrics, SmartSense, SparK2, hbase, Kafka, hbase, and Kafka are optional.

    8. Master configuration: Configure components based on the preceding cluster planning:

    9. Slave configuration: Components are configured according to the preceding cluster planning:

    10. Mysql :mysql://dp-slave3/hive :mysql: mysql://dp-slave3/hive –jdbc-driver=/usr/share/ Java /mysql-connector-java-5.1.47.jar –jdbc-driver=/usr/share/ Java /mysql-connector-java-5.1.47.jar

      You do not need to manually create data directories. If HDFS: Hadoop users and groups already exist, ensure that they have read and write permissions; otherwise, services may fail to start. Change the directory to a data directory with large disk space based on disk mounting.

    11. review:

    12. Install and start services:

      After the installation is complete, some services may fail to start. Log in to the management page and try to manually restart the services. The causes are mostly caused by directory permission or port occupation:

      Finally, ensure that the following components start normally:

HDFS HA structures,

HA is set up after a fully distributed cluster is set up and services are started

  1. Enable the HA configuration in the HDFS service:

  2. Setting a namespace for an HA cluster:

  3. Select nodes to deploy additional NameNode:

  4. List of components removed and added between nodes:

  5. Changing the parameter Parameter Changing the parameter parameter Changing the configuration of a NameNode node

  6. Component configuration:

  7. JournalNode configuration, log in to the NameNode node to execute the given command:

  8. Service installation:

  9. ZKFC formatting and NameNode metadata initialization (note that ZKFC formatting is performed in NameNode1 and NameNode metadata initialization is performed in the new NameNode node) :

  10. Service startup:

Yarn HA structures,

Yarn HA is also set up after a fully distributed cluster is set up and services are startedCopy the code

  1. After yarn HA is enabled in the YARN component:

  2. Select a node to install ResourceManager.

  3. Add and delete service list:

  4. Installation services:

Hiveserver2 HA structures,

  1. Enter the node where you want to install extra and iveserver2 in hosts:

  2. Select Hiveserver2 from add to install:

The service start

After all services have been installed, subsequent service management should not start maintenance as root, here it is managed through ambari:

  1. Close all services in ambari-UI and then stop the Ambari service:

    #Execution of all nodes
service ambari-agent stop
#Primary node execution
service ambari-server stop
#Execution of all nodeschown -R ambari:ambari /var/log/ambari-agent/ /var/log/ambari-server/ chown -R ambari:ambari /var/lib/ambari-agent /var/lib/ambari-server chown -R ams:hadoop /var/lib/ambari-metrics-monitor/ chown -R ambari:ambari /var/run/ambari-agent  /var/run/ambari-serverCopy the code

  2. Then switch to the Ambari user to restart the Ambari service, and log in to the Ambari -UI to start the cluster service:

    #Execution of all nodes
su ambari
#Primary node execution
sudo service ambari-server start
#Execution of all nodes
sudo service ambari-agent start
Copy the code

  3. Finally select the service to use to start:

Problem solving

  1. UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 2: ordinal not in range(128):

    Python STR default is ASCII encoding and unicode conflict, will report the error, in the/usr/lib/python2.7 / site – packages/sitecustomize. Py add the following content:

    # encoding=utf-8

import sys

reload(sys)
sys.setdefaultencoding('utf-8')
Copy the code