Cloudera Manager installation experience for CDH5.x

Needless to say, first show the results of these days

  • Cloudera Manager admin configuration interface

  • HBase management interface and HBase Web UI

[Image uploading…(image-1c6abb-1513138023093)]

  • HDFS management interface and NameNode Web UI

[Image uploading…(image-82C66F-1513138023093)]

  • Hive management interface and HiveServer2 Web UI



  • Hue management interface and Web UI

The first time you log in, you will be prompted to create an administrator account and password

  • Oozie management interface and Oozie Web UI

# Oozie Web Console is disabled. To enable Oozie Web Console To install the Ext JS library. Reference: http://cdh01.aniu.so:11000/oozie/docs/DG_QuickStart.html
  • YARN (MR2 Included) admin interface and Web UI

  • ResourceManager Web UI

  • HistoryServer Web UI

[Image uploading…(image-47c126-1513138023093)]

  • ZooKeeper admin interface

The author chose the default when installing ZooKeeper here, so only one ZooKeeper is installed. However, I feel that the number of interfaces of ZooKeeper should be increased in the later stage

Let’s start with the installation notes

  • 1, the configuration environment should meet the requirements, to pure system environment
CM env 192.168.1.137 cdh01.aniu.so CentOS6.9 16G Memory 100G LVM volume 192.168.1.148 cdh02.aniu.so Aniu. So Centos6.9 4G Memory 70G CDH03. Aniu. So Centos6.9 4G Memory 70G CDH04. Aniu Centos6.9 4G Memory 70G LVM is available for use in the system. The host name can be defined by the user. Yum update-y & yum -y groupinstall "Development Tools"
  • 2. Close the firewall and disable SELinux
/etc/init.d/ip6tables stop chkconfig iptables off && chkconfig ip6tables off # Suggest that with the method of modifying kernel parameters close ip6tables vim/etc/modprobe. D/dist. Conf # editing this file, in the final to join: # Disable Disable ipv6 alias net-pF-10 off alias ipv6 off # selinux sed - I 's/ selinux =enforcing/ selinux =disabled/g' /etc/selinux/config setEnforce 0 #
  • 3. Kernel parameter adjustment
/etc/rc.local: /etc/rc.local: /etc/rc.local: /etc/rc.local: /etc/rc.local: /etc/rc.local: /etc/rc.local: /etc/rc.local  echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled

Note: All of the above operations need to be performed on all nodes

  • 4. Secrecy free authentication is configured between all nodes
# ssh-keygen-t rsa-b 2048 # Conconfirm that ssh-keygen-t rsa-b 2048 Cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys # author hosts.conf # CM node Aniu.so 192.168.1.149 cdh03.aniu.so 192.168.1.150 cdh04.aniu.so # Synchronize the key for ip in $(awk '{print $1}' hosts.conf ); do scp ~/.ssh/authorized_keys root@$ip:/root/.ssh ; Done or use ssh-copy-id [email protected] ssh-copy-id if ssh-copy-id id if ssh-copy-id [email protected] # The above operations also need to be performed on all nodes
  • 5. Install CM using Cloudera-Manger Repo
# in CM nodes perform wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo - P/etc/yum. Repos. D wget https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo -P /etc/yum.repos.d # yum clean all && yum Yum install oracle-j2sdk1.7-y yum install cloudera-manager-daemons cloudera-manager-server-y # In other nodes perform wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo - P/etc/yum. Repos. D yum install /etc/profile export JAVA_HOME=/usr/ Java /jdk1.7.0_67- Cloudera export $JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar :$JAVA_HOME/lib/tools.jar Source /etc/profile enforces the changed environment variable # Configure JAVA_HOME on all nodes
  • 6. CM node installs database or uses existing data
[mysql57-community] name= mysql5.7 community Server [mysql57-community] name= mysql5.7 community Server baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/ enabled = 1 gpgcheck = 0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql # yum install mysql-community-embedded mysql-community-server Mysql-community-devel mysql-community-client-y # mysql. CNF [root@cdh01 yum.repos. D]# cat /etc/my.cnf [client] port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] datadir = /opt/mysql socket = /var/lib/mysql/mysql.sock #skip-grant-tables skip-ssl disable-partition-engine-check port = 3306 skip-external-locking key_buffer_size = 16M max_allowed_packet = 1M table_open_cache = 64 sort_buffer_size = 512K net_buffer_length = 8K read_buffer_size = 256K read_rnd_buffer_size = 512K myisam_sort_buffer_size = 8M thread_cache_size = 8 query_cache_size = 8M tmp_table_size = 16M performance_schema_max_table_instances = 500 explicit_defaults_for_timestamp = true max_connections = 500 max_connect_errors = 100 open_files_limit = 8192 log-bin=mysql-bin binlog_format=mixed server-id = 1 expire_logs_days = 10 early-plugin-load = "" default_storage_engine = InnoDB innodb_file_per_table = 1 innodb_data_home_dir = /opt/mysql innodb_data_file_path = ibdata1:1024M; ibdata2:10M:autoextend innodb_log_group_home_dir = /opt/mysql innodb_buffer_pool_size = 16M innodb_log_file_size = 5M innodb_log_buffer_size = 8M innodb_flush_log_at_trx_commit = 1 innodb_lock_wait_timeout = 50 innodb_log_files_in_group =  3 innodb_buffer_pool_size = 12G innodb_log_file_size = 512M innodb_log_buffer_size = 256M innodb_flush_log_at_trx_commit = 2 innodb_lock_wait_timeout = 150 innodb_open_files = 600 innodb_max_dirty_pages_pct = 50 innodb_file_per_table = 1 [mysqldump] quick max_allowed_packet = 16M [mysql] no-auto-rehash [myisamchk] key_buffer_size = 20M sort_buffer_size = 20M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] xinteractive-timeout symbolic-links=0 slow_query_log long_query_time = 5 slow_query_log_file = /var/log/mysql-slow.log log-error = The/var/log/mysqld. Log pid - file = / var/run/mysqld/mysqld. Pid # initialized mysql. And set the start the database set the root password/usr/sbin/mysqld - the initialize - user = mysql - socket = / var/lib/mysql/mysql. The sock # executed first Mysql -u root-paniuops123. -e "create database CMF DEFAULT CHARACTER SET mysql -u root-paniuops123. -e "create database CMF DEFAULT CHARACTER SET  utf8;" mysql -u root -pAniuops123. -e "GRANT ALL PRIVILEGES ON `cmf`.* TO 'cmf'@'localhost' IDENTIFIED BY 'Aniunas123.'";"
  • Start cloudera-scm-server and configure Parcel
# generate db configuration file/usr/share/CMF/schema/scm_prepare_database. Sh mysql CMF CMF Aniucmf123. # start cloudera - SCM server. - /etc/init.d/cloudera-scm-server start: /etc/init.d/cloudera-scm-server start: /etc/init.d/cloudera-scm-server start: /etc/init.d/cloudera-scm-server start: /etc/init.d/cloudera-scm-server start: /opt/cloudera/parcel-repo/ http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json # note: CDH-5.13.1-1.cdh5.13.1.p0.2-el6. Parcel. Sha1 Cdh-5.13.1-1.cdh5.13.1.p0.2-el6. Parcels. Sha # Enforces and defaults to use local parcels. If sha1 is not changed, go to cloudera's official website to find the matching parcel and restart cloudera-scm-server. /etc/init.d/cloudera-scm-server restart tailf /var/log/cloudera-scm-server/cloudera-scm-server /cloudera-scm-server/cloudera-scm-server.log
  • Install CDH through CM management interface, precautions
# CM server startup success can be accessed through http://192.168.1.137:7180, the default account password: admin admin # * * * * the important content of the following words is important: do not check: Single user mode, the author in this mode has been installed many times have not been successful, interested people can test
  • The best one-time installation is successful. If the installation is not successful, it is recommended to try more than a few times. Snapshot operation is performed on the virtual machine initialized to facilitate recovery