021 – cdh6.2 + kylin2.6.2

This is the 21st article in the Technical Writing Project (including translation). Set a small goal of 999, at least 2 articles per week.

This paper mainly introduces how to use Kylin(2.6.2) to connect to CDH6.2.

prompt

• Because cdh6.2 uses hadoop3 and the current kylin3.0beta is just hadoop2, only kylin2.5+ can be installed, here select kylin2.6.2-cdh60 (cdh6.0)

Download the kylin2.6.2 binary package

Wget http://mirrors.tuna.tsinghua.edu.cn/apache/kylin/apache-kylin-2.6.2/apache-kylin-2.6.2-bin-cdh60.tar.gz tar ZXF Apache - kylin - 2.6.2 - bin - cdh60. Tar. Gz - C/usr /local/
ln -s /usr/local/ apache - kylin - 2.6.2 - bin - cdh60 / usr /local/kylin
Copy the code

cat << EOF | sudo tee -a /etc/profile
Set up the Java environment
exportJAVA_HOME = / usr/Java/jdk1.8.0 _181 cloudera /export CLASSPATH=.:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib:\$CLASSPATH
export KYLIN_HOME=/usr/local/kylin
export PATH=\$JAVA_HOME/bin:\$JAVA_HOME/jre/bin:\$PATH
export CDH_HOME=/opt/cloudera/parcels/CDH
export HBASE_HOME=\${CDH_HOME}/lib/hbase
export HBASE_CLASSPATH=\${HBASE_HOME}/ lib/hbase - common - 2.1.0 - cdh6.2.0. Jar EOFsource /etc/profile
Copy the code

Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
hbase-common lib not found
Copy the code

Create the Kylin and Spark directories in the HDFS

export HADOOP_USER_NAME=hdfs
Copy the code

Otherwise you will quote

$KYLIN_HOME/bin/check-env.sh
Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /usr/local/kylin
mkdir: Permission denied: user=root, access=WRITE, inode="/kylin":hdfs:supergroup:drwxr-xr-x
Failed to create hdfs:///kylin/spark-history. Please make sure the user has right to access hdfs:///kylin/spark-history
Copy the code

yum install -y net-tools
Copy the code

Otherwise you will quote

$KYLIN_HOME/bin/check-env.sh
Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /usr/local/kylin
/usr/local/kylin/bin/check-port-availability.sh: line 27: netstat: command not found
Copy the code

$KYLIN_HOME/bin/download-spark.sh
Copy the code

$KYLIN_HOME/bin/kylin.sh start Retrieving hadoop conf dir... Error: cannot find or fail to load the main class. Org. Apache hadoop, hbase. Util. GetJavaProperty KYLIN_HOME isset to /usr/local/kylin Retrieving hive dependency... Retrieving hbase dependency... Error: cannot find or fail to load the main class. Org. Apache hadoop, hbase. Util. GetJavaProperty Retrieving hadoop conf dir... Error: cannot find or fail to load the main class. Org. Apache hadoop, hbase. Util. GetJavaProperty Retrieving kafka dependency... Retrieving Spark dependency... spark not found,set SPARK_HOME, or run bin/download-spark.sh
Copy the code

Start the kylin

$KYLIN_HOME/bin/kylin.sh start
Copy the code

Output if successful

A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /usr/local/kylin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
Copy the code

Open the browser http://IP:7070/kylin. The user name and password are ADMIN and KYLIN

Import data

$KYLIN_HOME/bin/sample.sh

Retrieving hadoop conf dir...
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
Loading sample data into HDFS tmp path: /tmp/kylin/sample_cube/data
Going to create sample tables in hive to database DEFAULT by cli
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in[the jar file: / opt/cloudera/parcels/CDH 6.2.0-1. Cdh6.2.0. P0.967373 / jars /log4 j - slf4j - impl - 2.8.2 jar! /org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found bindingin[the jar file: / opt/cloudera/parcels/CDH 6.2.0-1. Cdh6.2.0. P0.967373 / jars/slf4j - log4j12-1.7.25. Jar! /org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration inJar: file: / opt/cloudera/parcels/CDH 6.2.0-1. Cdh6.2.0. P0.967373 / jars/hive - common - 2.1.1 - cdh6.2.0. Jar! /hive-log4j2.properties Async:false
OK
//....
Sample cube is created successfully in project 'learn_kylin'.

** Restart Kylin Server or click Web UI => System Tab => Reload Metadata to take effect **

Copy the code

Reload metadata

Build a Cube

Select Model, select kylin_Sales_model, select Build

>

Explain the demo table

Kylin’s example is sales business analysis

• KYLIN_SALES fact table that holds details of sales orders (seller, category of goods, order amount, quantity of goods, etc.)
• KYLIN_COUNTRY dimension table with country information (abbreviation, name, etc.)
• KYLIN_CATEGORY_GROUPINGS Dimension table with detailed description of product categories (category names, etc.)
• KYLIN_CAL_DT dimension table with time extension information (year, month, week, year, month, etc.)
• KYLIN_ACCOUNT dimension table with account information (account ID, seller rank, buyer rank, country, etc.)

Run the query

Select count(1) from kylin_sales and click Submit. The result and the execution time (1.8 seconds in this case) are displayed below. Kylin caches the results of the execution, and the reexecution of the discovery becomes 0.18 seconds

select sum(KYLIN_SALES.PRICE) 
as price_sum,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME 
from KYLIN_SALES inner join KYLIN_CATEGORY_GROUPINGS
on KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID and 
KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
group by KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
order by KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME asc,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME desc
Copy the code

Want ads

Friends in Jinan, Shandong, welcome to join us and do things together. Long-term recruitment, Java programmer, big data engineer, operation and maintenance engineer, front-end engineer.