Hive installation

  1. Download the decompressed installation package
  2. Configure the environment variable HIVE_HOME
  3. Configure the Hive CONF file

Sh cp hive-default.xml.template hive-site. XML conf/hive-env.sh Specify HADOOP_HOME conf/hive-site.xml

<! Metadata database information -->
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>JDBC: mysql: / / 192.168.1.178:3306 / hive? createDatabaseIfNotExist=true</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
  <description>username to use against metastore database</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>hive</value>
  <description>password to use against metastore database</description>
</property>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9083</value>
</property>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive/warehouse</value>
</property>
<! -- specify the cache directory -->
<property>
  <name>hive.exec.local.scratchdir</name>
  <value>/home/hadoop/iotmp</value>
  <description>Local scratch space for Hive jobs</description>
</property>
<property>
  <name>hive.downloaded.resources.dir</name>
  <value>/home/hadoop/iotmp</value>
  <description>Temporary local directory for added resources in the remote file system.</description>
</property>
Copy the code
  1. Download the mysql-connector-java-5.1.27-bin.jar file and place it in $HIVE_HOME/lib
  2. Run the schematool -initschema-dbtype mysql command to initialize the database
  3. Start the metastore

vim hvie-site.xml

<property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9001</value>
</property>
Copy the code

Hive –service metastore 7. Run the hive test

CREATE TABLE t_hive (a int, b int, c int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
cat <<EOF >/tmp/t_hive.txt 1 2 3 4 1 2 1 2 8 EOF
LOAD DATA LOCAL INPATH '/tmp/t_hive.txt'OVERWRITE INTO TABLE t_hive ; [root @ hd222 hadoop - 2.7.5]# ./bin/hadoop dfs -cat /user/hive/warehouse/t_hive/test1, 2, 3, 4, 1, 2, 1, 2, 8Copy the code
  1. hiveserver2
nohup hive --service hiveserver2 &
kill -9 pid
Copy the code

Tez installation

  1. download

Wget mirrors.tuna.tsinghua.edu.cn/apache/tez/… 2. Modify pom.xml. (Optional) Because tez-UI relies on Node.js and Bower, tez-UI relies on Node.js and Bower. MVN Clean package-dskiptests =true -dmaven.javadoc. Skip =true -phadoop28 -dhadoop. version=3.1.2 Files generated in the tez-dict/target directory 3. Gz -c tez-0.9.1 HDFS dfS-put./tez-0.9.1/* /apps/tez/ mkdir tez-0.9.1/conf vim Tez 0.9.1 / conf/tez – site. XML

<configuration>
  <property>
    <name>tez.lib.uris</name>
    <value>hdfs://ns1/apps/tez/,hdfs://ns1/apps/tez/lib/</value>
  </property>
</configuration>
Copy the code

vim hive/conf/hive-env.sh

exportTEZ_HOME = / home/hadoop/apps/tez - 0.9.1exportTEZ_CONF_DIR = / home/hadoop/apps/tez - 0.9.1 / confexport HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${TEZ_CONF_DIR}:${TEZ_HOME}/ * :${TEZ_HOME}/lib/*
Copy the code

Pressure test

  1. Download and install
wget https://github.com/hortonworks/hive-testbench/archive/hive14.zip
unzip hive14.zip
cd hive-testbench-hive14/
./tpcds-build.sh
Copy the code

2 Generate test data and query scripts

export FORMAT=parquet
./tpcds-setup.sh 1000
Copy the code

The unit is G, change the FORMAT, such as ORc, parquet, etc

  1. Run the test

Test SQL script directory: sample-queries-tpcds

cd sample-queries-tpcds
hive> use tpcds_bin_partitioned_parquet_10;
hive> source query12.sql;
Copy the code
  1. Batch test

Modify hive configurations as required: sample-queries- TPCDS /testbench. Settings Modify the test script (Perl) as required: runsuite. pl perl runsuitecommon.pl

Problems encountered

  1. note: tpcds_kit.zip may be a plain executable, not an archive

Solution: sh -x./tpcds-build.sh

Building TPC-DS Data Generator
+ cd tpcds-gen
+ make
test -d target/tools/ || (cd target; unzip tpcds_kit.zip)
Archive:  tpcds_kit.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
note:  tpcds_kit.zip may be a plain executable, not an archive
unzip:  cannot find zipfile directory in one of tpcds_kit.zip or
        tpcds_kit.zip.zip, and cannot find tpcds_kit.zip.ZIP, period.
Copy the code

vim tpcds-gen/Makefile

target/tpcds_kit.zip: tpcds_kit.zip
        mkdir -p target/
        cp tpcds_kit.zip target/tpcds_kit.zip

tpcds_kit.zip:
        curl http://dev.hortonworks.com.s3.amazonaws.com/hive-testbench/tpcds/README
        curl --output tpcds_kit.zip http://dev.hortonworks.com.s3.amazonaws.com/hive-testbench/tpcds/TPCDS_Tools.zip
Copy the code

Delete curl tpcds_kit.zip from Makefile