This is the seventh day of my participation in the First Challenge 2022. For details: First Challenge 2022. Hive is a data warehouse tool based on Hadoop, so we need to install Hive first. This article hadoop and Hive installation and I stepped on some pit record to share with you
1. Installation environment
JDK 1.8
Install Hadoop
1. Download Hadoop
Mirror.bit.edu.cn/apache/hado… Choose the appropriate version
Download hadoop
Wget HTTP: / / http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gzCopy the codeTo decompress mv, change the name of mv for easy use
Tar -xzvf hadoop-3.2.0.tar. gz mv hadoop-3.2.0.tar. gz HadoopCopy the code2. Modify environment variables
Write hadoop environment information to environment variables
vim /etc/profile
export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$PATH
Copy the codeRun source etc/profile for it to take effect
3. Modify the configuration file
Sh file, vim etc/hadoop/hadoop-env.sh modify JAVA_HOME information
Export JAVA_HOME = / usr/lib/JVM/Java -- 1.8.0 comes with its - 1.8.0.262. B10-0. El7_8. X86_64Copy the codeExecute hadoop jar share/hadoop/graphs/hadoop – graphs – examples – 3.3.0. Jar grep input output ‘DFS [a-z], hadoop’s own example, Verify that Hadoop is installed successfully
3. Install Hive
1. Download Hive
Wget mirror.bit.edu.cn/apache/hive…
Decompress tar -zxvf apache-hive-3.1.2-bin.tar.gz
Change the name mv apache-hive-3.1.2-bin hive
2. Modify environment variables
vim /etc/profile
export HIVE_HOME=/opt/hive
export PATH=$MAVEN_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
Copy the codesource etc/profile
3. Modify hivesite configurations
<! -- WARNING!!! This file is auto generated for documentation purposes ONLY! -- > <! -- WARNING!!! Any changes you make to this file will be ignored by Hive. --> <! -- WARNING!!! You must make your changes in hive-site.xml instead. --> <! -- Hive Execution Parameters --> <! -- The following configuration has the original configuration. Search after modified or deleted after adding in the same position - > < property > < name > javax.mail. Jdo. Option. ConnectionUserName < / name > user name < value > root < value > / < / property > The < property > < name > javax.mail. Jdo. Option. ConnectionPassword < / name > password < value > 123456 < value > / < / property > < property > <name>javax.jdo.option.ConnectionURL</name>mysql <value>jdbc:mysql: / / 127.0.0.1:3306 / hive < value > / < / property > < property > < name > javax.mail. Jdo. Option. ConnectionDriverName < / name > mysql driver <value>com.mysql.jdbc.Driver</value> </property> <property> <name>hive.exec.script.wrapper</name> <value/> <description/> </property>Copy the codeCopy the mysql driver to hive/lib and go to /hive/bin
schematool -dbType mysql -initSchema
Copy the code4. Check whether the installation is successful
Hive –version View the current version
Hive Check whether the hive command operation line is displayed. If yes, the command operation line is displayed
Iv. Error records
1. Abnormal characters exist in the configuration file
As specified
Logging the initialized using the configuration in the jar: file: / opt/hive/lib/hive - common - 3.1. 2. The jar! /hive-log4j2.properties Async: true Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java: 263 ) at org.apache.hadoop.fs.Path.<init>(Path.java: 221 ) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java: 710 ) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java: 627 ) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java: 591 ) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java: 747 ) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java: 683 ) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 62 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43 ) at java.lang.reflect.Method.invoke(Method.java: 498 ) at org.apache.hadoop.util.RunJar.run(RunJar.java: 323 ) at org.apache.hadoop.util.RunJar.main(RunJar.java: 236 ) Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at java.net.URI.checkPath(URI.java: 1823 ) at java.net.URI.<init>(URI.java: 745 ) at org.apache.hadoop.fs.Path.initialize(Path.java: 260 ) ... 12 moreCopy the codeSolution:
Find the specified number of config file lines and delete the description
<property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/< username> is created, with ${hive.scratch.dir.permission}.</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive/local</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/tmp/hive/resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property>Copy the codeSecond, guava version is inconsistent
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8 at [row,col,system-id]: [3215, 96, "file:/opt/hive/conf/hive-site.xml" ] at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java: 3051 ) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java: 3000 ) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java: 2875 ) at org.apache.hadoop.conf.Configuration.get(Configuration.java: 1484 ) at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java: 4996 ) at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java: 5069 ) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java: 5156 ) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java: 5104 ) at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java: 96 ) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java: 1473 ) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 62 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43 ) at java.lang.reflect.Method.invoke(Method.java: 498 ) at org.apache.hadoop.util.RunJar.run(RunJar.java: 323 ) at org.apache.hadoop.util.RunJar.main(RunJar.java: 236 ) Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8 at [row,col,system-id]: [3215, 96, "file:/opt/hive/conf/hive-site.xml" ] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java: 621 ) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java: 491 ) at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java: 2456 ) at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java: 2403 ) at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java: 2369 ) at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java: 1515 ) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java: 2828 ) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java: 1123 ) at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java: 3347 ) at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java: 3141 ) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java: 3034 ) ... 15 moreCopy the codeSolutions:
1, com.google.com mon. Base. The Preconditions. CheckArgument in jars for this class: guava. Jar
Hadoop-3.2.1 (path: hadoop-share/hadoop-common/lib) : guava-27.0-jre.jar; In hive-3.1.2(path: hive/lib), the jar package is guava-19.0.1.jar
3. Change the JAR packages to the same version: Delete the jar packages of the earlier version of Hive and copy the jar packages of the earlier version of Hadoop to the Hive lib.
Restart problem solved!