Introduction to environment:

Centos7

Sqoop1.99.6

Hadoop2.6

Mysql5.6

1 Sqoop1.99.6 installation

1.1 Sqoop1.99.6 installation

First of all, from the website (mirror.bit.edu.cn/apache/sqoo…). Download sqoop-1.99.6-bin-hadoop200.tar.gz and decompress it in /usr/local/sqoop.

Add the following configuration to /etc/profile
export SQOOP_HOME=/usr/local/sqoopexport CATALINA_HOME=
S Q O O P H O M E / s e r v e r e x p o r t L O G D I R = SQOOP_HOME/serverexport LOGDIR= SQOOP_HOME/logs

Add SQOOP_HOME to PATH, export PATH= PATH:PATH:PATH:JAVA_HOME/bin: $SQOOP_HOME/bin.

Enter/usr/local/sqoop/server/conf/directory, modify sqoop. Properties
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop (hadoop installation directory) org. Apache. Sqoop.. Log4j appenders. File. The file = / usr/local/sqoop/logs/sqoop log# org. Apache. Sqoop.. Log4j appenders. Fi le.File=@LOGDIR@/sqoop.log#org.apache.sqoop.auditlogger.default.file=@LOGDIR@/default.auditorg.apache.sqoop.auditlogger. default.file=/usr/local/sqoop/logs/default.audit

Modify the catalina. The properties

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina .home}/.. /lib/*.jar,/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop /share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/u sr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hado op/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar,/usr/local/ha doop/share/hadoop/httpfs/tomcat/lib/*.jarCopy the code

Copy the mysql connector – Java – 5.1.39 – bin. The jar to/usr/local/sqoop/server/lib directory. Create a new lib directory under /usr/local/sqoop/.

1.2 Sqoop1.99.6 start

Go to the /usr/local/sqoop/bin directory and run the
sqoop.sh server start

Run the following command to enter the command line interface (CLI) :
sqoop.sh client

Setting the Server
Set server – host localhost – port 12000 – webapp sqoop

Viewing version Information
show connector

​

Establish connections to MySQL and Hadoop respectively.​​

Viewing Connection Information

​​​​​​​​

Creating job Tasks

​​​​​​​​ 

See the sqoop.apache.org/docs/1.99.6…

The job is successfully started if the following information is displayed:

​

1.3 mysql data table

In the exported MySQL table, the user table is located under databases:test.

​​​​​​​​ 

 

1.4 Viewing information on Hadoop

Warning: Make sure your Hadoop installation is correct!!

The following two pages appear.

​

​

In /sqoop there are the following

​

These two files are generated TXT files, open to view

​

It is found that the data is exactly the same as the data in MySQL.

Note: If the following problems occur during the import process, please check the sqoop.log file under /usr/local/sqoop/log/file, according to the corresponding error to solve the problem.

Exception has occurred during processing command 
Exception:org.apache.sqoop.common.SqoopExceptionMessage: CLIENT_0001:Server has returned exception
Copy the code

​ 

​