Abstract: SqoOP-shell is a Loader shell, all its functions are implemented by executing the script “sqoop2-shell”.

This article is from FusionInsight SqoopShell Use Case in huawei Cloud community.

1. Introduction to SqoopShell

Sqoop-shell is a Loader shell tool, and all of its functions are implemented by executing the script “sqoop2-shell”.

The sqoOP-shell tool provides the following functions:

  • Support for creating and updating connectors

  • Support for creating and updating jobs

  • Support for deleting connectors and jobs

  • Jobs can be started synchronously or asynchronously

  • Support to stop working

  • You can query job status

  • You can query historical job execution records

  • Support for duplicating connectors and jobs

  • Support for creating and updating transformation steps

  • Support the specified line, notification character

Sqoop-shell supports the following modes:

  • Interactive mode

You can run the sqoop2-shell script without parameters to access the Loader specific interaction window. After the user enters the script, the tool returns the corresponding information to the interaction window.

  • Demonstration mode

The sqoop2-shell script takes a file name as an argument. This file stores multiple commands in a row. Sqoop-shell executes all the commands in the file in sequence. Alternatively, you can append a command to the sqoop2-shell script by using the -c parameter, and execute only one command at a time.

2. SqoopShell configuration

2.1 Configuring the Loader Client

1. Use PuTTY to log in to the client as the user who installed the client.

2. Run the following command to prevent timeout.

TMOUT=0

3. Run the following command to go to the Loader client installation directory. For example, the Loader client installation directory is /opt/ HadoopClient /Loader.

cd /opt/hadoopclient/Loader

4. Run the following command to configure environment variables.

Source/opt/hadoopclient/bigdata_env

5. Run the following command to decompress loader-tools-1.99.3.tar: decompress loader-tools-1.99.3.tar.

The tar XVF – loader – tools – 1.99.3. Tar

The new file is stored in the Loader-tools-1.99.3 directory.

6. Run the following command to modify the tool authorization configuration file login-info. XML, save the configuration file, and exit.

Vi loader – tools – 1.99.3 / loader – tool/job – config/login – info. XML

2.2 Configuring the SQoOpShell Configuration File

1. Use PuTTY to log in to the Loader client node as the user who installs the client.

2. Run the following command to go to the conf directory of sqoop-shell. For example, the Loader client installation directory is /opt/ HadoopClient /Loader.

CD/opt/hadoopclient/Loader/Loader – tools – 1.99.3 / sqoop – shell/conf

3. Run the following command to configure authentication information.

Vi client.properties

3. Sqoopshell example

3.1 Interactive Mode

1. Run the following command to enter the interactive mode (client** /opt/hadoopclient** is used as an example).

Source/opt/hadoopclient/bigdata_env

CD/opt/hadoopclient/Loader/Loader – tools – 1.99.3 / sqoop – shell

./sqoop2-shell

2. Get help information (\h get help information, \cr get help information for create, \up get updated help information, and so on)

3. View connectors

In the same way, you can view frames, work, connections, and so on

4. Create a connector

Select the corresponding connector according to the displayed connector information, create a connection according to the creation help information, and enter the corresponding connection information as prompted. Determine that the displayed parameters do not need to be set, and press the input to skip

5. Create homework

Select a link based on the displayed connection information, create a job based on the creation help information, and enter the corresponding information as prompted. Determine that the displayed parameters do not need to be set, and press enter to skip

You can also use — to help view options for creating jobs

Jason fetch method

  1. Login to the loader native page and create a job of the same type (such as TaiPingTab).

  1. Export the configuration JSON file of the job, copy the content of hops (including braces) in the JSON file, and save it as a new JSON file

  1. Adjust the JSON file based on service requirements.

3.2 Demonstration Mode

There are two modes of demonstration mode. One is the./sqoop2-shell+ script, in which the commands to be executed are configured

/sqoop2-shell -c “Commands to be executed” mode

Get help:

/sqoop2-shell -c “create connection-cngeneric -jdbc-connector –help” Obtain the help information about creating a connection

/sqoop2-shell -c “create job -xn mysql -t import–help” View the help information about the creation job

1. Script mode

CD/opt/hadoopclient/Loader/Loader – tools – 1.99.3 / sqoop – shell

Vi Batch processing command

The update operation

2. -c mode (with a command attached to the -c argument, sqoop-shell can execute only the attached command at a time)

Example Create an import task with the link as mysql TaipingTab

/sqoop2-shell -c “create job -xn mysql -t import-nametaipingTab — connecter-table-schemaname keshangbank –connector-table-tableNameTab –connector-table-needPartition false –connector-table-columnsid,name ,value –framework-output-storageType HIVE–framework-output-outputDirectory /user/loader/Tab–framework-throttling-partitionHandlers 10 –framework-throttling-extractors10 – trans/opt/client/Loader Loader – tools – 1.99.3 / Loader – tool/job – config/Taiping json – queue root. The default”

/sqoop2-shell-c “start job -n TaiPingTab -s”

/sqoop2-shell-c “delete job -n TaiPingTab”

Update: ./sqoop2-shell-c ‘update job -j 37 -name update-test –connector-table-sql”select * from keshangbank.update_test where time <“2020-2-2″ and ${CONDITIONS}” ‘

Note:

1. In the sqoop-shell directory createConnection createJob helpCmd startCmd, the parameters are described in detail

2. The method of obtaining Json files is the same as the reference method. For the method of obtaining Json files in demonstration mode, refer to the method of obtaining Json files in interactive mode

3. For details about parameters in this document, see the sqOOP-shell section of the product documentation

4. The updated parameters are consistent with the created parameters

  • Attachment: SqoopShell Use case docx743.88KB

    Attachment: Json. Rar

Click follow to learn about the fresh technologies of Huawei Cloud