What is Sqoop
Sqoop: SQL-to-Hadoop Bridge between traditional relational databases and Hadoop Import data from relational databases to Hadoop systems such as HDFS, HBase, and Hive. Data is extracted from Hadoop systems and exported to relational databases. Using MapReduce to speed up data transmission: Data synchronization is transformed into MR job batch processing for data transmission: real-time performance is not good enough
Advantages of Sqoop
Efficient and controllable use of resources, task parallelism, timeout time and other data types can be automatically mapped and converted, users can also customize support for a variety of databases MySQL, Oracle, PostgreSQL
Sqoop import
Step 1: Sqoop communicates with the database Server to obtain the metadata information of the database table. Step 2: Sqoop starts a Map-only MR job and writes data to Hadoop in parallel using metadata information. Features: you can specify the HDFS path, specify the table of the relational database, field, connection number (not crushing the database), can import multiple tables, support incremental import (manually specify the start ID, event, or automatically record the last end position, automatically complete the incremental import)
Sqoop Export
Step 1: Sqoop communicates with database Server to obtain metadata information of database tables. Step 2: Parallel data import: Divide files on Hadoop into several splits; Each split is imported by a Map Task.
Sqoop is integrated with other systems
Sqoop can be combined with Oozie, Hive, and Hbase. PS:
In the future, I will thank myself now. If I don't work hard now, I will only regret it in the future. We do not regret which, only do the best of their own. Pay attention to the official account, click "get information" in the background of the official account can get free learning video, below is the two-dimensional code of the official account, scan can pay attention to. If you want to engage in the above work or to develop in the direction of big data, you can click to contact us to obtain relevant information about big data and high-definition learning route map, hoping to help you on the road of development.Copy the code