The original connection: https://segmentfault.com/a/1190000021597763
The following methods can be used when importing data based on the auto-increment ID
Image source: http://cn.voidcc.com/question…
Below is the website https://sqoop.apache.org/docs document manual…
7.2.10. Incremental Imports
Sqoop provides an incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.
SQOOP provides an incremental import pattern that can be used to retrieve only rows that are new than some previously imported rowsets.
The following arguments control incremental imports:
The following parameters control incremental import:
Argument Description
–check-column (col)
Specifies the column to be examined when determining which rows to import. (the column should not be of type CHAR/NCHAR/VARCHAR/VARNCHAR/ LONGVARCHAR/LONGNVARCHAR)
Specifies the columns to check when determining the rows to import. (The column type must not be CHAR/NCHAR/VARCHAR/VARNCHAR/LONGVARCHAR/LONGNVARCHAR)
–incremental (mode)
Specifies how Sqoop determines which rows are new. Legal values for mode include append and lastmodified.
Specifies how SQOOP determines which rows are new. ModeIncludeAppend and the legal value of LastModified.
–last-value (value)
Specifies the maximum value of the check column from the previous import.
Specifies the maximum value of the checked column in the last import.
Sqoop supports two types of incremental imports:appendandlastmodified. You can use the–incrementalargument to specify the type of incremental import to perform.
SQOOP supports two types of incremental imports: Append and LastModified. You can use the –incremental parameter to specify the type of incremental import to perform.
You should specifyappendmode when importing a table where new rows are continually being added with increasing row id values. You specify the column containing the row’s id with–check-column. Sqoop imports rows where the check column has a value greater than the one specified with–last-value.
The schema should be specified when APPEND imports a table in which new rows will be added continuously as the row ID value increases. You can use the column –check-column that specifies that contains the row ID. SQOOP imports a row where the value of the check column is greater than the value specified by the use –last-value.
An alternate table update strategy supported by Sqoop is calledlastmodifiedmode. You should use this when rows of the source table may be updated, and each such update will set the value of a last-modified column to the current timestamp. Rows where the check column holds a timestamp more recent than the timestamp specified with–last-valueare imported.
SQOOP supports an alternate update strategy called the LastModified pattern. This method should be used when the rows of the source table are likely to be updated, and each such update sets the value of the last modified column to the current timestamp. — Last-value import checks for rows that hold the timestamp of the column compared to the timestamp updated with the specified timestamp.
At the end of an incremental import, the value which should be specified as–last-valuefor a subsequent import is printed to the screen. When running a subsequent import, you should specify–last-valuein this way to ensure you import only the new or updated data. This is handled automatically by creating an incremental import as a saved job, which is the preferred mechanism for performing a recurring incremental import. See the section on saved jobs later in this document for more information.
At the end of an incremental import, the –last-value values specified for subsequent imports will be displayed on the screen. When you run subsequent imports, –last-value should be specified in this way to ensure that only new or updated data is imported. This problem is handled automatically by creating incremental imports as saved jobs, which is the preferred mechanism for performing cyclic incremental imports. For more information, see the section on saved jobs later in this document.
Reference: http://cn.voidcc.com/question… https://sqoop.apache.org/docs…
The original connection: https://segmentfault.com/a/1190000021597763