sqoop2的使用测试

查看现有link

删除link

sqoop:000> delete link -name mysql2
sqoop:000> show link
+------+----------------+---------+
| Name | Connector Name | Enabled |
+------+----------------+---------+
+------+----------------+---------+

创建mysql link

sqoop:000> create link -c generic-jdbc-connector

Creating link for connector with name generic-jdbc-connector Please fill following values to create new link object

Name: mysqllink

Database connection

Driver class: com.mysql.jdbc.Driver

Connection String: jdbc:mysql://127.0.0.1:3306/test

Username: root

Password: ****

Fetch Size: 30

Connection Properties: There are currently 0 values in the map: entry#

SQL Dialect

Identifier enclose: 此处为空格

New link was successfully created with validation status OK and name mysqllink

查看新建的链接：

创建hdfs链接

sqoop:000> create link -c hdfs-connector

Creating link for connector with name hdfs-connector Please fill following values to create new link object

Name: hdfslink

HDFS cluster

URI: localhost:9000

Conf directory: /root/hadoop/hadoop-2.7.4/etc/hadoop

Additional configs:: There are currently 0 values in the map: entry#

New link was successfully created with validation status OK and name hdfslink

--------------------------------------------------------------------------------------

在创建mysql链接时，Identifier enclose：指定SQL中标识符的定界符，也就是说，有的SQL标示符是一个引号：select * from "table_name"，这种定界符在MySQL中是会报错的。这个属性默认值就是双引号，所以不能使用回车，必须将之覆盖，可以使用空格覆盖

hadoop fs -mkdir /sqoop

在sqoop客户端设置查看job详情：

set option --name verbose --value true

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root

如下处理：

在core-site.xml添加如下属性，其中

hadoop.proxyuser.root.groups中的root是用户，value里面的root是group

<property>

      <name>hadoop.proxyuser.root.groups</name>

      <value>root</value>

      <description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>

 </property>

 <property>

      <name>hadoop.proxyuser.root.hosts</name>

      <value>localhost</value>

      <description>The superuser can connect only from host1 and host2 to impersonate a user</description>

  </property>

--------------------------------------------------------------------------

create job -f "mysql-link" -t "hdfs-link"

-f指定from，即是数据源位置，-t指定to，即是目的地位置。本例是从MySQL传递数据到HDFS，所以就是from mysql to HDFS。参数值就是在创建链接（link）时指定的Name。

以下是各个属性

Name：一个标示符，自己指定即可。

Schema Name：指定Database或Schema的名字，在MySQL中，Schema同Database类似，具体什么区别没有深究过，但官网描述在创建时差不多。这里指定数据库名字为db_ez即可，本例的数据库。

Table Name：本例使用的数据库表为tb_forhadoop，自己指定导出的表。多表的情况请自行查看官方文档。

SQL Statement：就是sql查询语句，文档上说需要指定一个$condition，但我一直没有创建成功，貌似是一个条件子句。

配置完以上几项，又回出现element#提示符，提示输入一些hash值，这里我没有再指定什么。直接回车过。而以下几个配置我也是直接回车，使用默认值，大概是一些与数据库相关的参数。

Partition column:

Partition column nullable:

Boundary query

Last value

后面需要配置数据目的地各项值：

Null alue：大概说的是如果有空值用什么覆盖

File format：指定在HDFS中的数据文件是什么文件格式，这里使用TEXT_FILE，即最简单的文本文件。

Compression codec：用于指定使用什么压缩算法进行导出数据文件压缩，我指定NONE，这个也可以使用自定义的压缩算法CUSTOM，用Java实现相应的接口。

Custom codec：这个就是指定的custom压缩算法，本例选择NONE，所以直接回车过去。

Output directory：指定存储在HDFS文件系统中的路径，这里最好指定一个存在的路径，或者存在但路劲下是空的，貌似这样才能成功。

Append mode：用于指定是否是在已存在导出文件的情况下将新数据追加到数据文件中。

Extractors：不清楚是什么，我取了一个1

Loaders：同上

最后再次出现element#提示符，用于输入extra mapper jars的属性，可以什么都不写。直接回车。

至此若出现successful则证明已经成功创建。

创建job mysqltohdfs，将mysql导入到hdfs:

sqoop:000> show job -n mysqltohdfs
2018-05-28 17:12:55,421 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:12:55,425 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:12:55,431 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
1 job(s) to show:
Job with name mysqltohdfs (Enabled: true, Created by root at 5/28/18 3:52 PM, Updated by root at 5/28/18 3:52 PM)
Throttling resources
    Extractors:
    Loaders:
Classpath configuration
    Extra mapper jars:
From link: mysqllink
Database source
    Schema name: test   数据库必填
    Table name: role   表名必填
    SQL statement:
    Column names:
    Partition column:
    Partition column nullable:
    Boundary query:
Incremental read
    Check column:
    Last value:
To link: hdfslink
Target configuration
    Override null value:
    Null value:
    File format: TEXT_FILE    必选
    Compression codec: NONE 必选
    Custom codec:
    Output directory: hdfs://localhost:9000/sqoop   数据输出目标路径
    Append mode:

sqoop:000> start job -n mysqltohdfs
2018-05-28 17:07:02,252 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:07:02,256 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,259 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,262 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,263 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,264 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Found primary key columns [roleid]
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using partition column: roleid
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using min/max query: SELECT MIN( roleid ), MAX( roleid ) FROM test . role
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Boundaries for the job: min=1, max=650, columnType=4
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using dataSql: SELECT * FROM test . role WHERE ${CONDITIONS}
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Field names: roleid , name , dateid , addr , sex , level
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsToInitializer - Using working directory: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2
2018-05-28 17:07:02,354 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:02,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:02,906 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:02,909 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,008 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,009 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,014 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,015 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,018 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,020 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,103 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,104 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,107 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,108 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,116 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,123 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,178 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,179 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,182 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,183 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,184 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,186 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,188 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,273 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,274 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,276 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,279 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,281 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,340 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,340 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,351 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,423 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,424 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,428 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,429 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,439 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,442 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,446 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,535 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,535 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,538 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,538 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,539 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,540 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,541 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,567 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,618 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,618 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,621 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,636 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,685 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,686 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,688 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,691 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid <= ?
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,771 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,772 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,775 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,776 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,777 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,795 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,798 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt of size 53
2018-05-28 17:07:03,816 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt of size 120
2018-05-28 17:07:03,838 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt of size 0
2018-05-28 17:07:03,849 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt of size 50
2018-05-28 17:07:03,862 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt of size 58
2018-05-28 17:07:03,874 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/881898e2-8545-437e-b4cb-4838ac489c87.txt of size 102
2018-05-28 17:07:03,895 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt of size 0
2018-05-28 17:07:03,908 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt of size 0
2018-05-28 17:07:03,919 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt of size 0
2018-05-28 17:07:03,933 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt of size 0
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,961 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,979 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
Exception in thread "Thread-141" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
        at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
        at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-28 17:07:05,326 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:04 CST: FAILED

虽然有异常抛出，但数据成功写入hdfs,查看结果如下：

[root@host ~]# hdfs dfs -ls -R /sqoop
-rw-r--r--   1 root supergroup         53 2018-05-28 17:07 /sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
-rw-r--r--   1 root supergroup        120 2018-05-28 17:07 /sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
-rw-r--r--   1 root supergroup         50 2018-05-28 17:07 /sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
-rw-r--r--   1 root supergroup         58 2018-05-28 17:07 /sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
-rw-r--r--   1 root supergroup        102 2018-05-28 17:07 /sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt

[root@host ~]# hdfs dfs -cat /sqoop/*
400,NULL,'2017-11-15 14:49:56.000','anhui hefei',0,4
1,NULL,'2017-11-16 14:49:11.000','henan luohe linying',1,10
40,NULL,'2017-11-13 14:50:25.000','guangdong shenzhen',1,20
110,NULL,'2017-11-14 14:50:47.000','beijing',1,20
200,NULL,'2017-11-14 14:49:47.000','shandong qingdao',0,8
600,NULL,'2017-11-15 14:50:05.000','hunan changsha',0,91
650,NULL,'2017-11-01 17:24:34.000',NULL,1,29

查看作业的状态

sqoop:000> status job -n mysqltohdfs
2018-05-29 10:55:44,766 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 10:55:44,768 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,771 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,789 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
Submission details
Job Name: mysqltohdfs
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2018-05-28 17:07:02 CST
Lastly updated by: root
External ID: job_local1297704653_0002
http://localhost:8080/
2018-05-28 17:07:04 CST: FAILED

hdfs导入到mysql

创建空表role1：

mysql> create table role1 select * from role limit 0; Query OK, 0 rows affected (0.31 sec) Records: 0 Duplicates: 0 Warnings: 0

mysql> select * from role1 ;

Empty set (0.00 sec)

创建job

sqoop:000> create job -f hdfslink -t mysqllink

Creating job for links with from name hdfslink and to name mysqllink 2018-05-29 11:17:23,362 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,363 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,367 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,368 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit Please fill following values to create new job object Name: hdfs2018-05-29 11:17:27,053 [UpdateThread] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit

Name: hdfstomysql

Input configuration

Input directory: hdfs://localhost:9000/sqoop

Override null value: Null value:

Incremental import

Incremental type: 0 : NONE 1 : NEW_FILES

Choose: 0

Last imported date:

Database target

Schema name: test

Table name: role1

Column names:

There are currently 0 values in the list: element#

Staging table:

Clear stage table:

Throttling resources

Extractors: Loaders:

Classpath configuration

Extra mapper jars: There are currently 0 values in the list: element# 2018-05-29 11:18:02,792 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request 2018-05-29 11:18:02,793 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,795 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,798 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit New job was successfully created with validation status OK and name hdfstomysql

查看job:

sqoop:000> show job
2018-05-29 11:18:06,188 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:06,193 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,196 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,198 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
+----+-------------+------------------------------------+------------------------------------+---------+
| Id |    Name     |           From Connector           |            To Connector            | Enabled |
+----+-------------+------------------------------------+------------------------------------+---------+
| 15 | mysqltohdfs | mysqllink (generic-jdbc-connector) | hdfslink (hdfs-connector)          | true    |
| 17 | hdfstomysql | hdfslink (hdfs-connector)          | mysqllink (generic-jdbc-connector) | true    |
+----+-------------+------------------------------------+------------------------------------+---------+

执行job
sqoop:000> start job -n hdfstomysql
2018-05-29 11:18:19,429 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:19,432 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,434 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,437 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,438 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,440 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-29 11:18:19,484 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsFromInitializer - Input directory: hdfs://localhost:9000/sqoop
2018-05-29 11:18:19,489 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-29 11:18:19,509 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer - Using query to insert data: INSERT INTO test . role1 VALUES (?,?,?,?,?,?)
2018-05-29 11:18:19,514 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,666 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt
2018-05-29 11:18:20,386 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,388 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=0, end=26, length=26), hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=26, end=53, length=27)}
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 26
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 26
2018-05-29 11:18:20,470 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,473 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,489 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 26
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 27
2018-05-29 11:18:20,503 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 53
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,565 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,589 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,590 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 39
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,624 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,627 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,629 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,630 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,633 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=39, end=78, length=39)}
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 39
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 78
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,667 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 60
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,672 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,672 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,673 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=78, end=99, length=21), hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=99, end=120, length=21)}
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 78
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 99
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 21
2018-05-29 11:18:20,700 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 99
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 21
2018-05-29 11:18:20,712 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,717 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,717 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=0, end=25, length=25), hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=25, end=50, length=25)}
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 25
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 25
2018-05-29 11:18:20,750 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,751 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 25
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 50
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 25
2018-05-29 11:18:20,759 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,762 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,764 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,764 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=0, end=29, length=29), hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=29, end=58, length=29)}
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 29
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 29
2018-05-29 11:18:20,796 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,798 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,801 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 29
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 29
2018-05-29 11:18:20,808 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,813 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,813 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,815 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,817 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 39
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,843 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,844 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,849 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,849 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,851 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,852 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=39, end=70, length=31), hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=70, end=102, length=32)}
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 39
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 70
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 31
2018-05-29 11:18:20,882 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 57
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 70
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 32
2018-05-29 11:18:20,898 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,914 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
Exception in thread "Thread-255" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
        at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
        at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-29 11:18:22,031 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:21 CST: FAILED

查看mysql表中数据

mysql> select * from test.role1;
+--------+------+---------------------+---------------------+------+-------+
| roleid | name | dateid              | addr                | sex | level |
+--------+------+---------------------+---------------------+------+-------+
|    400 | NULL | 2017-11-15 14:49:56 | anhui hefei         |    0 |     4 |
|      1 | NULL | 2017-11-16 14:49:11 | henan luohe linying |    1 |    10 |
|     40 | NULL | 2017-11-13 14:50:25 | guangdong shenzhen |    1 |    20 |
|    110 | NULL | 2017-11-14 14:50:47 | beijing             |    1 |    20 |
|    200 | NULL | 2017-11-14 14:49:47 | shandong qingdao    |    0 |     8 |
|    600 | NULL | 2017-11-15 14:50:05 | hunan changsha      |    0 |    91 |
|    650 | NULL | 2017-11-01 17:24:34 | NULL                |    1 |    29 |
+--------+------+---------------------+---------------------+------+-------+
7 rows in set (0.00 sec)

sqoop2的使用测试的更多相关文章

sqoop1 与sqoop2的对比
Sqoop是一款开源的工具,主要用于在Hadoop和传统的数据库(mysql.postgresql等)进行数据的传递,可以将一个关系型数据库(例如:MySQL.Oracle.Postgres等)中的数 ...
sqoop2安装总结
sqoop2安装 1. 下载解压缩此次安装版本为1.99.6 # Decompress Sqoop distribution tarball tar -xvf sqoop-<version&g ...
Sqoop2开启Kerberos安全模式
Sqoop2开启Kerberos安全模式, 基于版本sqoop-1.99.7, 在已经安装好的sqoop2环境上配置kerberos. 1.安装规划 10.43.159.9 zdh-9 sqoop2k ...
SignalR系列续集[系列8&colon;SignalR的性能监测与服务器的负载测试]
目录 SignalR系列目录前言也是好久没写博客了,近期确实很忙,嗯..几个项目..头要炸..今天忙里偷闲.继续我们的小系列.. 先谢谢大家的支持.. 我们来聊聊SignalR的性能监测与服务器的 ...
Apache Ignite之集群应用测试
集群发现机制在Ignite中的集群号称是无中心的,而且支持命令行启动和嵌入应用启动,所以按理说很简单.而且集群有自动发现机制感觉对于懒人开发来说太好了,抱着试一试的心态测试一下吧. 在Apache ...
测试一下StringBuffer和StringBuilder及字面常量拼接三种字符串的效率
之前一篇里写过字符串常用类的三种方式<java中的字符串相关知识整理>,只不过这个只是分析并不知道他们之间会有多大的区别,或者所谓的StringBuffer能提升多少拼接效率呢?为此写个简 ...
TechEmpower 13轮测试中的ASP&period;NET Core性能测试
应用性能直接影响到托管服务的成本,因此公司在开发应用时需要格外注意应用所使用的Web框架,初创公司尤其如此.此外,糟糕的应用性能也会影响到用户体验,甚至会因此受到相关搜索引擎的降级处罚.在选择框架时, ...
&period;NET Core系列：4 测试
2016.6.27 微软已经正式发布了.NET Core 1.0 RTM,但是工具链还是预览版,同样的大量的开源测试库也都是至少发布了Alpha测试版支持.NET Core, 这篇文章 The Sta ...
渗透测试工具BurpSuite做网站的安全测试（基础版）
渗透测试工具BurpSuite做网站的安全测试(基础版) 版权声明:本文为博主原创文章,未经博主允许不得转载. 学习网址: https://t0data.gitbooks.io/burpsuite/c ...

随机推荐

MongoDB 介绍
MongoDB 是一个跨平台的,面向文档的数据库,提供高性能,高可用性和可扩展性方便. MongoDB工作在收集和文件的概念. 数据库数据库是一个物理容器集合.每个数据库都有自己的一套文件系统上的文 ...
Android--入门
之前自己在学校写过一些安卓应用,那时候没有系统地学过安卓,用到什么就网上找博客.找Demo,然后自己跟着敲一遍,有些东西也不太理解,现在打算做android开发这一块了,趁毕业之前赶紧多学些技术.先是 ...
VC++6&period;0 显示行号
通过VC6LineNumberAddin能够解决这个问题,方法如下. 一.下载该文件. 网上很多免费的. 二.使用方法: [1]:打开VC++6.0,点击菜单“工具(Tools) ...
opensuse pptp拨号设置注意事项
防火墙一定要关闭路由器要映射本地1723和1701端口 tcp协议(个别路由器和网络环境,可以先不设置) pptp配置注意协议的选择:
web登录与授权
web开发已经流行了很多年,登录与授权也基本有一套通用的流程,下面是我自己常用的登录与授权方式,欢迎大家讨论与吐槽. 概念: 登录是过程,授权是结果.登录只是为了获得页面的访问权限 or 操作权限 o ...
支持SMTP邮箱介绍
126邮箱:POP:POP.126.comSMTP:SMTP.126.comhttp://mail.126.com/help/client_04.htm 163邮箱:POP:pop.163.comSM ...
JAVA中extends 与implements有啥区别？
JAVA中extends 与implements有啥区别?1. 在类的声明中,通过关键字extends来创建一个类的子类.一个类通过关键字implements声明自己使用一个或者多个接口.extend ...
ReactiveX 学习笔记（3）转换数据流
Transforming Observables 本文的主题为转换 Observable 的操作符. 这里的 Observable 实质上是可观察的数据流. RxJava操作符(二)Transform ...
如何解决rar文件解压缩失败
附件经常会是一系列的压缩文件,下载是默认文件名是一个随机数字.因而下载完会出现压缩文件解压缩失败解决方法:下载时重命名为带一定顺序的文件名,如文件1,文件2,文件3等如何解决单个文件解压失败?论坛中 ...
Xamarin&period;Android RelativeLayout
初次接触Xamarin.Android. 由于国内Xamarin的资料少见,我大多参考JAVA原生代码,慢慢摸索过来. 我把摸索出来的结果广而告之,希望后来人能少走一点弯路,也希望你也能做出一份贡献. ...