Sqoop将mysql数据导入到hive表中
先在mysql创建表
CREATE TABLE `sqoop_test` (
`id` int() DEFAULT NULL,
`name` varchar() DEFAULT NULL,
`age` int() DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
插入数据
fz
dx
test
test_add
test_add-
test_add_2
在hive中创建表，表结构和mysql中一样
hive> create external table sqoop_test_table(id int,name string,age int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS TEXTFILE;
OK
Time taken: 0.083 seconds
开始导入
sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test
--hive-import --hive-overwrite --hive-table sqoop_test_table --fields-terminated-by ',' -m 1
EFdeMacBook-Pro:jarfile FengZhen$ sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --hive-import --hive-overwrite --hive-table sqoop_test_table --fields-terminated-by ',' -m 1
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3./lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
// :: INFO tool.CodeGenTool: Beginning code generation
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/FengZhen/Desktop/Hadoop/hadoop-2.8.
// :: INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-FengZhen/compile/241f28a04b0ece18cd4a07bd6939d50a/sqoop_test.jar
// :: WARN manager.MySQLManager: It looks like you are importing from mysql.
// :: WARN manager.MySQLManager: This transfer can be faster! Use the --direct
// :: WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
// :: INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
// :: INFO mapreduce.ImportJobBase: Beginning import of sqoop_test
// :: INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
// :: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
// :: INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
// :: INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:
// :: INFO db.DBInputFormat: Using read commited transaction isolation
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505439184374_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1505439184374_0001
// :: INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505439184374_0001/
// :: INFO mapreduce.Job: Running job: job_1505439184374_0001
// :: INFO mapreduce.Job: Job job_1505439184374_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1505439184374_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Other local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all map tasks=
Map-Reduce Framework
Map input records=
Map output records=
Input split bytes=
Spilled Records=
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
// :: INFO mapreduce.ImportJobBase: Transferred bytes in 22.0614 seconds (3.2636 bytes/sec)
// :: INFO mapreduce.ImportJobBase: Retrieved records.
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO hive.HiveImport: Loading uploaded data into Hive
// :: INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
// :: INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
// :: INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3./lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
// :: INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
// :: INFO hive.HiveImport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
// :: INFO hive.HiveImport: // :: WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
// :: INFO hive.HiveImport:
// :: INFO hive.HiveImport: Logging initialized using configuration in file:/Users/FengZhen/Desktop/Hadoop/hive/apache-hive-1.2.-bin/conf/hive-log4j.properties
// :: INFO hive.HiveImport: OK
// :: INFO hive.HiveImport: Time taken: 1.502 seconds
// :: INFO hive.HiveImport: Loading data to table default.sqoop_test_table
// :: INFO hive.HiveImport: Table default.sqoop_test_table stats: [numFiles=, numRows=, totalSize=, rawDataSize=]
// :: INFO hive.HiveImport: OK
// :: INFO hive.HiveImport: Time taken: 0.762 seconds
// :: INFO hive.HiveImport: Hive import complete.
// :: INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
导入成功后，会在hdfs中产生数据文件
在路径 /user/hive/warehouse/sqoop_test_table 下
EFdeMacBook-Pro:jarfile FengZhen$ hadoop fs -ls /user/hive/warehouse/sqoop_test_table
Found items
-rwxr-xr-x FengZhen supergroup -- : /user/hive/warehouse/sqoop_test_table/part-m- EFdeMacBook-Pro:jarfile FengZhen$ hadoop fs -text /user/hive/warehouse/sqoop_test_table/part-m-
,fz,
,dx,
,test,
,test_add,
,test_add-,
,test_add_2,
hive中查看表数据
hive> select * from sqoop_test_table;
OK
fz
dx
test
test_add
test_add-
test_add_2
Time taken: 0.507 seconds, Fetched: row(s)
完成。