hadoop执行hbase插入表操作,出错:Stack trace: ExitCodeException exitCode=1:(xjl456852原创)

时间：2022-04-14 13:04:32

在执行hbase和mapreduce融合时,将hdfs上的文本文件插入到hbase中,我没有使用"胖包"(胖包就是将项目依赖的jar包放入项目打包后的lib目录中),而是直接将hbase的lib加入到了hadoop的classpath中.

我是将HBase的jar包加到/opt/modules/hadoop/etc/hadoop/hadoop-env.sh中。配置好这个文件,分发到各个节点,改这个配置不用重启集群.

TEMP=`ls /opt/modules/hbase/lib/*.jar`

HBASE_JARS=`echo $TEMP | sed 's/ /:/g'`

HADOOP_CLASSPATH=$HBASE_JARS
可以通过命令查看hadoop的classpath现在包含哪些jar包: 
[hadoop@master ~]$ hdfs classpath

　　

运行hadoop的mapreduce程序
hadoop jar xx.jar input '表名'
 
执行后出错:
16/08/11 08:52:05 INFO mapreduce.Job: Running job: job_1470930593079_0001

16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 running in uber mode : false

16/08/11 08:52:24 INFO mapreduce.Job:  map 0% reduce 0%

16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 failed with state FAILED due to: Application application_1470930593079_0001 failed 2 times due to AM Container for appattempt_1470930593079_0001_000002 exited with  exitCode: 1

For more detailed output, check application tracking page:http://master.xjl456852.com:8088/cluster/app/application_1470930593079_0001Then, click on links to logs of each attempt.

Diagnostics: Exception from container-launch.

Container id: container_1470930593079_0001_02_000001

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)

	at org.apache.hadoop.util.Shell.run(Shell.java:456)

	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)

	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

	at java.util.concurrent.FutureTask.run(FutureTask.java:266)

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

	at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

Failing this attempt. Failing the application.

16/08/11 08:52:24 INFO mapreduce.Job: Counters: 0

　　

 
网上查询到需要在/opt/modules/hadoop/etc/hadoop/yarn-site.xml中加入mapreduce运行时需要的类库,需要设置yarn.application.classpath:
所以我在yarn-site.xml中加入了如下配置,并加入了hbase的lib目录,配置好这个文件,分发到各个节点,这个配置需要重启集群
<property>

  <name>yarn.application.classpath</name>

      <value>

           /opt/modules/hadoop/etc/*,

           /opt/modules/hadoop/etc/hadoop/*,

           /opt/modules/hadoop/lib/*,

           /opt/modules/hadoop/share/hadoop/common/*,

           /opt/modules/hadoop/share/hadoop/common/lib/*,

           /opt/modules/hadoop/share/hadoop/mapreduce/*,

           /opt/modules/hadoop/share/hadoop/mapreduce/lib/*,

           /opt/modules/hadoop/share/hadoop/hdfs/*,

           /opt/modules/hadoop/share/hadoop/hdfs/lib/*,

           /opt/modules/hadoop/share/hadoop/yarn/*,

           /opt/modules/hadoop/share/hadoop/yarn/lib/*,

		/opt/modules/hbase/lib/*

      </value>

</property>
 
 

相关文章