Hadoop Map Reduce任务:类org.apache.hive.hcatalog.mapreduce。HCatInputFormat未找到

I'm using kylin. It is a data warehouse tool and it uses hadoop, hive and hbase. It is shipped with sample data so that we can test the system. I was building this sample. It is a multi-step process many of the steps are map-reduce jobs. Second step is Extract Fact Table Distinct Columns which is a MR job. This job is failing without writing anything in hadoop logs. After digging deeper I find one Exception in logs/userlogs/application_1450941430146_0002/container_1450941430146_0002_01_000004/syslog

我用麒麟。它是一个数据仓库工具，它使用hadoop, hive和hbase。它附带了示例数据，以便我们可以测试系统。我正在构建这个样本。这是一个多步骤的过程，许多步骤都是map-reduce任务。第二步是提取事实表不同的列，这是一个MR job。如果不使用hadoop日志编写任何内容，此任务将失败。在深入挖掘之后，我发现日志/userlogs/application_1450941430146_0002/container_1450941430146_0002_01_000004/syslog中有一个异常

2015-12-24 07:31:03,034 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:174)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
    ... 8 more

2015-12-24 07:31:03,037 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task

My question is should I copy all dependencies jar of mapper class to all hadoop node? This job succeeds if I restarts kylin server and resume cube building job. This behavior is observed again when restart it after cleaning up everything.

我的问题是我是否应该将mapper类的所有依赖项jar复制到所有的hadoop节点?如果重新启动kylin服务器并恢复多维数据集构建作业，此作业将成功。当清理完所有东西后重新启动时，将再次观察此行为。

I am using 5 node cluster, each node is 8 core and 30GB. NameNode is running on one node. DataNode is running on all 5 nodes. For Hbase; HMaster and HQuorumPeer is running on same node as NameNode and HRegionServer is running on all nodes. Hive and Kylin are deployed on Master Node.

我正在使用5个节点集群，每个节点是8个核心，30GB。NameNode在一个节点上运行。DataNode在所有5个节点上运行。Hbase;HMaster和HQuorumPeer在相同的节点上运行，NameNode和hlocationserver在所有节点上运行。Hive和Kylin部署在主节点上。

Version information:

Ubuntu 12.04 (64 bit)
Hadoop 2.7.1
Hbase  0.98.16
Hive   0.14.0
Kylin  1.1.1

2 个解决方案

#1

The issue here is Kylin assumes the same Hive jars on all Hadoop nodes. And when certain node missing the Hive jars (or even in different location), you get the ClassNotFoundException on HCatInputFormat.

这里的问题是Kylin假设所有Hadoop节点上都有相同的Hive jar。当某些节点丢失了Hive jar(甚至在不同的位置)时，您可以在HCatInputFormat上获得ClassNotFoundException。

Btw, you should be able to get a clear error message from Yarn job console. This is a met issue.

顺便说一下，您应该能够从纱线作业控制台得到一个明确的错误信息。这是一个常见的问题。

Deploying Hive to all cluster nodes can surely fix the problem, like you have tried.

将Hive部署到所有集群节点肯定可以解决这个问题，就像您已经尝试过的那样。

Or another (cleaner) workaround is manually configure Kylin to submit Hive jars as additional job dependencies. See https://issues.apache.org/jira/browse/KYLIN-1021

另一个(更干净的)解决方案是手动配置Kylin，将Hive jar作为附加的作业依赖项提交。见https://issues.apache.org/jira/browse/kylin - 1021

Finally there's also a open JIRA suggests that Kylin should submit Hive jars by default. See https://issues.apache.org/jira/browse/KYLIN-1082

最后还有一个开放的JIRA建议，Kylin应该默认提交Hive jar。见https://issues.apache.org/jira/browse/kylin - 1082

#2

What Li Yang suggested is correct. Here is my sequence of experiments. Kylin ships a jar file, specified by kylin.job.jar property in kylin.properties. So first I made a fat jar file with missing dependency, set path of this file in kylin.job.jar and ran the job again. The missing class was now shipped with MR job. The newly added dependencies had some dependencies which were not available on all nodes hence the job failed again. In next iteration I added those missing dependencies and tried again. Same result: newly added dependencies had some more dependencies which were not available on all nodes. Finally I extracted all classes available in $HIVE_HOME/lib/hive-*.jar and created a new jar. This time it worked. Size of the jar file was over 20MB. Since these jar files are used every time I run a cube job I coppied all these jar files to all the nodes under $HADOOP_HOME/share/hadoop/common/hive-metastore-0.14.0.jar.

李阳的建议是正确的。这是我的实验序列。Kylin提供一个jar文件，由kylin.job指定。在kylin.properties jar房产。首先，我创建了一个包含缺失依赖项的胖jar文件，在kylin.job中设置这个文件的路径。jar，又开始工作了。失踪的那班现在和约伯一起被运走了。新添加的依赖项有一些在所有节点上不可用的依赖项，因此作业再次失败。在下一次迭代中，我添加了那些丢失的依赖项并再次尝试。同样的结果:新添加的依赖项有一些在所有节点上都不可用的依赖项。最后，我提取了$HIVE_HOME/lib/hive-*中的所有类。jar并创建了一个新的jar。这次工作。jar文件的大小超过20MB。由于每次运行多维数据集作业时都要使用这些jar文件，所以我将所有这些jar文件复制到$HADOOP_HOME/share/hadoop/common/hive-metastore-0.14.0.jar下的所有节点。

I think kylin-job-1.1.1-incubating.jar should be created to include all its dependencies.

我认为kylin-job-1.1.1-incubating。应该创建jar来包含它的所有依赖项。

#1