Hadoop map-reduce操作在写入输出时失败

I am finally able to start a map-reduce job on Hadoop (running on a single debian machine). However, the map reduce job always fails with the following error:

我终于能够在Hadoop上启动一个map-reduce作业(在一台debian机器上运行)。但是，map reduce任务总是失败，错误如下:

hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main /user/hadoopmachine/input /user/hadoopmachine/output
Warning: $HADOOP_HOME is deprecated.

12/04/03 07:29:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
****hdfs://localhost:9000/user/hadoopmachine/input
12/04/03 07:29:35 INFO input.FileInputFormat: Total input paths to process : 1
12/04/03 07:29:35 INFO mapred.JobClient: Running job: job_201204030722_0002
12/04/03 07:29:36 INFO mapred.JobClient:  map 0% reduce 0%
12/04/03 07:29:41 INFO mapred.JobClient: Task Id : attempt_201204030722_0002_m_000002_0, Status : FAILED
Error initializing attempt_201204030722_0002_m_000002_0:
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
at java.lang.Thread.run(Thread.java:636)

12/04/03 07:29:41 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0002_m_000002_0&filter=stdout
12/04/03 07:29:41 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0002_m_000002_0&filter=stderr

Unfortunately, it only says: "ENOENT: No such file or directory", it doesn't say what directory it actually tries to access. Pinging localhost works, and the input directory does exist. The jar location is also correct.

不幸的是，它只说:“ENOENT:没有这样的文件或目录”，它没有说实际要访问的目录。ping localhost工作，输入目录确实存在。jar位置也是正确的。

Can anybody give me a pointer on how to fix for this error, or how to find out which file Hadoop is trying to access?

谁能给我指点一下如何修复这个错误，或者如何找出Hadoop要访问的文件?

I found several similar problems on the Hadoop mailing list, but no responses on those...

我在Hadoop邮件列表中发现了几个类似的问题，但是没有人回复。

Thanks!

谢谢!

P.S. The config for mapred.local.dir looks like this (in mapred-site.xml):

mapreed。local的配置。dir(在mapred-site.xml中)如下:

<property>
  <name>mapred.local.dir</name>
  <value>/home/hadoopmachine/hadoop_data/mapred</value>
  <final>true</final>
</property>

As requested, the output of ps auxww | grep TaskTracker is:

根据要求，ps auxww | grep任务跟踪器的输出为:

1000      4249  2.2  0.8 1181992 30176 ?       Sl   12:09   0:00
/usr/lib/jvm/java-6-openjdk/bin/java -Dproc_tasktracker -Xmx1000m -Dhadoop.log.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/../logs
-Dhadoop.log.file=hadoop-hadoopmachine-tasktracker-debian.log -Dhadoop.home.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/.. 
-Dhadoop.id.str=hadoopmachine -Dhadoop.root.logger=INFO,DRFA -Dhadoop.security.logger=INFO,NullAppender
-Djava.library.path=/home/hadoopmachine/hadoop-1.0.1/libexec/../lib/native/Linux-i386-32 
-Dhadoop.policy.file=hadoop-policy.xml -classpath [ommitted very long list of jars] org.apache.hadoop.mapred.TaskTracker

1 个解决方案

#1

From the job tracker, identify which hadoop node this task executed on. SSH to that node and identify the location of the hadoop.log.dir directory (check the mapred-site.xml for this node) - my guess is the hadoop user does not have the correct permissions to create sub-directories in this folder

从作业跟踪程序中，识别该任务在哪个hadoop节点上执行。SSH到该节点并标识hadoop.log的位置。目录目录(检查地图站点。这个节点的xml) -我的猜测是hadoop用户没有在这个文件夹中创建子目录的正确权限。

The actual folder it's trying to create lies under the ${hadoop.log.dir}/userlogs folder - check this folder has the correct permissions

它试图创建的实际文件夹位于${hadoop.log之下。目录}/userlogs文件夹—请检查该文件夹的权限是否正确

In your case, looking at the ps output, i'm guessing this is the folder you need to examine the permission of:

在你的例子中，看ps输出，我猜这是你需要检查权限的文件夹:

/home/hadoopmachine/hadoop-1.0.1/libexec/../logs

#1