Hadoop 2.5.2缓慢的响应能力

I've setup Hadoop 2.5.2 is pseudo distributed mode according the the documentation on a MacBook pro w/16GB ram currently running OS X 10.9.5.

根据目前运行OS X 10.9.5的MacBook pro w/16GB ram文件，我已经安装了Hadoop 2.5.2是伪分布式模式。

Everything seems to be working, but the issue is that it's extremely slow.

一切似乎都在起作用，但问题是它太慢了。

• start-dfs.sh takes a 3-5 minutes to start namenodes
• start-dfs。sh需要3-5分钟来启动namenodes。
• hdfs dfs -ls / takes about a minute or two to run.
• hdfs dfs -ls /运行大约需要一到两分钟。

I've ingested no data, and haven't done anything outside the documentation. I've tried export HADOOP_ROOT_LOGGER=DEBUG, but I'm not getting any extra output, or am seeing anything unusual in the logs. Perhaps I'm not looking at the right place.

我没有摄入任何数据，也没有在文档之外做任何事情。我尝试过导出hadoop - root_logger =DEBUG，但是我没有得到任何额外的输出，或者看到日志中有任何异常。也许我看错地方了。

The only error I ever see, which I've always seen with older versions of hadoop is:

我所见过的唯一错误，我在旧版本的hadoop上经常看到的错误是:

"Unable to load realm info from SCDynamicStore"

Any ideas on why everything might be so slow? Also, according to my Activity Monitor, I'm barely using any resources.

你知道为什么每件事都这么慢吗?而且，根据我的活动监视器，我几乎没有使用任何资源。

I just noticed this in my namenode log when I run the dfs -ls

我只是在运行dfs -ls时在namenode日志中注意到了这一点

2014-11-25 11:47:24,064 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-11-25 11:47:54,063 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds
2014-11-25 11:47:54,064 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-11-25 11:48:24,064 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
2014-11-25 11:48:24,064 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-11-25 11:48:54,065 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds

Configurations

配置

core-site:

核心位点:

<configuration>
  <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

hdfs-site:

hdfs-site:

<configuration>
   <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>
</configuration>

NameNode

NameNode

1 个解决方案

#1

With nothing else to try, I decided to try removing the "Unable to load realm info from SCDynamicStore" warning. The solution to that was to set HADOOP_OPTS.

没有别的办法，我决定尝试删除“无法从SCDynamicStore加载领域信息”警告。解决方法是设置hadoop op_opts。

Although doing the following did NOT remove the warning, it for some reason FIXED my speed issue.

虽然这样做并没有删除警告，但它出于某些原因修正了我的速度问题。

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK
-Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

If anyone can explain why, I'll definitely accept that answer.

如果有人能解释为什么，我一定会接受这个答案。