ubuntu安装hadoop1.x(伪分布式)

时间:2022-10-20 06:06:42

参考 kinglau - 二、Ubuntu14.04下安装Hadoop2.4.0 (伪分布模式)

接上篇 ubuntu安装hadoop(单机)

1. 配置core-site.xml

# hadoop1.x.x 的配置文件都在 $HADOOP_HOME/conf/下,hadoop2.x.x 的配置文件都在 $HADOOP_HOME/etc/hadoop/下
# 本文将都已hadoop1.x.x为例,hadoop2.x.x配置对应文件
$ sudo gedit /usr/local/hadoop/conf/core-site.xml
# add
  <configuration>
  <properties
>
  <name
>fs.default.name</name>
  <value
>hdfs://localhost:9000</value>
  <
/properties>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
        <description>临时目录,其他临时目录的父目录</description>
        <final>true</final>
    </property>
  </configuration>

2. 配置 hdfs-site.xml

<configuration>
<property
>
<name
>dfs.data.dir</name>
<value
>/usr/local/hadoop/tmp/dfs/data</value>
<description
>data node的数据目录</description>
<final
>true</final>
<
/property>
<property
>
<name
>dfs.name.dir</name>
<value
>/usr/local/hadoop/tmp/dfs/name</value>
<description
>name node的元数据</description>
<final
>true</final>
<
/property>
<property
>
<name
>dfs.replication</name>
<value
>1</value>
<description
>文件复制的副本数</description>
<
/property>
<
/configuration>

3. 格式化namenode

$ /usr/local/hadoop/bin/hadoop namenode -format
# when be asked "Re-format filesystem in /usr/local/hadoop/tmp/dfs/namenode ? (Y or N)" input Y
# when you see something like "Storage directory /usr/local/hadoop-1.2.1/tmp/dfs/name has been successfully formatted." ,you succeed.

4. 启动hadoop

$ /usr/local/hadoop/bin/start-dfs.sh
# starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-erbin-ubuntu.out
#
localhost: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-erbin-ubuntu.out
#
localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-erbin-ubuntu.out
$ jps -m
# 14841 DataNode
#
14983 SecondaryNameNode
#
15073 Jps -m
#
14701 NameNode

5. WordCount测试

$ cd /usr/local/hadoop
## 在hdfs创建文件夹
$ bin/hadoop dfs -mkdir /user
$ bin/hadoop dfs -ls /
#drwxr-xr-x - hadoop supergroup 0 2015-04-24 15:08 /user
$ bin/hadoop dfs -mkdir /user/hadoop
## hadoop会默认将/user/hadoop作为home
$ bin/hadoop dfs -mkdir input
$ bin/hadoop dfs -ls
#Found 1 items
#
drwxr-xr-x - hadoop supergroup 0 2015-04-24 15:12 /user/hadoop/input
#
# 上传本地文件值hdfs
#
# hadoop dfs -put [local] [hdfs]
$ bin/hadoop dfs -put input input
$ bin/hadoop dfs -ls input
#Found 7 items
#
-rw-r--r-- 1 hadoop supergroup 7457 2015-04-24 15:12 /user/hadoop/input/capacity-scheduler.xml
#
-rw-r--r-- 1 hadoop supergroup 178 2015-04-24 15:12 /user/hadoop/input/core-site.xml
#
-rw-r--r-- 1 hadoop supergroup 327 2015-04-24 15:12 /user/hadoop/input/fair-scheduler.xml
#
-rw-r--r-- 1 hadoop supergroup 4644 2015-04-24 15:12 /user/hadoop/input/hadoop-policy.xml
#
-rw-r--r-- 1 hadoop supergroup 178 2015-04-24 15:12 /user/hadoop/input/hdfs-site.xml
#
-rw-r--r-- 1 hadoop supergroup 2033 2015-04-24 15:12 /user/hadoop/input/mapred-queue-acls.xml
#
-rw-r--r-- 1 hadoop supergroup 178 2015-04-24 15:12 /user/hadoop/input/mapred-site.xml
$ bin/hadoop jar hadoop-examples-1.x.x.jar wordcount input output
$ bin/hadoop dfs -ls output
#Found 2 items
#
-rw-r--r-- 1 hadoop supergroup 0 2015-04-24 15:25 /user/hadoop/output/_SUCCESS
#
-rw-r--r-- 1 hadoop supergroup 6289 2015-04-24 15:25 /user/hadoop/output/part-r-00000
$ bin/hadoop dfs -cat output/*

6. 关闭hadoop

$ /usr/local/hadoop-1.2.1/bin/stop-dfs.sh 
# stopping namenode
# localhost: stopping datanode
# localhost: stopping secondarynamenode
$ jps -m
# 15317 Jps -m

7. 备注

有些文章,在配置core-site.xml,hdfs-site.xml时,里面的文件路径前总是会添加' file:/ '或' file:/// '.这种写法在2.x是可以的,但是在1.x是不可以的

测试过发现不加file前缀的路径写法,在1.x,
2.x都是可以通过的。

如果dfs.name.dir和dfs.data.dir文件夹是实现创建的。dfs.data.dir的文件权限可能会报错,如果报错,就按错误提示更改一下文件夹的权限即可。