1. 环境的需求(这里虚拟机下Centos6的安装就不说了)
Centos6 + hadoop-0.21.0.tar
2. 服务器的配置(我在这里均为2个CPU,2G内存,100硬盘)
在这里,服务器IP最好是固定的,也就是说,能相互之间用ping命令ping通的IP
建议在公司开发,因为公司有这个条件,我在这里配置了三台datanode,分别为
Datanode1,Datanode2,Datanode3(这些都是创建虚机的主机名)
服务器名称 IP地址(自己定)
Namenode 192.168.16.1
Datanode1 192.168.16.2
Datanode2 192.168.16.3
Datanode3 192.168.16.4
2.1 修改服务器IP的方法如下:
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
#这里是你网卡的物理地址，通常检测到的网卡你就不用输入了
#打开后该项已经存在,无需修改
HWADDR="00:0C:29:95:1D:A5"
BOOTPROTO="static"
ONBOOT="yes"
#这里是IP地址,不能重复,从253降序排列,被占用了的不能再次使用.
IPADDR=172.16.101.245
NETMASK=255.255.255.0
NETWORK=172.16.101.0
BROADCAST=172.16.101.255
GATEWAY=172.16.101.254
退出保存后,执行如下命令,使设置的网关马上生效:
shell>> ifdown eth0
shell>> ifup eth0
shell>> /etc/init.d/network restart
3. 安装JDK6(我用的是jdk-6u26-linux-x64-rpm.bin)
3.1. 创建安装目录 mkdir /usr/java/
3.2. 把jdk-6u26-linux-x64-rpm.bin移动到/usr/java/下然后执行
./jdk-6u26-linux-x64-rpm.bin
运行中会要求输入，顺序yes和按回车就行.
执行后会看到一个文件夹名为:jdk1.6.0_26
3.3. 设置环境变量
/etc/profile增加如下内容
#config java
JAVA_HOME=/usr/java/jdk1.6.0_26
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
PATH=$JAVA_HOME/bin:$HOME/bin:$PATH
export PATH JAVA_HOME CLASSPATH
3.4. 让设置生效: source /etc/profile
4. 安装ssh服务器和客户端
a. yum search ssh
b. 找到要安装的server(这里拿openssh-server.x86_64)
c. 安装server: yum install openssh-server.x86_64
d. 安装client(这里拿openssh-clients.x86_64)
e. 安装client: yum install openssh-clients.x86_64
5. 设置ssh进行Namenode和Datanode之间无密码访问
a. 用 ssh-key-gen 在本地主机上创建公钥和密钥
[root@Namenode ~]# ssh-keygen -t rsa
Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Pess enter key]
Your identification has been saved in /home/jsmith/.ssh/id_rsa.
Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.
The key fingerprint is: 33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9
root@Namenode
b. 用 ssh-copy-id 把公钥复制到远程主机上
[root@Namenode ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@Datanode1
root@Datanode1's password:
Now try logging into the machine, with ―ssh ?root@Datanode1‘‖, and check in:
.ssh/authorized_keys to make sure we haven‘t added extra keys that you weren‘t expecting.
[注: ssh-copy-id 把密钥追加到远程主机的 .ssh/authorized_key 上.]
c. 直接登录远程主机
[root@Namenode ~]# ssh Datanode1
Last login: Sun Nov 16 17:22:33 2008 from 192.168.1.2
[注: SSH 不会询问密码.]
[root@Datanode1 ~]
[注: 你现在已经登录到了远程主机上]
d. 注意:在这里,执行都在Namenode上面,而且Namenode也需要对自己进行无密码操作即
[root@Namenode ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@Namenode操作,
其他的,按照a-c重复操作Datanode2和Datanode3就行了
一定要能无密码访问,否则不能集群Hadoop一定失败.
6. 安装Hadoop(这里,每一台服务器的JDK和Hadoop安装路径都相同)
a. 创建安装目录 mkdir /usr/local/hadoop/
b. 解压安装文件hadoop-0.21.0.tar放入到安装目录
tar -zxvf hadoop-0.21.0.tar
c. 设置环境变量
/etc/profile增加如下内容
#config hadoop
export HADOOP_HOME=/usr/local/hadoop/
export PATH=$HADOOP_HOME/bin:$PATH
#hadoop的日志文件路径的配置
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
让设置生效: source /etc/profile
d. 设置主从配置
Namenode中/etc/hosts的配置如下:
192.168.16.1 Namenode
192.168.16.2 Datanode1
192.168.16.3 Datanode2
192.168.16.4 Datanode3
Namenode中/usr/local/hadoop/conf/masters的配置如下:
Namenode
Namenode中/usr/local/hadoop/conf/slaves的配置如下:
Datanode1
Datanode2
Datanode3
Datanode1中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)
192.168.16.1 Namenode
192.168.16.2 Datanode1
Datanode2中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)
192.168.16.1 Namenode
192.168.16.3 Datanode2
Datanode3中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)
192.168.16.1 Namenode
192.168.16.4 Datanode3
e. 修改配置文件/usr/local/hadoop/conf/ hadoop-env.sh
把JAVA_HOME该为安装jdk的路径
# The java implementation to use. Required.
export JAVA_HOME=/usr/java/jdk1.6.0_26/
f. 修改配置文件 core-site.xml内容如下:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://Namenode:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp/</value>
</property>
</configuration>
g. 修改配置文件 hdfs-site.xml内容如下:
<configuration>
<property>
<name>dfs.replication</name>
#设置备份文件数
<value>1</value>
</property>
</configuration>
h. 修改配置文件 mapred-site.xml内容如下:
<configuration>
<property>
<name>mapred.job.tracker</name>
#一般jobtracker和namenode设置到同一台机器上,但是同样可以集群
<value>Namenode:9001</value>
</property>
</configuration>
i. 注意:上面讲的配置文件全部是在Namenode中配置的,只要把这三个配置文件拷贝复制到其他的Datanode上就行了
j. 初始化Hadoop: cd /usr/local/hadoop/
./bin/hadoop namenode -format
出现类似如下的信息:但是不能出现ERORR字段.
.2.jar:/usr/local/hadoop/bin/../lib/paranamer-generator-2.2.jar:/usr/local/hadoop/bin/../lib/qdox-1.10.1.jar:/usr/local/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hadoop/bin/../lib/slf4j-api-1.5.11.jar:/usr/local/hadoop/bin/../lib/slf4j-log4j12-1.5.11.jar:/usr/local/hadoop/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/bin/../lib/jsp-2.1/*.jar:/usr/local/hadoop/hdfs/bin/../conf:/usr/local/hadoop/hdfs/bin/../hadoop-hdfs-*.jar:/usr/local/hadoop/hdfs/bin/../lib/*.jar:/usr/local/hadoop/bin/../mapred/conf:/usr/local/hadoop/bin/../mapred/hadoop-mapred-*.jar:/usr/local/hadoop/bin/../mapred/lib/*.jar:/usr/local/hadoop/hdfs/bin/../hadoop-hdfs-*.jar:/usr/local/hadoop/hdfs/bin/../lib/*.jar
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010
************************************************************/
Re-format filesystem in /usr/local/hadoop/tmp/dfs/name ? (Y or N) y
Format aborted in /usr/local/hadoop/tmp/dfs/name
11/06/16 13:04:17 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at namenode/172.16.101.251
************************************************************/
k. 启动Hadoop ./bin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh
starting namenode, logging to /usr/local/hadoop//logs/hadoop-root-namenode-namenode.out
datanode1: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode1.out
datanode2: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode2.out
datanode3: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode3.out
namenode: starting secondarynamenode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-secondarynamenode-namenode.out
starting jobtracker, logging to /usr/local/hadoop//logs/hadoop-root-jobtracker-namenode.out
datanode3: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode3.out
datanode2: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode2.out
datanode1: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode1.out
启动后用命令JPS查看结果如下:
[root@namenode hadoop]# jps
1806 Jps
1368 NameNode
1694 JobTracker
1587 SecondaryNameNode
然后到Datanode1/2/3上去查看,执行JPS,结果如下:
[root@datanode2 hadoop]# jps
1440 Jps
1382 TaskTracker
1303 DataNode
[root@datanode2 hadoop]# jps
1382 TaskTracker
1303 DataNode
1452 Jps
说明你成功集群安装了Hadoop
7. HDFS操作
运行bin/目录的hadoop命令，可以查看Haoop所有支持的操作及其用法，这里以几个简单的操作为例。
建立目录
[root@namenode hadoop]# ./bin/hadoop dfs -mkdir testdir
在HDFS中建立一个名为testdir的目录
复制文件
[root@namenode hadoop]# ./bin/hadoop dfs -put /home/dbrg/large.zip testfile.zip
把本地文件large.zip拷贝到HDFS的根目录/user/dbrg/下，文件名为testfile.zip
查看现有文件
[root@namenode hadoop]# ./bin/hadoop dfs -ls