Centos 7 配置单机Hadoop
2018年10月11日 09:48:13 GT_Stone 阅读数:82
系统镜像:CentuOS-7-x86_64-Everything-1708
Java环境:JDK-8u181-linux-x64.tar.gz
Hadoop版本:hadoop-2.7.6.tar.gz
1、安装配置JDK,配置环境变量(路径很重要,我的路径是:/usr/local/java/jdk1.8.0_181)
java环境的具体安装过程不做记录,下载好相应的压缩包,解压至你想要的路径就好
环境变量如下::
-
export JAVA_HOME=/usr/local/java/jdk1.8.0_181
-
export JRE_HOME=${JAVA_HOME}/jre
-
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
-
export PATH=${JAVA_HOME}/bin:$PATH
2、下载hadoop
http://mirror.bit.edu.cn/apache/hadoop/common/
3、解压到/opt目录下
tar -zxvf hadoop-2.7.6.tar.gz -C /opt/
4、配置hadoop环境变量:
-
vim /etc/profile
-
export HADOOP_HOME=/opt/hadoop-2.7.6
-
export PATH=$PATH:$HADOOP_HOME/bin
-
source /etc/profile
5、配置hadoop
5.1 配置hadoop-env.sh
-
vim /opt/hadoop-2.7.6/etc/hadoop/hadoop-env.sh
-
export JAVA_HOME=/usr/local/java/jdk1.8.0_181
5.2 配置core-site.xml
-
vim /opt/hadoop-2.7.6/etc/hadoop/core-site.xml
-
<configuration>
-
<property>
-
<name>hadoop.tmp.dir</name>
-
<value>file:///opt/hadoop-2.7.6</value>
-
<description>Abase for other temporary directories.</description>
-
</property>
-
<property>
-
<name>fs.defaultFS</name>
-
<value>hdfs://192.168.139.129:8888</value>
-
</property>
-
</configuration>
5.3 配置 hdfs-site.xml
vim /opt/hadoop-2.7.6/etc/hadoop/hdfs-site.xml
-
<configuration>
-
<property>
-
<name>dfs.replication</name>
-
<value>1</value>
-
</property>
-
<property>
-
<name>dfs.namenode.name.dir</name>
-
<value>file:///opt/hadoop-2.7.6/tmp/dfs/name</value>
-
</property>
-
<property>
-
<name>dfs.datanode.data.dir</name>
-
<value>file:///opt/hadoop-2.7.6/tmp/dfs/data</value>
-
</property>
-
</configuration>
6、配置ssh免密码登录
-
ssh-keygen -t rsa
-
cd ~/.ssh
-
cat id_rsa.pub>>authorized_keys
-
#如果不能绵密登录,执行以下命令
-
chmod 710 authorized_keys
7、启动
7.1 格式化HDFS
hdfs namenode -format
7.2 启动
./sbin/start-dfs.sh
7.3 测试 http://192.168.139.129:50070
如果访问不成功,请关闭linux系统防火墙,或者开放端口
8、配置yarn
8.1 配置mapred-site.xml
-
cd /opt/hadoop-2.7.6/etc/hadoop/
-
cp mapred-site.xml.template mapred-site.xml
-
vim mapred-site.xml
-
<configuration>
-
<!-- 通知框架MR使用YARN -->
-
<property>
-
<name>mapreduce.framework.name</name>
-
<value>yarn</value>
-
</property>
-
</configuration>
8.2 配置yarn-site.xml
-
vim yarn-site.xml
-
<configuration>
-
<!-- reducer取数据的方式是mapreduce_shuffle -->
-
<property>
-
<name>yarn.nodemanager.aux-services</name>
-
<value>mapreduce_shuffle</value>
-
</property>
-
</configuration>
8.3启动
-
cd /opt/hadoop-2.7.6
-
./sbin/start-yarn.sh
8.4 测试 http://192.168.139.129:8088