一：测试环境搭建规划：
| 主机名称 | IP | 用户 | HDFS | YARN |
| hadoop11 | 192.168.1.101 | hadoop | NameNode,DataNode | NodeManager |
| hadoop12 | 192.168.1.102 | hadoop | DataNode | NodeManager |
| hadoop13 | 192.168.1.103 | hadoop | DataNode,SecondaryNameNode | NodeManager |
| hadoop14 | 192.168.1.104 | hadoop | DataNode | ResourceManager,NodeManager |
二：hadoop安装：
1:新建hadoop用户：
useradd hadoop
passwd hadoop
su hadoop
mkdir /home/hadoop/apps
mkdir /home/hadoop/data
三：关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld
四：关闭selinux：
vim /etc/sysconfig/selinux
修改SELINUX=enforcing为SELINUX=disabled
五：安装Java：
5-1下载JDK：
https://www.oracle.com/technetwork/java/javase/downloads/index.html
https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html
点选 Accept License Agreement
下载jdk-11.0.1_linux-x64_bin.tar.gz
把下载好的jdk-11.0.1_linux-x64_bin.tar.gz，上传到服务器 /usr/local目录
tar -zxvf jdk-11.0.1_linux-x64_bin.tar.gz
5-2添加环境变量：
vim /etc/profile
在结尾添加：
#java
export JAVA_HOME='/usr/local/jdk-11.0.1'
export PATH=$JAVA_HOME/bin:$PATH
保存退出后，执行：
source /etc/profile
查看版本，确定环境变量是否生效：
java -version
六：安装hadoop
6-1下载hadoop：
https://hadoop.apache.org/releases.html
选择binary文件
拷贝镜像地址，下载
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
6-2 解压到指定目录：
tar -zxvf hadoop-2.8.5.tar.gz -C /home/hadoop/apps
6-3修改配置文件：
cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：
#export JAVA_HOME='/usr/local/jdk-11.0.1'
export JAVA_HOME='/usr/local/jdk1.8.0_191'
6-3-1 vim core-site.xml
<!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop11.zm.com:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data/hadoop/tmp</value>
</property>
</configuration>
fs.fefaultFS是NameNode的地址，
hadoop.tmp.dir是hadoop临时目录的地址，
应保证目录是已存在的，如不存在，需先创建。
6-3-2 vim hdfs-site.xml
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hadoop/name</value>
<description>为了保证元数据的安全一般配置多个不同目录</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hadoop/data</value>
<description>datanode 的数据存储目录</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>HDFS 的数据块的副本存储个数, 默认是3</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop13.zm.com:50090</value>
</property>
</configuration>
dfs.namenode.name.dir：namenode数据的存放地点。也就是namenode元数据存放的地方，记录了hdfs系统中文件的元数据。
dfs.datanode.data.dir： datanode数据的存放地点。也就是block块存放的目录了。
dfs.replication：hdfs的副本数设置。也就是上传一个文件，其分割为block块后，每个block的冗余副本个数，默认配置是3。
dfs.secondary.http.address：secondarynamenode 运行节点的信息，和 namenode 不同节点
dfs.namenode.secondary.http-address是指定secondaryNameNode的http访问地址和端口号，因为在规划中，我们将hadoop13规划为SecondaryNameNode服务器。
所以这里设置为：hadoop13.zm.com:50090
6-3-3 配置slave文件,slaves文件是指定HDFS上有哪些DataNode节点：
cd /home/hadoop/apps/hadoop-2.8.5
[hadoop@hadoop11 hadoop-2.8.5]$ vim slaves
hadoop11.zm.com
hadoop12.zm.com
hadoop13.zm.com
hadoop14.zm.com
6-3-4 cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop14.zm.com</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value></value>
</property>
</configuration>
yarn.resourcemanager.hostname这个指定resourcemanager服务器指向hadoop14.zm.com，
yarn.log-aggregation-enable是配置是否启用日志聚集功能，
yarn.log-aggregation.retain-seconds是配置聚集的日志在HDFS上最多保存多长时间。
6-3-5 cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop11.zm.com:</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop11.zm.com:</value>
</property>
</configuration>
mapreduce.framework.name设置mapreduce任务运行在yarn上。
mapreduce.jobhistory.address是设置mapreduce的历史服务器安装在hadoop11机器上。
mapreduce.jobhistory.webapp.address是设置历史服务器的web页面地址和端口号。
七：配置hosts文件：
回到root权限：su
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.201 hadoop11 hadoop11.zm.com 192.168.1.202 hadoop12 hadoop12.zm.com
192.168.1.203 hadoop13 hadoop13.zm.com 192.168.1.204 hadoop14 hadoop14.zm.com
八：SSH免密登陆
su hadoop
所有hadoop集群服务器：
su hadoop
cd /home/hadoop/.ssh
ssh-keygen -t rsa
将所有终端公钥文件拷贝到hadoop11：
所有终端运行：
ssh-copy-id -i hadoop11
登陆hadoop11终端的/home/hadoop/.ssh修改权限：
chmod 600 authorized_keys
分发授权文件到其他终端：
scp /home/hadoop/.ssh/authorized_keys hadoop12:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys hadoop13:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys hadoop14:/home/hadoop/.ssh/
每台机器用ssh命令连接下所有机器，第一次连接要输入yes，以后不用输入
九：hadoop分发到其他终端
拷贝hadoop11 /home/hadoop/apps目录内容到其他终端:
scp -rq data hadoop12:/home/hadoop/
拷贝hadoop11 /home/hadoop/data目录内容到其他终端:
scp -rq data hadoop12:/home/hadoop/
十：配置hadoop环境变量（所有节点均需配置）
注意：
1、如果使用root用户进行安装。 vi /etc/profile 即可 系统变量
2、如果使用普通用户进行安装。 vi ~/.bashrc 用户变量
vim .bashrc
# User specific aliases and functions export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[hadoop@hadoop11 ~]$ source .bashrc
查看hadoop版本：hadoop version
[hadoop@hadoop11 ~]$ hadoop version
Hadoop 2.8.
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0b8464d75227fcee2c6e7f2410377b3d53d3d5f8
Compiled by jdu on --10T03:32Z
Compiled with protoc 2.5.
From source with checksum 9942ca5c745417c14e318835f420733
This command was run using /home/hadoop/apps/hadoop-2.8./share/hadoop/common/hadoop-common-2.8..jar
十一：hadoop初始化
HDFS初始化只能在主节点上进行
如果需要重新初始化，先删除相关目录文件，
粗略做法是删除/home/hadoop/data/hadoop路径下
[hadoop@hadoop14 hadoop]$ ll
总用量
drwx------. hadoop hadoop 10月 : data
drwxrwxr-x. hadoop hadoop 10月 : name
drwxrwxr-x. hadoop hadoop 10月 : tmp
[hadoop@hadoop14 hadoop]$ pwd
/home/hadoop/data/hadoop
这几个目录的子目录；
[hadoop@hadoop11 ~]$ hadoop namenode -format
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--198.8.8.201-
// :: INFO common.Storage: Storage directory /home/hadoop/data/hadoop/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop11/198.8.8.201
************************************************************/
初始化后会自动在/home/hadoop/data/hadoop/name路径
建立current目录
生成：
-rw-rw-r-- hadoop hadoop 10月 : fsimage_0000000000000000000
-rw-rw-r-- hadoop hadoop 10月 : fsimage_0000000000000000000.md5
-rw-rw-r-- hadoop hadoop 10月 : seen_txid
-rw-rw-r-- hadoop hadoop 10月 : VERSION
十二：启动HDFS：
可以在集群中任何节点执行：
[hadoop@hadoop12 ~]$ start-dfs.sh
报错
[hadoop@hadoop12 ~]$ start-dfs.sh
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/hadoop/apps/hadoop-2.8./share/hadoop/common/lib/hadoop-auth-2.8..jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Starting namenodes on [hadoop11.zm.com]
The authenticity of host 'hadoop11.zm.com (198.8.8.201)' can't be established.
ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.
ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.
Are you sure you want to continue connecting (yes/no)? no
关闭HDFS:
stop-dfs.sh
降低java版本到jdk1.8.0_191
如步骤6-3修改集群所有节点配置文件：
cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop
配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：
#export JAVA_HOME='/usr/local/jdk-11.0.1'
export JAVA_HOME='/usr/local/jdk1.8.0_191'
再次启动HDFS（任何节点均可执行）:
start-dfs.sh
[hadoop@hadoop11 sbin]$ start-dfs.sh
Starting namenodes on [hadoop11.zm.com]
hadoop11.zm.com: starting namenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-namenode-hadoop11.out
hadoop11: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop11.out
hadoop13: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop13.out
hadoop14: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop14.out
hadoop12: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop12.out
Starting secondary namenodes [hadoop13.zm.com]
hadoop13.zm.com: starting secondarynamenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-secondarynamenode-hadoop13.out
在其他节点启动时会报询问信息
The authenticity of host 'hadoop13.zm.com (198.8.8.203)' can't be established.
ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.
ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.
Are you sure you want to continue connecting (yes/no)?
解决办法（所有节点执行）：
进入root用户
su
vim /etc/ssh/ssh_config
加入如下两句
# StrictHostKeyChecking ask
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
start-dfs.sh执行后：
在/home/hadoop/data/hadoop/data中自动生成
[hadoop@hadoop11 data]$ ll
总用量
drwxrwxr-x hadoop hadoop 10月 : current
-rw-rw-r-- hadoop hadoop 10月 : in_use.lock
在/home/hadoop/data/hadoop/name中自动生成
[hadoop@hadoop11 name]$ ll
总用量
drwxrwxr-x hadoop hadoop 10月 : current
-rw-rw-r-- hadoop hadoop 10月 : in_use.lock
十三：启动YARN（只能在ResourceManager主节点执行）：
[hadoop@hadoop14 ~]$ start-yarn.sh
[hadoop@hadoop14 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop11.out
hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out
hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out
hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out
hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out
start-yarn.sh执行后：
会在/home/hadoop/data/hadoop/tmp中自动生成
[hadoop@hadoop14 tmp]$ ll
总用量
drwxr-xr-x hadoop hadoop 10月 : nm-local-dir
[hadoop@hadoop14 nm-local-dir]$ ll
总用量
drwxr-xr-x hadoop hadoop 10月 : filecache
drwx------ hadoop hadoop 10月 : nmPrivate
drwxr-xr-x hadoop hadoop 10月 : usercache
此配置启动后，ResourceManager没有启动，无法访问：http://198.8.8.204:8088/
需要在所有节点添加ResourceManager配置：
vim vim yarn-site.xml
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration> <!-- Site specific YARN configuration properties --> <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop14</value>
</property> <property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等。
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等。
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等。
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:</value>
#ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等。
</property> <property>
<name>yarn.resourcemanager.webapp.address</name>
# <value>198.8.8.204:</value>
<value>${yarn.resourcemanager.hostname}:</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value></value>
</property> </configuration>
保存配置，
在ResourceManager主节点hadoop14重启yarn：
stop-yarn.sh
start-yarn.sh
[hadoop@hadoop14 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop14.out
hadoop11: Warning: Permanently added 'hadoop11,198.8.8.201' (ECDSA) to the list of known hosts.
hadoop12: Warning: Permanently added 'hadoop12,198.8.8.202' (ECDSA) to the list of known hosts.
hadoop14: Warning: Permanently added 'hadoop14,198.8.8.204' (ECDSA) to the list of known hosts.
hadoop13: Warning: Permanently added 'hadoop13,198.8.8.203' (ECDSA) to the list of known hosts.
hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out
hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out
hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out
hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out
[hadoop@hadoop14 hadoop]$ jps
ResourceManager
DataNode
NodeManager
Jps
可以访问：http://198.8.8.204:8088/
十四：查看所有节点进程
jps
[hadoop@hadoop11 nm-local-dir]$ jps
NodeManager
DataNode
Jps
NameNode
[hadoop@hadoop12 nm-local-dir]$ jps
NodeManager
DataNode
Jps
[hadoop@hadoop13 nm-local-dir]$ jps
NodeManager
DataNode
Jps
SecondaryNameNode
[hadoop@hadoop14 nm-local-dir]$ jps
DataNode
NodeManager
Jps
十五：HDFS和YARN的WEB管理界面：
http://198.8.8.201:50070
十六：hadoop简单应用：
创建目录
[hadoop@hadoop11 ~]$ hadoop fs -mkdir -p /test/input
查看创建文件
hadoop fs -ls /
新建文件
vim words.txt
hello zhangsan
hello lisi
hello wangwu
上传文件
hadoop fs -put ~/words.txt /test/input
查看是否上传成功
hadoop fs -ls /test/input
下载文件
hadoop fs -get /test/input/words.txt /home/hadoop/
查看是否下载成功
cd /home/hadoop
ll
用wordcount测试：
[hadoop@hadoop11 hadoop]$ hadoop jar /home/hadoop/apps/hadoop-2.8.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /test/input /test/output
// :: INFO client.RMProxy: Connecting to ResourceManager at hadoop14/198.8.8.204:
// :: INFO input.FileInputFormat: Total input files to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541142769004_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1541142769004_0001
// :: INFO mapreduce.Job: The url to track the job: http://hadoop14:8088/proxy/application_1541142769004_0001/
// :: INFO mapreduce.Job: Running job: job_1541142769004_0001
// :: INFO mapreduce.Job: Job job_1541142769004_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1541142769004_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Launched reduce tasks=
Data-local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total time spent by all reduce tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total vcore-milliseconds taken by all reduce tasks=
Total megabyte-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=
Map output records=
Map output bytes=
Map output materialized bytes=
Input split bytes=
Combine input records=
Combine output records=
Reduce input groups=
Reduce shuffle bytes=
Reduce input records=
Reduce output records=
Spilled Records=
Shuffled Maps =
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
BAD_ID=
CONNECTION=
IO_ERROR=
WRONG_LENGTH=
WRONG_MAP=
WRONG_REDUCE=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
查看输出文件：
[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output
[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output
Found items
-rw-r--r-- hadoop supergroup -- : /test/output/_SUCCESS
-rw-r--r-- hadoop supergroup -- : /test/output/part-r-
[hadoop@hadoop11 hadoop]$ hadoop fs -cat /test/output/part-r-
hello
lisi
wangwu
zhangsan
参考文档：
https://www.cnblogs.com/qingyunzong/p/8496127.html
https://blog.csdn.net/hliq5399/article/details/78193113