Spark进阶之路-Standalone模式搭建
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.Spark的集群的准备环境
1>.master节点信息（s101）
2>.worker节点信息（s102)
3>.worker节点信息（s103)
4>.worker节点信息（s104)
二.Spark的Standalone模式搭建
1>.下载Spark安装包
Spark下载地址：https://archive.apache.org/dist/spark/
[yinzhengjie@s101 download]$ sudo yum -y install wget
[sudo] password for yinzhengjie:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 :1.14-.el7_4. will be installed
--> Finished Dependency Resolution Dependencies Resolved =====================================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================================
Installing:
wget x86_64 1.14-.el7_4. base k Transaction Summary
=====================================================================================================================================================================
Install Package Total download size: k
Installed size: 2.0 M
Downloading packages:
wget-1.14-.el7_4..x86_64.rpm | kB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : wget-1.14-.el7_4..x86_64 /
Verifying : wget-1.14-.el7_4..x86_64 / Installed:
wget.x86_64 :1.14-.el7_4. Complete!
[yinzhengjie@s101 download]$
安装wget软件包（[yinzhengjie@s101 download]$ sudo yum -y install wget）
[yinzhengjie@s101 download]$ wget https://archive.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-hadoop2.7.tgz #下载你想要下载的版本
2>.解压配置文件
[yinzhengjie@s101 download]$ ll
total
-rw-r--r--. yinzhengjie yinzhengjie Aug hadoop-2.7..tar.gz
-rw-r--r--. yinzhengjie yinzhengjie May jdk-8u131-linux-x64.tar.gz
-rw-r--r--. yinzhengjie yinzhengjie Jul spark-2.1.-bin-hadoop2..tgz
-rw-r--r--. yinzhengjie yinzhengjie Jun : zookeeper-3.4..tar.gz
[yinzhengjie@s101 download]$
[yinzhengjie@s101 download]$ tar -xf spark-2.1.-bin-hadoop2..tgz -C /soft/ #加压Spark安装包到指定目录
[yinzhengjie@s101 download]$ ll /soft/
total
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : hadoop -> /soft/hadoop-2.7./
drwxr-xr-x. yinzhengjie yinzhengjie Aug : hadoop-2.7.
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : jdk -> /soft/jdk1..0_131/
drwxr-xr-x. yinzhengjie yinzhengjie Mar jdk1..0_131
drwxr-xr-x. yinzhengjie yinzhengjie Apr spark-2.1.-bin-hadoop2.
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : zk -> /soft/zookeeper-3.4./
drwxr-xr-x. yinzhengjie yinzhengjie Mar : zookeeper-3.4.
[yinzhengjie@s101 download]$ ll /soft/spark-2.1.-bin-hadoop2./ #查看目录结构
total
drwxr-xr-x. yinzhengjie yinzhengjie Apr bin
drwxr-xr-x. yinzhengjie yinzhengjie Apr conf
drwxr-xr-x. yinzhengjie yinzhengjie Apr data
drwxr-xr-x. yinzhengjie yinzhengjie Apr examples
drwxr-xr-x. yinzhengjie yinzhengjie Apr jars
-rw-r--r--. yinzhengjie yinzhengjie Apr LICENSE
drwxr-xr-x. yinzhengjie yinzhengjie Apr licenses
-rw-r--r--. yinzhengjie yinzhengjie Apr NOTICE
drwxr-xr-x. yinzhengjie yinzhengjie Apr python
drwxr-xr-x. yinzhengjie yinzhengjie Apr R
-rw-r--r--. yinzhengjie yinzhengjie Apr README.md
-rw-r--r--. yinzhengjie yinzhengjie Apr RELEASE
drwxr-xr-x. yinzhengjie yinzhengjie Apr sbin
drwxr-xr-x. yinzhengjie yinzhengjie Apr yarn
[yinzhengjie@s101 download]$
3>.编辑slaves配置文件，将worker的节点主机名输入，默认是localhost
[yinzhengjie@s101 download]$ cd /soft/spark-2.1.-bin-hadoop2./conf/
[yinzhengjie@s101 conf]$ ll
total
-rw-r--r--. yinzhengjie yinzhengjie Apr docker.properties.template
-rw-r--r--. yinzhengjie yinzhengjie Apr fairscheduler.xml.template
-rw-r--r--. yinzhengjie yinzhengjie Apr log4j.properties.template
-rw-r--r--. yinzhengjie yinzhengjie Apr metrics.properties.template
-rw-r--r--. yinzhengjie yinzhengjie Apr slaves.template
-rw-r--r--. yinzhengjie yinzhengjie Apr spark-defaults.conf.template
-rwxr-xr-x. yinzhengjie yinzhengjie Apr spark-env.sh.template
[yinzhengjie@s101 conf]$ cp slaves.template slaves
[yinzhengjie@s101 conf]$ vi slaves
[yinzhengjie@s101 conf]$ cat slaves
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# # A Spark Worker will be started on each of the machines listed below.
s102
s103
s104
[yinzhengjie@s101 conf]$
4>.编辑spark-env.sh文件，指定master节点和端口号
[yinzhengjie@s101 ~]$ cp /soft/spark/conf/spark-env.sh.template /soft/spark/conf/spark-env.sh
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ echo export JAVA_HOME=/soft/jdk >> /soft/spark/conf/spark-env.sh
[yinzhengjie@s101 ~]$ echo SPARK_MASTER_HOST=s101 >> /soft/spark/conf/spark-env.sh
[yinzhengjie@s101 ~]$ echo SPARK_MASTER_PORT= >> /soft/spark/conf/spark-env.sh
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ grep -v ^# /soft/spark/conf/spark-env.sh | grep -v ^$
export JAVA_HOME=/soft/jdk
SPARK_MASTER_HOST=s101
SPARK_MASTER_PORT=
[yinzhengjie@s101 ~]$
5>.将s101的spark配置信息分发到worker节点
[yinzhengjie@s101 ~]$ more `which xrsync.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com #判断用户是否传参
if [ $# -lt ];then
echo "请输入参数";
exit
fi #获取文件路径
file=$@ #获取子路径
filename=`basename $file` #获取父路径
dirpath=`dirname $file` #获取完整路径
cd $dirpath
fullpath=`pwd -P` #同步文件到DataNode
for (( i=;i<=;i++ ))
do
#使终端变绿色
tput setaf
echo =========== s$i %file ===========
#使终端变回原来的颜色，即白灰色
tput setaf
#远程执行命令
rsync -lr $filename `whoami`@s$i:$fullpath
#判断命令是否执行成功
if [ $? == ];then
echo "命令执行成功"
fi
done
[yinzhengjie@s101 ~]$
需要配置无秘钥登录，之后执行启动脚本进行同步（[yinzhengjie@s101 ~]$ more `which xrsync.sh`）
关于配置无秘钥登录请参考我之间的笔记：https://www.cnblogs.com/yinzhengjie/p/9065191.html。配置好无秘钥登录后，直接执行上面的脚本进行同步数据。
[yinzhengjie@s101 ~]$ xrsync.sh /soft/spark-2.1.-bin-hadoop2./
=========== s102 %file ===========
命令执行成功
=========== s103 %file ===========
命令执行成功
=========== s104 %file ===========
命令执行成功
[yinzhengjie@s101 ~]$
6>.修改配置文件，将spark运行脚本添加至系统环境变量
[yinzhengjie@s101 ~]$ ln -s /soft/spark-2.1.-bin-hadoop2./ /soft/spark #这里做一个软连接，方便简写目录名称
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ sudo vi /etc/profile #修改系统环境变量的配置文件
[sudo] password for yinzhengjie:
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ tail - /etc/profile
#ADD SPARK_PATH by yinzhengjie
export SPARK_HOME=/soft/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ source /etc/profile #重写加载系统配置文件，使其变量在当前shell生效。
[yinzhengjie@s101 ~]$
7>.启动spark集群
[yinzhengjie@s101 ~]$ more `which xcall.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com #判断用户是否传参
if [ $# -lt ];then
echo "请输入参数"
exit
fi #获取用户输入的命令
cmd=$@ for (( i=;i<=;i++ ))
do
#使终端变绿色
tput setaf
echo ============= s$i $cmd ============
#使终端变回原来的颜色，即白灰色
tput setaf
#远程执行命令
ssh s$i $cmd
#判断命令是否执行成功
if [ $? == ];then
echo "命令执行成功"
fi
done
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ more `which xcall.sh`
[yinzhengjie@s101 ~]$ /soft/spark/sbin/start-all.sh #启动spark集群
starting org.apache.spark.deploy.master.Master, logging to /soft/spark/logs/spark-yinzhengjie-org.apache.spark.deploy.master.Master--s101.out
s102: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-yinzhengjie-org.apache.spark.deploy.worker.Worker--s102.out
s103: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-yinzhengjie-org.apache.spark.deploy.worker.Worker--s103.out
s104: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/logs/spark-yinzhengjie-org.apache.spark.deploy.worker.Worker--s104.out
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ xcall.sh jps #查看进程master和slave节点是否起来了
============= s101 jps ============
Jps
Master
命令执行成功
============= s102 jps ============
Jps
Worker
命令执行成功
============= s103 jps ============
Jps
Worker
命令执行成功
============= s104 jps ============
Jps
Worker
命令执行成功
[yinzhengjie@s101 ~]$
8>.检查Spark的webUI界面
9>.启动spark-shell
三.在Spark集群中执行Wordcount
1>.链接到master集群（[yinzhengjie@s101 ~]$ spark-shell --master spark://s101:7077）
2>.登录webUI，查看正在运行的APP
3>.查看应用细节
4>.查看job的信息
5>.查看stage
6>.查看具体的详细信息
7>.退出spark-shell
8>.查看spark的完成应用，发现日志没了？
那么问题来了。如果看日志呢？详情请参考：https://www.cnblogs.com/yinzhengjie/p/9410989.html。