基于hadoop的Spark环境搭建

时间：2022-09-25 07:45:33


一、Scala安装

  1. 下载scala安装包，地址：http://www.scala-lang.org/download/

  2. 配置环境变量 
    下载完成后，解压到指定的目录下，在/etc/profile文件中配置环境变量：

    <code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SCALA_HOME=/usr/local/jiang/scala-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.10</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SCALA_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

  3. 验证scala的安装

    <code class="hljs d has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">@logsrv03</span> etc]# scala -<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">version</span>
    Scala code runner <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">version</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.10</span>.6 -- Copyright <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2002</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2013</span>, LAMP/EPFL</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

二、安装spark

  1. 下载spark安装包，地址：http://spark.apache.org/downloads.html （选择without-hadoop版本）

  2. 下载完成后，拷贝到指定的目录下：/usr/local/jiang/，然后解压：

    <code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar <span class="hljs-attribute" style="box-sizing: border-box;">-zxvf</span> spark-1.6.2-bin-without-hadoop</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

  3. 解压后在配置中添加java环境变量、scala环境变量、hadoop环境变量等conf/spark-env.sh

    <code class="hljs http has-numbering" style="display: block; padding: 0px; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background-image: initial; background-attachment: initial; background-color: transparent; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;"><span style="color:rgba(0, 0, 0, 0);">
    </span></code><ol start="1" class="dp-c" style="padding: 0px; border: none; border-image-source: initial; border-image-slice: initial; border-image-width: initial; border-image-outset: initial; border-image-repeat: initial; font-family: Consolas, 'Courier New', Courier, mono, serif; line-height: 26px; margin: 0px 0px 1px 45px !important;"><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;">###jdk安装目录</span><span style="margin: 0px; padding: 0px; border: none; background-color: inherit;">  </span></span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;">export JAVA_HOME=/usr/local/java/jdk1.8.0_92  </span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;"></span> </span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;">###scala安装目录</span><span style="margin: 0px; padding: 0px; border: none; background-color: inherit;">  </span></span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;">export SCALA_HOME=/home/scala/scala-2.11.8  </span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;"></span> </span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;">###spark集群的master节点的ip</span><span style="margin: 0px; padding: 0px; border: none; background-color: inherit;">  </span></span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;">export SPARK_MASTER_IP=192.168.1.168  </span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;"></span> </span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;">###指定的worker节点能够最大分配给Excutors的内存大小</span><span style="margin: 0px; padding: 0px; border: none; background-color: inherit;">  </span></span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;">export SPARK_WORKER_MEMORY=2g  </span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;"></span> </span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"><span class="preprocessor" style="margin: 0px; padding: 0px; border: none; color: gray; background-color: inherit;">###hadoop集群的配置文件目录</span><span style="margin: 0px; padding: 0px; border: none; background-color: inherit;">  </span></span></li><li style="color: rgb(92, 92, 92); border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;">export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.7.2/etc/hadoop  </span></li><li class="alt" style="color: inherit; border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; color: black; background-color: inherit;"> </span></li><li class="alt" style="border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important;"><span style="margin: 0px; padding: 0px; border: none; border-image-source: initial; border-image-slice: initial; border-image-width: initial; border-image-outset: initial; border-image-repeat: initial; background-color: inherit;"><span style="color:#ff0000;">#指定与现有的hadoop库的关联 </span></span></li><li style="border-style: none none none solid; border-left-width: 3px; border-left-color: rgb(108, 226, 108); list-style: decimal-leading-zero outside; line-height: 18px; margin: 0px !important; padding: 0px 3px 0px 10px !important; background-color: rgb(248, 248, 248);"><span style="margin: 0px; padding: 0px; border: none; border-image-source: initial; border-image-slice: initial; border-image-width: initial; border-image-outset: initial; border-image-repeat: initial; background-color: inherit;"><span style="color:#ff0000;">export SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop-2.7.2/bin/hadoop classpath)  </span></span></li></ol>

  5. 配置从机conf/slaves

    <code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">logsrv02
    logsrv04</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
  6. <ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;"></li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

  7. 启动集群

    <code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">sbin/start-all.sh </span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

  8. 启动完成后，查看主从机的进程： 
    主机：

    <code class="hljs ruby has-numbering" style="display: block; padding: 0px; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background-image: initial; background-attachment: initial; background-color: transparent; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;"><span style="color:rgba(0, 0, 0, 0);">[root</span><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@logsrv03</span><span style="color:rgba(0, 0, 0, 0);"> spark-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.6</span><span style="color:rgba(0, 0, 0, 0);">.</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span><span style="color:rgba(0, 0, 0, 0);">-bin-hadoop2.</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span><span style="color:rgba(0, 0, 0, 0);">]</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># jps </span><span style="color:rgba(0, 0, 0, 0);">
    </span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25325</span><span style="color:rgba(0, 0, 0, 0);"> </span><span class="hljs-constant" style="color: inherit; box-sizing: border-box;">NameNode</span><span style="color:rgba(0, 0, 0, 0);">
    </span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23973</span><span style="color:rgba(0, 0, 0, 0);"> </span><span class="hljs-constant" style="box-sizing: border-box;"><span style="color:#ff0000;">Master</span></span><span style="color:rgba(0, 0, 0, 0);">
    </span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17643</span><span style="color:rgba(0, 0, 0, 0);"> </span><span class="hljs-constant" style="color: inherit; box-sizing: border-box;">ResourceManager</span><span style="color:rgba(0, 0, 0, 0);">
    </span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25523</span><span style="color:rgba(0, 0, 0, 0);"> </span><span class="hljs-constant" style="color: inherit; box-sizing: border-box;">SecondaryNameNode</span><span style="color:rgba(0, 0, 0, 0);">
    </span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">28839</span><span style="color:rgba(0, 0, 0, 0);"> </span><span class="hljs-constant" style="color: inherit; box-sizing: border-box;">Jps</span></code><ul class="pre-numbering" style="color: rgb(51, 51, 51); box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

    从机：

    <code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box; background-color: transparent;">@logsrv02</span> spark-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">1.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">0</span>-bin-hadoop2.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">6</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box; background-color: transparent;"># jps</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">744</span><span style="background-color: rgb(255, 0, 0);"> <span class="hljs-constant" style="box-sizing: border-box;">Worker</span></span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">4406</span> <span class="hljs-constant" style="box-sizing: border-box; background-color: transparent;">Jps</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">2057</span> <span class="hljs-constant" style="box-sizing: border-box; background-color: transparent;">DataNode</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box; background-color: transparent;">2170</span> <span class="hljs-constant" style="box-sizing: border-box; background-color: transparent;">NodeManager</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

    三、安装完成后，可以查看spark的UI： 
    基于hadoop的Spark环境搭建

  9. 也可以bin中执行一些验证spark小实验

    <span class="pln">./bin/run-example SparkPi 2>&1 | grep </span><span class="str">"Pi is roughly"</span>
    基于hadoop的Spark环境搭建

<ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

运行后spark的UI： 
基于hadoop的Spark环境搭建
基于hadoop的Spark环境搭建

标签：

相关文章