spark集群安装部署

安装部署环境：

首先免密钥配置

  ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

  cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

将本机的id_dsa.pub内容追加到其他机器的authorized_keys中

 

一、安装hadoop

1.将hadoop包进行解压缩：tar -zxvf hadoop-2.5.0-cdh5.3.6.tar.gz

2.配置环境变量vi /etc/profile

export JAVA_HOME=/usr/java/default

export JRE_HOME=/usr/java/default/jre

export HADOOP_HOME=/home/trs/hadoop

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

3.配置文件生效 source /etc/profile

4.修改hdfs-site.xml

<configuration>

<property>

   <name>dfs.replication</name>

     <value>1</value>

      </property>

 <property>

   <name>dfs.namenode.secondary.http-address</name>

     <value>mesos-slave-7:50090</value>

      </property>

 <property>

   <name>dfs.name.dir</name>

     <value>/home/trs/hadoop/hdfs/name</value>

      </property>

 <property>

    <name>dfs.data.dir</name>

      <value>/home/trs/hadoop/hdfs/data</value>

         </property>

</configuration>

5.修改mapred-site.xml

<configuration>

 <property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>mesos-slave-5:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>mesos-slave-5:19888</value>

</property>

</configuration>

6.修改yarn-site.xml

<configuration>

 

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value><!--在NM上自定义shuffle服务-->

</property>

<property>

      <name>yarn.app.mapreduce.am.env</name>

            <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>

              </property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value><!--shuffle服务的实现-->

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>mesos-slave-5:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>mesos-slave-5:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>mesos-slave-5:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>mesos-slave-5:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>mesos-slave-5:8088</value>

</property>

</configuration>

7.修改slaves文件

mesos-slave-7 其他机器名

mesos-slave-5

8.配置hadoop-env.sh

export JAVA_HOME=/usr/java/default  jdk的路径

9.将配置好的文件拷贝到其他机器

  10.bin/hdfs namenode -format

   sbin/start-dfs.sh

  11.访问http://ip:50070

二、spark安装部署

1.安装Scala并配置环境变量

2.  #进入spark配置目录

cp spark-env.sh.template spark-env.sh  #从配置模板复制

vim spark-env.sh    #添加配置内容

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export SPARK_MASTER_IP=mesos-slave-6

export SPARK_EXECUTOR_MEMORY=8g

export SPARK_WORKER_MEMORY=16g

export SPARK_DRIVER_MEMORY=8g

export SPARK_WORKER_CORES=4

export JAVA_HOME=/usr/java/default

export JRE_HOME=/usr/java/default/jre

export HADOOP_HOME=/home/trs/hadoop

export SCALA_HOME=/usr/scala

3.vi slaves 配置从节点

mesos-slave-5

mesos-slave-7

4.启动spark  命令：start-all.sh

5.访问地址：http://ip:8080