1,使用IDEA软件进行开发.
在idea中新建scala project, File-->New-->Project.选择Scala-->Scala
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTh4T0dGbU5tTTBZeTFrWlRNeUxUUmhNREV0WW1RMVpDMW1Oak01WW1WaFpEa3paR1F2TVM1d2JtYz0uanBn.jpg?w=700&webp=1)
2,在编辑窗口中完成WordCount程序的编写,完整程序如下:
object WordCount {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Word Count")
val sc = new SparkContext(conf)
val textFile = sc.textFile("/usr/local/spark-1.4.0/README.md")
val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
val counts = wordCounts.collect()
for(count <- counts) {
println(count._1 + ":" + count._2)
}
}
}
3.本地调试及debug程序:
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTgwT1RkallqRTNZeTA0TUdObExUUmpZVE10WWpneU5pMDNOVEZpWVRRNE56WXdNMkV2TWk1d2JtYz0uanBn.jpg?w=700&webp=1)
如上图所示,选择Main class,在VM options中设置参数 -Dspark.master=local表示以当前机器为master.
4,程序中结果在窗口中打印,如下
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTg0TkRRMk5tWTFNaTB6TXpFM0xUUTJOamN0T0dWaE5DMHhNek5rWWpkbFlXVXlaV0V2TXk1d2JtYz0uanBn.jpg?w=700&webp=1)
5,以spark-submit形式提交程序并运行
5.1,从idea打包导出可运行jar程序
“File”–> “Project Structure” –> “Artifact”,选择“+”–> “Jar” –> “From Modules with dependencies”,选择main函数,并在弹出框中选择输出jar位置,并选择“OK”。
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTg1TVRWallUTm1NUzFsT1RFMkxUUXdZbU10WWpVMk5DMDFOV0prWmpVMU5EWXlOVGd2TkM1d2JtYz0uanBn.jpg?w=700&webp=1)
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTloTWpGaU1EYzBNUzFpTmpVMUxUUXpOR0V0WVdNek1DMDRaV0ZrTmpBNU9XRmhNRFV2TlM1d2JtYz0uanBn.jpg?w=700&webp=1)
点击ok后,build-->Build Artifact,弹出如下窗口
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTgyTmpSaU5XWTFOaTB4WW1FMUxUUXpOelV0WWpnd09DMWpZV016TURCbU1XTXlaak12Tmk1d2JtYz0uanBn.jpg?w=700&webp=1)
选择Rebuild,在桌面上生成SparkTest.jar文件.
5.2,提交SparkTest.jar文件到Spark集群中并运行
本次将jar包从主机提交到虚拟机集群中
(1)master上启动spark,SPARK_HOME/sbin/start-all.sh.可在浏览器中通过:master:8080访问
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTgwWmpkaE5HVTJNeTB3WWpJeExUUXhOR0l0T1daaE5DMDBNREEwWkRnM01tSmpPRFF2Tnk1d2JtYz0uanBn.jpg?w=700&webp=1)
(2)使用spark-submit命令远程提交jar包,更多spark-submit命令参数配置可访问http://spark.apache.org/docs/latest/submitting-applications.html
进入本机的SPARK_HOME目录下,
命令:bin/spark-submit --class WordCount --master spark://master:7077 --executor-memory 4G --total-executor-cores 4 /home/file/SparkJob/SparkTest.jar 1000
任务提交后,
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTh5TlRWaVpUTmpNQzFsTmpFNExUUm1aV1l0WW1FeE9DMHpOV0ZoWVdKaFpqYzNPV0V2T1M1d2JtYz0uanBn.jpg?w=700&webp=1)
此时访问master:8080端口,可以看到正在运行的任务
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTh6TWpKaU1tTmxOaTFsTldGa0xUUXhZV1V0WVdFNE5TMHdZVGMwT1dRd01HUXdNRFV2T0M1d2JtYz0uanBn.jpg?w=700&webp=1)
待到任务运行完成,命令窗口中可以看到运行结果
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTh3TVdZMFl6SXlNeTFtTlRJd0xUUmpNek10WW1VME1DMDJNRGhqTldZellUbGhOV0l2TVRBdWNHNW4uanBn.jpg?w=700&webp=1)
此时再次访问master:8080,可以看到Completed Applications中有一个Word Count的任务已完成
![Spark程序开发-环境搭建-程序编写-Debug调试-项目提交 Spark程序开发-环境搭建-程序编写-Debug调试-项目提交](https://image.shishitao.com:8440/aHR0cHM6Ly9iYnNtYXguaWthZmFuLmNvbS9zdGF0aWMvTDNCeWIzaDVMMmgwZEhCekwyRndjQzU1YVc1NGFXRnVaeTVqYjIwdmMyaGhjbVF2Y3pJeUwzSmxjeTgwTVRSaU5qbGxNeTB4WlRJNExUUTROak10WVdJMU9TMDJNVEZpWTJRd04yUXdNall2TVRFdWNHNW4uanBn.jpg?w=700&webp=1)