Linux下Eclipse配置scala开发Spark-WordCount项目

时间:2024-03-27 16:11:04

Eclipse和Scala安装见:https://blog.csdn.net/qq_25948717/article/details/80404158‘

                                       https://blog.csdn.net/qq_25948717/article/details/80758713

去官网下载插件:http://scala-ide.org/download/prev-stable.html

解压后,把其features和plugins下的所有jar包都copy到eclipse下对应的features和plugins目录下

Linux下Eclipse配置scala开发Spark-WordCount项目

进入到eclipse安装目录:../eclipse/eclipse启动eclipse

Linux下Eclipse配置scala开发Spark-WordCount项目

Eclipse打开之后,从file-->new-->others-->Scala Wizards-->Scala Project新建一个project,我命名为SparkWordCount

右击SparkWordCount,-->properties-->Java Build Path-->Libraries-->Add External Jars,选择spark安装目录下lib/的assembly jar包

创建class:

Linux下Eclipse配置scala开发Spark-WordCount项目

Linux下Eclipse配置scala开发Spark-WordCount项目

package spark
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
class wordcount {
    def main(args: Array[String]) {
    val infile = "/public/users/yexin/SparkWorks/data/test.txt" // Should be some file on your system
    val conf = new SparkConf().setAppName("word count")
    val sc = new SparkContext(conf)
    val indata = sc.textFile(infile, 2).cache()
    val words = indata.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey((a,b) => (a+b))
    words.saveAsTextFile("/public/users/yexin/SparkWorks/data/output")
    println("All words are counted!")
  }
}