sparkStreaming 读kafka的数据

目标：sparkStreaming每2s中读取一次kafka中的数据，进行单词计数。

topic:topic1

broker list:192.168.1.126:9092,192.168.1.127:9092,192.168.1.128:9092

1、首先往一个topic中实时生产数据。

　　代码如下： 代码功能：每秒向topic1发送一条消息，一条消息里包含4个单词，单词之间用空格隔开。

 

package kafkaProducer

import java.util.HashMap

import org.apache.kafka.clients.producer._


object KafkaProducer {
def main(args: Array[String]) {
  val topic="topic1"
  val brokers="192.168.1.126:9092,192.168.1.127:9092,192.168.1.128:9092"
  val messagesPerSec=1 //每秒发送几条信息  
  val wordsPerMessage =4 //一条信息包括多少个单词  
  // Zookeeper connection properties  
    val props = new HashMap[String, Object]()  
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers)  
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,  
      "org.apache.kafka.common.serialization.StringSerializer")  
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,  
      "org.apache.kafka.common.serialization.StringSerializer")  
    val producer = new KafkaProducer[String, String](props) 
    // Send some messages  
     while(true) {  
      (1 to messagesPerSec.toInt).foreach { messageNum =>  
        val str = (1 to wordsPerMessage.toInt).map(x => scala.util.Random.nextInt(10).toString)  
          .mkString(" ")  
        val message = new ProducerRecord[String, String](topic, null, str)  
        producer.send(message)  
        println(message)  
      }  
      Thread.sleep(1000)  
    }  
  }  
}

 打包运行命令：hadoop jar jar包  （注意jar包是可运行的jar包）

消费者消费命令： ./kafka-console-consumer.sh  --zookeeper zk01:2181,zk02:2181  --topic topic1 --from-beginning

可以正常消费。

2、编写SparkStreaming代码读kafka中的数据，每2s读一次

　　代码如下：

package kafkaSparkStream

import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.kafka.KafkaUtils
import kafka.serializer.StringDecoder
/**
 * sparkStreaming读取kafka中topic的数据
 */
object KafkaToSpark {
def main(args: Array[String]) {
  if (args.length<2) {
  System.err.println("Usage: <brokers> <topics>");
  System.exit(1)
  }
  val Array(brokers,topics)=args
  //2s从kafka中读取一次
  val conf=new SparkConf().setAppName("KafkaToSpark");
  val scc=new StreamingContext(conf,Seconds(2))
  // Create direct kafka stream with brokers and topics  
  val topicSet=topics.split(",").toSet
  val kafkaParams=Map[String,String]("metadata.broker.list"->brokers)
  //获取信息
  val messages=KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
      scc,kafkaParams,topicSet)
  // Get the lines, split them into words, count the words and print 
 val lines= messages.map(_._2)
 val words=lines.flatMap(_.split(" "))
 val wordCouts=words.map(x =>(x,1L)).reduceByKey(_+_)
 wordCouts.print
 //开启计算
 scc.start()
 scc.awaitTermination()
}

}

 打包运行命令：./spark-submit --class kafkaSparkStream.KafkaToSpark --master yarn-client /home/hadoop/sparkJar/kafkaToSpark.jar 192.168.1.126:9092,192.168.1.127:9092,192.168.1.128:9092 topic1

运行成功！