如何从Cassandra表加载数据

I am working on Spark version: 2.0.1 and Cassandra 3.9. I want to read data from a table in cassandra by CassandraSQLContext. However, Spark 2.0 was changed and using sparkSession. I am trying to use sparkSession and I am lucky, the following is my code.

我正在研究Spark版本:2.0.1和Cassandra 3.9。我想通过CassandraSQLContext从cassandra中的表中读取数据。但是,Spark 2.0已更改并使用sparkSession。我正在尝试使用sparkSession,我很幸运,以下是我的代码。

Could you please review and give your advice?

能否请您复习并提出建议?

def main(args: Array[String], date_filter: String): Unit = {
    val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost")
    val sc = new SparkContext(conf)
     val sparkSession = SparkSession.builder
      .master("local")
      .appName("my-spark-app")
      .config(conf)
      .getOrCreate()
       import sparkSession.implicits._
       import org.apache.spark.sql._
    val rdd = sparkSession
      .read
      .format("org.apache.spark.sql.cassandra")
      .options(Map("table" -> "users", "keyspace" -> "monita"))
      .load() 
      println("count: " +rdd.count())  

  }

1 个解决方案

#1

Your code looks ok. You don't need to create SC. You can set Cassandra connection properties in config like below.

你的代码看起来不错。您不需要创建SC。您可以在配置中设置Cassandra连接属性,如下所示。

val sparkSession = SparkSession
  .builder
  .master("local")
  .appName("my-spark-app")
  .config("spark.cassandra.connection.host", "127.0.0.1")
  .config("spark.cassandra.connection.port", "9042")
  .getOrCreate()