spark记录

Filtering multiple values in multiple columns:

In the case where you're pulling data from a database (Hive or SQL type db for this example) and need to filter on multiple columns, it might just be easier to load the table with the first filter, then iterate your filters through the RDD (multiple small iterations is the encouraged way of Spark programming):

{

    import org.apache.spark.sql.hive.HiveContext

    val hc = new HiveContext(sc)

    val first_data_filter = hc.sql("SELECT col1,col2,col2 FROM tableName WHERE col3 IN ('value_1', 'value_2', 'value_3)")

    val second_data_filter = first_data_filter.filter(rdd => rdd(1) == "50" || rdd(1) == "20")

    val final_filtered_data = second_data_filter.filter(rdd => rdd(0) == "1500")

}

https://segmentfault.com/a/1190000002614456

相关文章