Spark SQL 自定义函数类型

时间：2022-05-15 15:50:24

一、spark读取数据

前段时间一直在研究GeoMesa下的Spark JTS，Spark JTS支持用户自定义函数，然后有一份数据，读取文件：

package com.geomesa.spark.SparkCore

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.types.{ArrayType, DataTypes, StringType, StructField, StructType}

object test {

  def main(args: Array[String]): Unit = {

    import org.locationtech.geomesa.spark.jts._

    //spark

    val spark: SparkSession = {

      SparkSession.builder()

        .appName("test")

        .master("local[*]")

        .getOrCreate()

        //需注入spark.jts._包

        .withJTS

    }

    val dataFile = this.getClass.getClassLoader.getResource("gsmc.txt").getPath

    val df = spark.read

      .schema(schema)

      .json(dataFile)

    //.show(5, false)

    //.printSchema()

  }

}

二、自定义函数结构

然后打印出来的数据结构如下，通过spark sql的自定义函数构建这个结构的数据，主要构建features下的相关数据结构，之前耗时N久，各种不会构建以及构建错误，后，皇天不负有心人，搞就是了，搞出来了。

    root

   |-- crs: struct (nullable = true)

   |    |-- properties: struct (nullable = true)

   |    |    |-- name: string (nullable = true)

   |    |-- type: string (nullable = true)

   |-- features: array (nullable = true)

   |    |-- element: struct (containsNull = true)

   |    |    |-- geometry: struct (nullable = true)

   |    |    |    |-- coordinates: array (nullable = true)

   |    |    |    |    |-- element: array (containsNull = true)

   |    |    |    |    |    |-- element: array (containsNull = true)

   |    |    |    |    |    |    |-- element: double (containsNull = true)

   |    |    |    |-- type: string (nullable = true)

   |    |    |-- geometry_name: string (nullable = true)

   |    |    |-- id: string (nullable = true)

自定义格式如下：

    val schema = StructType(Array(

      StructField("crs", StringType),

      StructField("features", ArrayType(

        StructType(Array(StructField("geometry",

          StructType(Array(StructField("coordinates",

            ArrayType(DataTypes.createArrayType(ArrayType((DataTypes.DoubleType)))))

          )))))))

    ))

经过printSchema()方法测试，结构如上面的features结构一模一样，nice。

三、附上长长的各种pom

<properties>

        <geospark.version>1.2.0</geospark.version>

        <geotools.version>14.1</geotools.version>

        <spark.version>2.3.1</spark.version>

        <encoding>UTF-8</encoding>

        <scala.binary.version>2.11</scala.binary.version>

    </properties>

    <dependencies>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-client</artifactId>

            <version>2.7.0</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_2.11</artifactId>

            <version>2.2.0</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_2.11</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.scala-lang</groupId>

            <artifactId>scala-library</artifactId>

            <version>2.11.0</version>

        </dependency>

        <dependency>

            <groupId>org.datasyslab</groupId>

            <artifactId>geospark</artifactId>

            <version>${geospark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.geotools</groupId>

            <artifactId>gt-geometry</artifactId>

            <version>20.0</version>

        </dependency>

        <dependency>

            <groupId>com.vividsolutions</groupId>

            <artifactId>jts</artifactId>

            <version>1.13</version>

        </dependency>

        <dependency>

            <groupId>org.json</groupId>

            <artifactId>json</artifactId>

            <version>20180813</version>

        </dependency>

        <dependency>

            <groupId>com.esri.geometry</groupId>

            <artifactId>esri-geometry-api</artifactId>

            <version>2.2.2</version>

        </dependency>

        <dependency>

            <groupId>org.geotools</groupId>

            <artifactId>gt-geojson</artifactId>

            <version>${geotools.version}</version>

        </dependency>

        <dependency>

            <groupId>org.geotools</groupId>

            <artifactId>gt-api</artifactId>

            <version>${geotools.version}</version>

        </dependency>

        <dependency>

            <groupId>org.geotools</groupId>

            <artifactId>gt-referencing</artifactId>

            <version>${geotools.version}</version>

        </dependency>

        <dependency>

            <groupId>org.locationtech.geomesa</groupId>

            <artifactId>geomesa-spark-jts_2.11</artifactId>

            <version>2.1.0</version>

        </dependency>

        <dependency>

            <groupId>org.geotools</groupId>

            <artifactId>gt-epsg-hsql</artifactId>

            <version>${geotools.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_${scala.binary.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_${scala.binary.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-catalyst_${scala.binary.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.locationtech.jts.io</groupId>

            <artifactId>jts-io-common</artifactId>

            <version>1.16.0</version>

        </dependency>

        <dependency>

            <groupId>org.locationtech.jts</groupId>

            <artifactId>jts-core</artifactId>

            <version>1.16.0</version>

        </dependency>

        <dependency>

            <groupId>org.locationtech.spatial4j</groupId>

            <artifactId>spatial4j</artifactId>

            <version>0.6</version>

        </dependency>

        <dependency>

            <groupId>com.alibaba</groupId>

            <artifactId>fastjson</artifactId>

            <version>1.2.47</version>

        </dependency>

        <!--redis-->

        <dependency>

            <groupId>redis.clients</groupId>

            <artifactId>jedis</artifactId>

            <version>2.9.0</version>

        </dependency>

    </dependencies>

相关文章