在apache beam中使用JdbcIO访问Hive抛出java.lang.NoClassDefFoundError：org / apache / avro / reflect / AvroSchema

I could access my mysql table using JDBCIO and Avro Coder. Now I am trying to load my hive database using JdbcIO. Below exception has thrown while connecting to hive from dataflow. Any help from the beam geeks would be really helpful.

我可以使用JDBCIO和Avro Coder访问我的mysql表。现在我正在尝试使用JdbcIO加载我的hive数据库。从数据流连接到配置单元时抛出异常。梁极客的任何帮助都会非常有帮助。

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/reflect/AvroSchema
at org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.recurse(AvroCoder.java:426)
at org.apache.beam.sdk.coders.AvroCoder$AvroDeterminismChecker.check(AvroCoder.java:419)
at org.apache.beam.sdk.coders.AvroCoder.<init>(AvroCoder.java:259)
at org.apache.beam.sdk.coders.AvroCoder.of(AvroCoder.java:120)
at com.google.cloud.bigquery.csv.loader.GoogleSQLPipeline.main(GoogleSQLPipeline.java:101)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.reflect.AvroSchema
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more

Below code snippet is trying to access hive :

下面的代码片段正在尝试访问配置单元：

dataflowPipeline
            .apply(JdbcIO.<Customer>read()
                    .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
                            .create("org.apache.hive.jdbc.HiveDriver", "jdbc:hive2://<ip>/mydb")
                            .withUsername("username").withPassword("password"))
                    .withQuery(
                            "select c_customer_id,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_customer_sk,c_current_cdemo_sk,c_current_hdemo_sk from customer")
                    .withRowMapper(new JdbcIO.RowMapper<Customer>() {
                        @Override
                        public Customer mapRow(ResultSet resultSet) throws Exception

POM Dependecies :

POM依赖：

---

<dependencies>
    <dependency>
        <groupId>com.google.cloud.dataflow</groupId>
        <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
        <version>2.0.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-jdbc -->
    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-sdks-java-io-jdbc</artifactId>
        <version>2.0.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc -->
    <dependency>
        <groupId>org.apache.hive</groupId>
        <artifactId>hive-jdbc</artifactId>
        <version>1.2.1</version>
    </dependency>
    <dependency>
        <groupId>jdk.tools</groupId>
        <artifactId>jdk.tools</artifactId>
        <version>1.8.0_131</version>
        <scope>system</scope>
        <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.8.1</version>
    </dependency>

    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>18.0</version>
    </dependency>
    <!-- slf4j API frontend binding with JUL backend -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>
</dependencies>

1 个解决方案

#1

Added dependency for Avro 1.8.1 and manually imported import org.apache.avro.reflect.AvroSchema; Now that issue is solved. Again, issues related to com.google.protobuf.GeneratedMessageV3 is solved by manually importing this class.

为Avro 1.8.1添加了依赖项并手动导入了导入org.apache.avro.reflect.AvroSchema;现在这个问题已经解决了。同样，通过手动导入此类来解决与com.google.protobuf.GeneratedMessageV3相关的问题。