I'm getting below error while running spark program using spark-submit.
我使用spark-submit运行spark程序时遇到错误。
My spark-cluster is of version 2.0.0 and I use sbt to compile my code and below is my sbt dependencies.
我的spark-cluster是版本2.0.0,我使用sbt编译我的代码,下面是我的sbt依赖项。
libraryDependencies ++= Seq(
"commons-io" % "commons-io" % "2.4",
"com.google.guava" % "guava" % "19.0",
"jfree" % "jfreechart" % "1.0.13",
("org.deeplearning4j" % "deeplearning4j-core" % "0.5.0").exclude("org.slf4j", "slf4j-log4j12"),
"org.jblas" % "jblas" % "1.2.4",
"org.nd4j" % "canova-nd4j-codec" % "0.0.0.15",
"org.nd4j" % "nd4j-native" % "0.5.0" classifier "" classifier "linux-x86_64",
"org.deeplearning4j" % "dl4j-spark" % "0.4-rc3.6" ,
"org.apache.spark" % "spark-sql_2.10" % "1.3.1",
"org.apache.spark" % "spark-hive_2.10" % "1.3.1",
"org.apache.hive" % "hive-serde" % "0.14.0",
("org.deeplearning4j" % "arbiter-deeplearning4j" % "0.5.0"))
16/11/14 22:57:03 INFO hive.HiveSharedState: Warehouse path is 'file:/home/hduser/spark-warehouse'.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.sql(Ljava/lang/String;)Lorg/apache/spark/sql/DataFrame;
at poc.common.utilities.StockData$.fetchStockData(StockData.scala:15)
at poc.analaticsEngine.AnalaticsStockWorkBench.fetchTrainingDataSet(AnalaticsStockWorkBench.scala:69)
at poc.analaticsEngine.AnalaticsStockWorkBench.trainModel(AnalaticsStockWorkBench.scala:79)
at test.poc.analatics.StockPrediction$.testTrainSaveModel(StockPrediction.scala:21)
at test.poc.analatics.StockPrediction$.main(StockPrediction.scala:10)
at test.poc.analatics.StockPrediction.main(StockPrediction.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/11/14 22:57:03 INFO spark.SparkContext: Invoking stop() from shutdown hook
2 个解决方案
#1
0
First of all, you're saying that you use Spark 2.0.0, but in your dependencies you have
首先,你说你使用Spark 2.0.0,但在你的依赖项中你有
"org.apache.spark" % "spark-sql_2.10" % "1.3.1",
"org.apache.spark" % "spark-hive_2.10" % "1.3.1",
You need to change those dependencies to version 2.0.0 to keep it consistent with Spark. What is more, you don't need to specify spark-sql
dependency separately, because it's already contained in spark-hive
. hive-serde
also comes in 2.1.0 version now, so 0.14 is probably obsolete.
您需要将这些依赖项更改为2.0.0版,以使其与Spark保持一致。更重要的是,您不需要单独指定spark-sql依赖项,因为它已经包含在spark-hive中。 hive-serde现在也有2.1.0版本,所以0.14可能已经过时了。
#2
0
Please follow the dl4j examples for versioning. I'm not sure where or how you got canova in there (we haven't used it for nearly 6 months now?)
请按照dl4j示例进行版本控制。我不知道你在哪里或如何得到canova(我们现在已经有近6个月未使用它了?)
https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/pom.xml
#1
0
First of all, you're saying that you use Spark 2.0.0, but in your dependencies you have
首先,你说你使用Spark 2.0.0,但在你的依赖项中你有
"org.apache.spark" % "spark-sql_2.10" % "1.3.1",
"org.apache.spark" % "spark-hive_2.10" % "1.3.1",
You need to change those dependencies to version 2.0.0 to keep it consistent with Spark. What is more, you don't need to specify spark-sql
dependency separately, because it's already contained in spark-hive
. hive-serde
also comes in 2.1.0 version now, so 0.14 is probably obsolete.
您需要将这些依赖项更改为2.0.0版,以使其与Spark保持一致。更重要的是,您不需要单独指定spark-sql依赖项,因为它已经包含在spark-hive中。 hive-serde现在也有2.1.0版本,所以0.14可能已经过时了。
#2
0
Please follow the dl4j examples for versioning. I'm not sure where or how you got canova in there (we haven't used it for nearly 6 months now?)
请按照dl4j示例进行版本控制。我不知道你在哪里或如何得到canova(我们现在已经有近6个月未使用它了?)
https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/pom.xml