Apache Spark - java.lang.NoSuchMethodError：breeze.linalg.Vector $ .scalarOf（）Lbreeze / linalg / support / ScalarOf

Here is the error:

这是错误:

Exception in thread "main" java.lang.NoSuchMethodError: breeze.linalg.Vector$.scalarOf()Lbreeze/linalg/support/ScalarOf;
at org.apache.spark.ml.knn.Leaf$$anonfun$4.apply(MetricTree.scala:95)
at org.apache.spark.ml.knn.Leaf$$anonfun$4.apply(MetricTree.scala:93)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48)
at org.apache.spark.ml.knn.Leaf$.apply(MetricTree.scala:93)
at org.apache.spark.ml.knn.MetricTree$.build(MetricTree.scala:169)
at org.apache.spark.ml.knn.KNN.fit(KNN.scala:388)
at org.apache.spark.ml.classification.KNNClassifier.train(KNNClassifier.scala:109)
at org.apache.spark.ml.classification.KNNClassifier.fit(KNNClassifier.scala:117)
at SparkKNN$.main(SparkKNN.scala:23)
at SparkKNN.main(SparkKNN.scala)

Here is the program that is triggering the error:

这是触发错误的程序:

object SparkKNN {
    def main(args: Array[String]) {
        val spark = SparkSession.builder().master("local").config("spark.sql.warehouse.dir", "file:///c:/tmp/spark-warehouse").getOrCreate()
        val sc = spark.sparkContext
        import spark.implicits._
        //read in raw label and features
        val training = spark.read.format("com.databricks.spark.csv").option("header", true).load("E:/Machine Learning/knn_input.csv")
        var df = training.selectExpr("cast(label as double) label", "cast(feature1 as int) feature1","cast(feature2 as int) feature2","cast(feature3 as int) feature3")
        val assembler = new VectorAssembler().setInputCols(Array("feature1","feature2","feature3")).setOutputCol("features")
        df = assembler.transform(df)
        //MLUtils.loadLibSVMFile(sc, "C:/Program Files (x86)/spark-2.0.0-bin-hadoop2.7/data/mllib/sample_libsvm_data.txt").toDF()

        val knn = new KNNClassifier()
            .setTopTreeSize(df.count().toInt / 2)
            .setK(10)
        val splits = df.randomSplit(Array(0.7, 0.3))
        val (trainingData, testData) = (splits(0), splits(1))
        val knnModel = knn.fit(trainingData)

        val predicted = knnModel.transform(testData)
        predicted.show()
    }
}

I am using Apache spark 2.0 with scala version 2.11.8. It looks like a version difference issue. Any ideas?

我正在使用Apache spark 2.0和scala版本2.11.8。它看起来像版本差异问题。有任何想法吗?

1 个解决方案

#1

Spark MLLib 2.0 brings in this version of Breeze:

Spark MLLib 2.0带来了这个版本的Breeze:

"org.scalanlp" % "breeze_2.11" % "0.11.2"

“org.scalanlp”%“breeze_2.11”%“0.11.2”

You must have another library in your classpath that also has a dependency on Breeze but a different version, and that's the one being loaded. As a result, MLLib is operating with a different version of Breeze at runtime than was around at compile-time.

你的类路径中必须有另一个库,它也依赖于Breeze但是一个不同的版本,那就是正在加载的库。因此,MLLib在运行时使用的是不同版本的Breeze,而不是在编译时运行。

You have multiple options. You can find that undesirable transitive dependency on Breeze and exclude it. You can add a direct dependency on the version of that library that has the same Breeze dependency MLLib does. Or you can add a direct dependency on the Breeze version MLLib needs.

你有多种选择。您可以在Breeze上找到不受欢迎的传递依赖并将其排除。您可以直接依赖于具有与MLLib相同的Breeze依赖项的库的版本。或者您可以直接依赖Breeze版本的MLLib需求。

#1