为什么Scala编译器失败并且“无法使用此类签名解析引用reduceByKeyAndWindow”？

I've got the following line in my Spark Streaming application that compiles fine:

我的Spark Streaming应用程序中有以下行编译好:

val kafkaDirectStream: InputDStream[ConsumerRecord[String,String]] = KafkaUtils.createDirectStream(...)
kafkaDirectStream.map(_ => ("mockkey", 1)).reduceByKeyAndWindow(_+_, Seconds(30))

When I use the variant of reduceByKeyAndWindow with two Durations as follows:

当我使用reduceByKeyAndWindow的变体和两个持续时间时,如下所示:

.reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

I face the below compiler error:

我面对以下编译器错误:

Cannot resolve reference reduceByKeyAndWindow with such signature

无法使用此类签名解析引用reduceByKeyAndWindow

Why?

1 个解决方案

#1

After kafkaDirectStream.map(_ => ("mockkey", 1)), you'll have DStream[(String, Int)] (which you can read about in the official documentation at org.apache.spark.streaming.dstream.DStream).

在kafkaDirectStream.map(_ =>(“mockkey”,1))之后,您将拥有DStream [(String,Int)](您可以在org.apache.spark.streaming.dstream的官方文档中阅读。 DSTREAM)。

It appears that implicit scope does not give enough knowledge about types and hence the error:

似乎隐式范围没有提供关于类型的足够知识,因此错误:

missing parameter type for expanded function ((x$3, x$4) => x$3.$plus(x$4))

扩展函数缺少参数类型(($ 3,x $ 4)=> x $ 3. $ plus(x $ 4))

Unfortunatelly, I can't really explain what the root cause of the compilation error is, but a solution is to define a method or function with the types specified explicitly and use it instead (not underscores alone, i.e. _ + _).

不幸的是,我无法解释编译错误的根本原因是什么,但解决方案是使用显式指定的类型定义方法或函数并使用它(而不是单独使用下划线,即_ + _)。

val add: (Int, Int) => Int = _ + _
// or def add(x: Int, y: Int) = x + y
mapped.reduceByKeyAndWindow(add, Seconds(30), Seconds(10))

That will pass the Scala compiler.

这将通过Scala编译器。

(wish I knew if there's a better solution somehow helping the Scala type inferencer).

(希望我知道是否有更好的解决方案以某种方式帮助Scala类型推理器)。

#1