具有常用功能的Google Cloud Dataflow自定义密钥

时间：2021-10-08 15:27:22

We are using the Dataflow Java SDK and we have an increasing number of custom key classes that are almost the same.

我们正在使用Dataflow Java SDK，并且我们有越来越多的自定义密钥类几乎相同。

I would like to have them extend a common abstract class however the Dataflow SDK seems to try to instantiate the abstract class causing an InstantiationException.

我想让它们扩展一个公共抽象类，但Dataflow SDK似乎试图实例化导致InstantiationException的抽象类。

Caused by: java.lang.RuntimeException: java.lang.InstantiationException
    at org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:316)
    at org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:332)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
    at com.google.cloud.dataflow.sdk.coders.AvroCoder.decode(AvroCoder.java:242)
    at com.google.cloud.dataflow.sdk.coders.KvCoder.decode(KvCoder.java:97)
    at com.google.cloud.dataflow.sdk.coders.KvCoder.decode(KvCoder.java:42)
    at com.google.cloud.dataflow.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:156)
    at com.google.cloud.dataflow.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:139)
    at com.google.cloud.dataflow.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:133)
    at com.google.cloud.dataflow.sdk.util.MutationDetectors$CodedValueMutationDetector.<init>(MutationDetectors.java:108)
    at com.google.cloud.dataflow.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:45)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$ImmutabilityCheckingOutputManager.output(ParDo.java:1218)
    at com.google.cloud.dataflow.sdk.util.DoFnRunner$DoFnContext.outputWindowedValue(DoFnRunner.java:329)
    at com.google.cloud.dataflow.sdk.util.DoFnRunner$DoFnProcessContext.output(DoFnRunner.java:483)
    at com.telstra.cdf.rmr.model.pardos.ParDoAbstractCampaignUAKeyExtractor.processElement(ParDoAbstractCampaignUAKeyExtractor.java:5

here is our abstract class,

这是我们的抽象类，

@DefaultCoder(AvroCoder.class)
public abstract class SuperClassKey  {
    public SuperClassKey(){}
    public abstract double getSomeValue();
}

and this is the sub class

这是子类

@DefaultCoder(AvroCoder.class)
public class SubClassKey extends SuperClassKey {
    public String foo;

    public SubClassKey() {
    }

    public SubClassKey(String foo){
        this.foo = foo;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        SubClassKey that = (SubClassKey) o;

        if (!foo.equals(that.foo)) return false;

        return true;
    }

    @Override
    public int hashCode() {
        return foo.hashCode();
    }

    @Override
    public double getSomeValue() {
        return foo;
    }
}

I have also tried using an interface without success.

我也试过使用界面但没有成功。

Is it possible to have a common abstract class or interface between Keys?

键之间是否可以有一个共同的抽象类或接口？

1 个解决方案

#1

2  

The issue is likely from using a PCollection<SuperClassKey> instead of PCollection<SubClassKey>. The PCollection needs to be typed with a concrete class. The coder can be explicitly specified with .setCoder(AvroCoder.of(SubClassKey.class)) if type inference is not sufficient.

问题可能来自使用PCollection 而不是PCollection 。 PCollection需要使用具体类进行输入。如果类型推断不充分，可以使用.setCoder（AvroCoder.of（SubClassKey.class））显式指定编码器。

#1

2  

The issue is likely from using a PCollection<SuperClassKey> instead of PCollection<SubClassKey>. The PCollection needs to be typed with a concrete class. The coder can be explicitly specified with .setCoder(AvroCoder.of(SubClassKey.class)) if type inference is not sufficient.

问题可能来自使用PCollection 而不是PCollection 。 PCollection需要使用具体类进行输入。如果类型推断不充分，可以使用.setCoder（AvroCoder.of（SubClassKey.class））显式指定编码器。

标签：

相关文章