使用Java持久化Spark Jobserver NamedObjects

时间：2021-06-27 23:12:16

Using Spark Jobserver 0.6.2 and Apache Spark 2.0.2, I have already some functionalities implemented. But I can't find how to persist a Dataset to be shared across diferents jobs on a specific context in java. It's this functionality only available for Scala? Because on internet I only can find Scala examples. I have check on java and there are some NamedObject* classes in the job-server-api_2.11.jar but not idea how can I use them. Actually my Job classes extends from spark.jobserver.JavaSparkJob.

使用Spark Jobserver 0.6.2和Apache Spark 2.0.2,我已经实现了一些功能。但是我找不到如何在java中的特定上下文中保持数据集在不同的作业之间共享。这个功能仅适用于Scala吗?因为在互联网上我只能找到Scala的例子。我检查了java,在job-server-api_2.11.jar中有一些NamedObject *类,但不知道如何使用它们。实际上我的Job类来自spark.jobserver.JavaSparkJob。

Thanks.

1 个解决方案

#1

0  

  1. You can't use Job server 0.6.2 with Spark 2.0.2. See the compatibility matrix in README.md (https://github.com/spark-jobserver/spark-jobserver#version-information)

    您不能将Job server 0.6.2与Spark 2.0.2一起使用。请参阅README.md中的兼容性矩阵(https://github.com/spark-jobserver/spark-jobserver#version-information)

  2. You might have to use spark-2.0-preview branch.

    您可能必须使用spark-2.0-preview分支。

  3. Here is a thread which talks about new NamedObjects API in java. One of the thread has latest example using new API as attachment too https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/spark-jobserver/lFpuiQp4toU/xkZtkxTHBAAJ

    这是一个讨论java中新的NamedObjects API的线程。其中一个主题有使用新API作为附件的最新示例https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/spark-jobserver/lFpuiQp4toU/xkZtkxTHBAAJ

#1

0  

  1. You can't use Job server 0.6.2 with Spark 2.0.2. See the compatibility matrix in README.md (https://github.com/spark-jobserver/spark-jobserver#version-information)

    您不能将Job server 0.6.2与Spark 2.0.2一起使用。请参阅README.md中的兼容性矩阵(https://github.com/spark-jobserver/spark-jobserver#version-information)

  2. You might have to use spark-2.0-preview branch.

    您可能必须使用spark-2.0-preview分支。

  3. Here is a thread which talks about new NamedObjects API in java. One of the thread has latest example using new API as attachment too https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/spark-jobserver/lFpuiQp4toU/xkZtkxTHBAAJ

    这是一个讨论java中新的NamedObjects API的线程。其中一个主题有使用新API作为附件的最新示例https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/spark-jobserver/lFpuiQp4toU/xkZtkxTHBAAJ

标签：

相关文章