如何在不使用spark-submit的情况下将java程序中的spark作业提交到独立的spark集群？

I am using spark to perform some computations but want it to be submitted from java application.It works proper using when submitted using spark-submit script.Has anyone tried to do this?

我正在使用spark来执行一些计算但是希望它从java应用程序提交。当使用spark-submit脚本提交时,它正常工作。有人试过这样做吗?

Thanks.

2 个解决方案

#1

Don't forget to add the fat JAR containing your code to the context.

不要忘记将包含代码的胖JAR添加到上下文中。

val conf = new SparkConf()
   .setMaster(...)
   .setAppName(...)
   .setJars("/path/to/code.jar")
val sc = new SparkContext(conf)

#2

As long as you have a master and available worker started, you should be able to if you have the following in your java application:

只要你有一个master和可用的worker已经启动,你应该能够在java应用程序中拥有以下内容:

String master = "spark://IP:7077"; //set IP address to that of your master
String appName = "Name of your Application Here";
SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);;
JavaSparkContext sc = new JavaSparkContext(conf);

I was able to run junit tests from within IntelliJ that utilized the JavaSparkContext without having to use the spark-submit script. I am running into issues when performing actions on DataFrames though (not sure if that's related).

我能够在IntelliJ中运行使用JavaSparkContext的junit测试,而不必使用spark-submit脚本。我在DataFrame上执行操作时遇到了问题(不确定这是否相关)。

#1