My application has requirement to real time processing data using EMR spark Jobs.
我的应用程序要求使用EMR spark Jobs实时处理数据。
I am using addJobFlowSteps API of EMR from Lambda, but which is sequential in nature.
我正在使用来自Lambda的EMR的addJobFlowSteps API,但这本质上是顺序的。
Is there any other way to submit Jobs to EMR from Lambda which is concurrent in nature ?
有没有其他方法可以将Lambda中的作业提交给EMR,这是同时存在的?
1 个解决方案
#1
0
You can use custom 'spark-submit' script.
您可以使用自定义'spark-submit'脚本。
I use this:
我用这个:
cat /usr/bin/spark-submit-forked
#!/bin/bash
(nohup spark-submit "$@" > /dev/null 2>&1 &)
#1
0
You can use custom 'spark-submit' script.
您可以使用自定义'spark-submit'脚本。
I use this:
我用这个:
cat /usr/bin/spark-submit-forked
#!/bin/bash
(nohup spark-submit "$@" > /dev/null 2>&1 &)