无法解析主URL：'spark.bluemix.net'

I'm trying to connect to IBM's Spark as a Service running on Bluemix from RStudio running on my desktop machine.

我正在尝试从桌面计算机上运行的RStudio连接到运行在Bluemix上的IBM的Spark as a Service。

I have copied the config.yml from the automatically configured RStudio environment running on IBM's Data Science Experience:

我从IBM的Data Science Experience上运行的自动配置的RStudio环境中复制了config.yml:

default:
    method: "shell"

CS-DSX:
    method: "bluemix"
    spark.master: "spark.bluemix.net"
    spark.instance.id: "myinstanceid"
    tenant.id: "mytenantid"
    tenant.secret: "mytenantsecret"
    hsui.url: "https://cdsx.ng.bluemix.net"

I am attempting to connect like so:

我试图像这样连接:

install.packages("sparklyr")

library(sparklyr)
spark_install(version = "1.6.2") # installed spark to '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'

spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'

config = spark_config(file = "./config.yml", use_default = FALSE, config = "CSX-DSX")

sc <- spark_connect(spark_home = spark_home, config = config)

The error:

17/03/07 09:36:19 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'spark.bluemix.net'
    at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2735)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:522)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
    at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    ...

There are a few other questions on * with similar error messages, but they are not trying to connect to the Spark service running on Bluemix.

*上还有一些其他问题与类似的错误消息,但他们没有尝试连接到Bluemix上运行的Spark服务。

Update 1

I've changed my config.yml to look like this:

我已将config.yml更改为如下所示:

default:
    method: "bluemix"
    spark.master: "spark://spark.bluemix.net:7070"
    spark.instance.id: "7a4089bf-3594-4fdf-8dd1-7e9fd7607be5"
    tenant.id: "sdd1-7e9fd7607be53e-39ca506ba762"
    tenant.secret: "6146a713-949f-4d4e-84c3-9913d2165b9e"
    hsui.url: "https://cdsx.ng.bluemix.net"

... and my connection code to look like this:

...和我的连接代码看起来像这样:

install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE)
sc <- spark_connect(spark_home = spark_home, config = config)

However, the error is now:

但是,错误现在是:

Error in force(code) : 
  Failed during initialize_connection: java.lang.NullPointerException
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
    at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sparklyr.Invoke$.invoke(invoke.scala:94)
    ...

3 个解决方案

#1

The library tries to parse a URL, but you're giving it a hostname.

该库试图解析一个URL,但是你给它一个主机名。

Try spark://spark.bluemix.net for spark.master.

为spark.master尝试spark://spark.bluemix.net。

#2

Please follow the blog post http://datascience.ibm.com/blog/access-ibm-analytics-for-apache-spark-from-rstudio/ to connect Bluemix SparkaaS from DSX RStudio.

请关注博客文章http://datascience.ibm.com/blog/access-ibm-analytics-for-apache-spark-from-rstudio/,从DSX RStudio连接Bluemix SparkaaS。

#3

I received the following response from the engineering team:

我收到了工程团队的以下回复:

RStudio desktop version doesn't support at this time to use sparklyr package to connect Bluemix SparkaaS service

RStudio桌面版本目前不支持使用sparklyr包连接Bluemix SparkaaS服务

#1