对火花流进行S3检查点会导致错误

I have enabled checkpointing for my sparkstreaming application using the getOrCreate method. The checkpoint directory points to an S3 bucket. The problem i have is a credential issue in accessing S3 :

我已使用getOrCreate方法为我的sparkstreaming应用程序启用了检查点。检查点目录指向S3存储桶。我遇到的问题是访问S3时的凭据问题：

Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).

引起：java.lang.IllegalArgumentException：必须将AWS Access Key ID和Secret Access Key指定为s3 URL的用户名或密码，或者通过设置fs.s3.awsAccessKeyId或fs.s3.awsSecretAccessKey属性（分别）。

I have already set the environment variables (AWS_SECRET_KEY and AWS_ACCESS_KEY). Also my fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey have been specified in the application.conf.. So i dont know why it still fails.

我已经设置了环境变量（AWS_SECRET_KEY和AWS_ACCESS_KEY）。另外我的fs.s3.awsAccessKeyId或fs.s3.awsSecretAccessKey已在application.conf中指定。所以我不知道为什么它仍然失败。

1 个解决方案

#1

The environment variables (AWS_SECRET_KEY and AWS_ACCESS_KEY) no longer works after Spark 1.3.

Spark 1.3之后，环境变量（AWS_SECRET_KEY和AWS_ACCESS_KEY）不再有效。

Please refer to for the new approach:
How to read input from S3 in a Spark Streaming EC2 cluster application

请参阅新方法：如何在Spark Streaming EC2集群应用程序中读取S3的输入

val conf = new SparkConf().setAppName("Simple Application").setMaster("local")      
val sc = new SparkContext(conf)
val hadoopConf=sc.hadoopConfiguration;
hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId",myAccessKey)
hadoopConf.set("fs.s3.awsSecretAccessKey",mySecretKey)

#1