获取com.google.api.services.datastore.client.DatastoreException：从Dataflow访问云数据存储区时出现后端错误

I am using below code snippet to access Cloud Datastore from Dataflow. Datastore is created in my project and I am able to run Dataflow samples that's accessing only cloud storage.

我使用下面的代码段从Dataflow访问Cloud Datastore。数据存储区是在我的项目中创建的，我可以运行仅访问云存储的Dataflow示例。

    Pipeline p = Pipeline.create(options);
    String datasetId = "myGoogleProjectId";
    p.apply(
             Read.from(DatastoreIO.source()
                    .withDataset(datasetId)
                    .withQuery(getQuery())))
                .apply(new ParseEntity())
                .apply(new ProcessEntity())
                .apply(TextIO.Write.named("WriteEntityAttributes").to(options.getOutput()));        
    p.run();

I am using below code to create the query,

我使用下面的代码来创建查询，

        Query.Builder q = Query.newBuilder();
        q.addKindBuilder().setName("MyKind");
        return q.build();

But I am getting below error while running the dataflow using DirectPiplelineRunner

但是在使用DirectPiplelineRunner运行数据流时，我遇到了以下错误

Aug 17, 2016 1:02:15 AM com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner run
INFO: Executing pipeline using the DirectPipelineRunner.
Aug 17, 2016 1:06:26 AM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 503, will NOT retry: https://www.googleapis.com/datastore/v1beta2/datasets/myGoogleProjectId/runQuery
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: com.google.api.services.datastore.client.DatastoreException: Backend Error
    at com.google.cloud.dataflow.sdk.io.Read$Bounded$1.evaluateReadHelper(Read.java:189)
    at com.google.cloud.dataflow.sdk.io.Read$Bounded$1.evaluate(Read.java:168)
    at com.google.cloud.dataflow.sdk.io.Read$Bounded$1.evaluate(Read.java:164)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:858)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:219)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:215)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:102)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:259)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:814)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:526)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:96)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:180)
    at com.google.cloud.dataflow.examples.WordCount.main(WordCount.java:291)
Caused by: java.io.IOException: com.google.api.services.datastore.client.DatastoreException: Backend Error
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$DatastoreReader.advance(DatastoreIO.java:920)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$DatastoreReader.start(DatastoreIO.java:911)
    at com.google.cloud.dataflow.sdk.io.Read$Bounded$1.evaluateReadHelper(Read.java:178)
    ... 12 more
Caused by: com.google.api.services.datastore.client.DatastoreException: Backend Error
    at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:115)
    at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:81)
    at com.google.api.services.datastore.client.BaseDatastoreFactory$RemoteRpc.call(BaseDatastoreFactory.java:41)
    at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$DatastoreReader.getIteratorAndMoveCursor(DatastoreIO.java:969)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$DatastoreReader.advance(DatastoreIO.java:918)
    ... 14 more
Caused by: com.google.api.client.http.HttpResponseException: 503 Service Unavailable
Backend Error
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
    at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:78)
    ... 18 more

Highly appreciate your help!

非常感谢您的帮助！

1 个解决方案

#1

To use this version of Dataflow, you need to enable the "Google Cloud Datastore API (v1beta2)" API in the Cloud Console.

要使用此版本的Dataflow，您需要在云控制台中启用“Google Cloud Datastore API（v1beta2）”API。