Google Dataflow - 无效的区域端点 - 不可能在nodejs客户端的模板上设置区域

I have a pipeline stored as a template. I'm using the node.js client to run this pipeline from a cloud function. Everything works fine, but when I need to run this template from different regions I get errors.

我有一个管道存储为模板。我正在使用node.js客户端从云功能运行此管道。一切正常，但当我需要从不同地区运行此模板时，我会收到错误。

According to the documentation, I can set it through the location parameter in the payload

根据文档，我可以通过有效负载中的location参数进行设置

{
  projectId: 123,
  resource: {
    location: "europe-west1",
    jobName: `xxx`,
    gcsPath: 'gs://xxx'
  }
}

That gives me the following error:

这给了我以下错误：

The workflow could not be created, since it was sent to an invalid regional endpoint (europe-west1). 
Please resubmit to a valid Cloud Dataflow regional endpoint.

I get the same error if I move the location parameter out of resource node, such as:

如果我将location参数移出资源节点，我会得到相同的错误，例如：

{
  projectId: 123,
  location: "europe-west1",
  resource: {
    jobName: `xxx`,
    gcsPath: 'gs://xxx'
  }
}

If I set the zone in the environment and remove the location such as:

如果我在环境中设置区域并删除位置，例如：

{
  projectId: 123,
  resource: {
    jobName: `xxx`,
    gcsPath: 'gs://xxx',
    environment: {
        zone: "europe-west1-b"
    }
   }
}

I do not get any errors anymore, but dataflow UI tells me the job is running in us-east1

我不再有任何错误，但数据流UI告诉我作业正在us-east1中运行

How can I run this template and providind the region / zone I

如何运行此模板并提供区域/区域I.

2 个解决方案

#1

As explained here, there are actually two endpoints:

如此处所述，实际上有两个端点：

dataflow.projects.locations.templates.launch (API Explorer)
dataflow.projects.locations.templates.launch（API Explorer）
dataflow.projects.templates.launch (API Explorer)
dataflow.projects.templates.launch（API Explorer）

For Dataflow regional endpoints to work, the first one must be used (dataflow.projects.locations.templates.launch). This way, the location parameter in the request will be accepted. Code snippet:

要使Dataflow区域端点起作用，必须使用第一个端点（dataflow.projects.locations.templates.launch）。这样，将接受请求中的location参数。代码段：

var dataflow = google.dataflow({
    version: "v1b3",
    auth: authClient
});

var opts = {
    projectId: project,
    location: "europe-west1",
    gcsPath: "gs://path/to/template",
    resource: {
        parameters: launchParams,
        environment: env
    }
};
dataflow.projects.locations.templates.launch(opts, (err, result) => {
    if (err) {
        throw err;
    }
    res.send(result.data);
});

#2

I have been testing this though both the API explorer and the console using Google-provided templates. Using the wordcount example I get the same generic error than you do with the API explorer, which is the same if the location name is incorrect. However, the Console provides more information:

我通过API浏览器和使用Google提供的模板的控制台对此进行了测试。使用wordcount示例，我得到的相同的一般错误比使用API资源管理器时更复杂，如果位置名称不正确，则会出现相同的错误。但是，控制台提供了更多信息：

Templated Dataflow jobs using Java or Python SDK version prior to 2.0 are not supported outside of the us-central1 Dataflow Regional Endpoint. The provided template uses Google Cloud Dataflow SDK for Java 1.9.1.

在us-central1数据流区域端点之外不支持使用2.0之前的Java或Python SDK版本的模板化数据流作业。提供的模板使用Google Cloud Dataflow SDK for Java 1.9.1。

Which is documented here as I previously commented. Running it confirms it's using a deprecated SDK version. I would recommend doing the same process to see if this is actually your case, too.

这是我之前评论过的文献。运行它确认它使用的是已弃用的SDK版本。我建议你做同样的过程，看看这是否也是你的情况。

Choosing a different template, in my case the GCS Text to BigQuery option from the Console's drop-down menu (which uses Apache Beam SDK for Java 2.2.0) with location set to europe-west1 works fine for me (and the job actually runs in that region).

选择一个不同的模板，在我的例子中，从控制台的下拉菜单（使用Apache Beam SDK for Java 2.2.0）将位置设置为europe-west1的GCS Text to BigQuery选项对我来说很好（并且该作业实际运行在该地区）。

TL;DR: your request is correct in your first example but you'll need to update the template to a newer SDK if you want to use regional endpoints.

TL; DR：您的请求在第一个示例中是正确的，但如果您想使用区域端点，则需要将模板更新为较新的SDK。

#1