当我尝试在相同的管道执行中创建不同的BigQuery表时出错

I have a pipeline execution with the below code:

我有一个使用以下代码的管道执行：

PCollection<TableRow> test1 = ...
test1
    .apply(BigQueryIO.Write
        .named("test1 write")
        .to("project_name:dataset_name.test1")
        .withSchema(tableSchema)
        .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
        .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));

PCollection<TableRow> test2 = ...
test2
    .apply(BigQueryIO.Write
        .named("test2 write")
        .to("project_name:dataset_name.test2")
        .withSchema(tableSchema)
        .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
        .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));

If I execute the pipeline and neither table "test1" nor "test2" exists, I obtain the below information:

如果我执行管道并且表“test1”和“test2”都不存在，我将获得以下信息：

jun 09, 2015 12:29:24 PM com.google.cloud.dataflow.sdk.util.BigQueryTableInserter tryCreateTable
INFORMACIÓN: Trying to create BigQuery table: project_name:dataset_name.test1
jun 09, 2015 12:29:27 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
ADVERTENCIA: Request failed with code 404, will NOT retry: https://www.googleapis.com/bigquery/v2/projects/pragmatic-armor-455/datasets/audit/tables/project_name:dataset_name.test2/insertAll
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table project_name:dataset_name.test2",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table project_name:dataset_name.test2"
}

Why only the first table is created?

为什么只创建第一个表？

Thanks in advance.

提前致谢。

1 个解决方案

#1

Thanks for reporting this. The cause was a bug in BigQueryIO that caused the second table to occasionally not be created. This bug has now been fixed in github with this commit. The fix will be pushed to maven later this month. Sorry for the trouble!

谢谢你报道这个。原因是BigQueryIO中的一个错误导致偶尔不创建第二个表。现在已经使用此提交在github中修复了此错误。修复程序将在本月晚些时候推送到maven。抱歉，添麻烦了！