Dataflow Java SDK中的BigQuery修补程序操作

时间:2021-02-27 15:30:09

Per the question re: making a custom Sink to update Schemas dynamically in Dataflow, I was wondering if the Patch operation is exposed in BigQueryIO API somewhere?

根据问题re:使用自定义接收器在Dataflow中动态更新Schema,我想知道Patch操作是否在某个地方的BigQueryIO API中公开?

This is a crucial piece of updating schemas on the fly. We are merging schemas as they come in in backwards compatible ways.

这是动态更新模式的关键部分。我们正在合并模式,因为它们以向后兼容的方式进入。

1 个解决方案

#1


1  

This is not in the SDK itself, but can use a standard pattern of side-effecting from a DoFn. Specifically, you'll want to make sure that the BigQuery client you create is marked transient, as DoFns must be serializable.

这不在SDK本身,但可以使用DoFn的标准副作用模式。具体来说,您需要确保您创建的BigQuery客户端标记为瞬态,因为DoFns必须是可序列化的。

class PatchFn extends DoFn<> {
  private transient BigQuery bq;
  ...
}

#1


1  

This is not in the SDK itself, but can use a standard pattern of side-effecting from a DoFn. Specifically, you'll want to make sure that the BigQuery client you create is marked transient, as DoFns must be serializable.

这不在SDK本身,但可以使用DoFn的标准副作用模式。具体来说,您需要确保您创建的BigQuery客户端标记为瞬态,因为DoFns必须是可序列化的。

class PatchFn extends DoFn<> {
  private transient BigQuery bq;
  ...
}