Apache beam:从多个订阅中读取

时间:2023-01-30 15:38:32

I want to create a dataflow that listens to multiple subscriptions and writes to BigQuery. As per Google's documentation, I can read multiple PCollection objects and combine them together.

我想创建一个侦听多个订阅并写入BigQuery的数据流。根据Google的文档,我可以读取多个PCollection对象并将它们组合在一起。

However, looking at PubsubIO.Read's javadoc here, it seems subscription method accepts only one String. So, do we need to chain the subscription methods in order to read from multiple subscriptions?

但是,看看PubsubIO.Read的javadoc,似乎订阅方法只接受一个String。那么,我们是否需要链接订阅方法以便从多个订阅中读取?

1 个解决方案

#1


2  

Something you can do is to have multiple Reads from PubSub, where each reads from one subscription, and then Flatten their results.

你可以做的就是从PubSub获得多个Read,其中每个读取一个订阅,然后展平他们的结果。

Would that do what you need?

这会做你需要的吗?

#1


2  

Something you can do is to have multiple Reads from PubSub, where each reads from one subscription, and then Flatten their results.

你可以做的就是从PubSub获得多个Read,其中每个读取一个订阅,然后展平他们的结果。

Would that do what you need?

这会做你需要的吗?