I have created a dataflow with ~30 pipelines all of which subscribe to their own topics on PubSub. When I try to launch, a random subscription always fails. However if I split my pipelines between several dataflows all the subscriptions are successful.
我创建了一个包含~30个管道的数据流,所有这些管道都在PubSub上订阅了他们自己的主题。当我尝试启动时,随机订阅总是失败。但是,如果我在多个数据流之间拆分管道,则所有订阅都会成功。
Is there a limit to the number of topics a single dataflow can sub to or the number concurrent subscription requests that can be made?
单个数据流可以提交的主题数量是否有限制,或者可以进行的并发订阅请求数量是否有限制?
Failing job: 2016-06-03_07_27_29-2535656819775551181
失败的工作:2016-06-03_07_27_29-2535656819775551181
One of the succesful ones: 2016-06-03_07_37_20-7740562448710226511
其中一个成功的:2016-06-03_07_37_20-7740562448710226511
2 个解决方案
#1
1
Dataflow has a known bug that causes subscription creation to be flaky; we are working on fixing it. In the meantime, launching pipelines with fewer pubsub inputs should be less likely to exhibit the issue.
Dataflow有一个已知的错误,导致订阅创建不稳定;我们正在努力修复它。与此同时,启动具有较少pubsub输入的管道应该不太可能出现问题。
#2
#1
1
Dataflow has a known bug that causes subscription creation to be flaky; we are working on fixing it. In the meantime, launching pipelines with fewer pubsub inputs should be less likely to exhibit the issue.
Dataflow有一个已知的错误,导致订阅创建不稳定;我们正在努力修复它。与此同时,启动具有较少pubsub输入的管道应该不太可能出现问题。