90秒后取消检索Tabledata

I am trying to implement parallel retrieve of data from table using API method Tabledata:list.

我正在尝试使用API​​方法Tabledata:list实现从表中并行检索数据。

For this I am using .NET NuGet package Google.Apis.Bigquery.v2 and from there either method TabledataResource.ListRequest.Execute() or method TabledataResource.ListRequest.ExecuteAsync() and waiting for task to complete.

为此我使用.NET NuGet包Google.Apis.Bigquery.v2并从那里使用方法TabledataResource.ListRequest.Execute()或方法TabledataResource.ListRequest.ExecuteAsync()并等待任务完成。

In both cases calling of this method thrown exception, that task was canceled. It happened roughly 90 seconds after calling method.

在调用此方法抛出异常的两种情况下,该任务都被取消。它在调用方法后大约90秒发生。

Have you experienced this? Do you have some suggestions how to change this limit? Or for some workaround?

你有没有经历过这个?您对如何更改此限制有一些建议吗?或者一些解决方法?

ParallelOptions po = new ParallelOptions
{
    MaxDegreeOfParallelism = 30
};
List<Int32> results = new List<Int32>();
const Int32 __NUM_OF_ITEMS = 10000;
Parallel.ForEach(Enumerable.Range(0, 30), po, index =>
{
    TabledataResource.ListRequest listRequest = _service.Tabledata.List(
        resultTable.ProjectId,
        resultTable.DatasetId,
        resultTable.TableId
    );
    listRequest.PageToken = pageToken;
    listRequest.StartIndex = (UInt64?) index * __NUM_OF_ITEMS;
    TableDataList tdl = listRequest.Execute();
});

EDITED:

I did further investigation and these are times which it took to retrieve 100 items from my table using different startIndex. PageToken was null in all cases.

我做了进一步的调查,这些是使用不同的startIndex从我的表中检索100个项目所花费的时间。在所有情况下,PageToken都为null。

[INF      2015-05-15 09:26:33.283] Testing retrieval speed.
[INF      2015-05-15 09:26:34.431] startIndex=0 retrievedItems=100 timeSpent=1147 ms.
[INF      2015-05-15 09:26:51.210] startIndex=1000000 retrievedItems=100 timeSpent=16779 ms.
[INF      2015-05-15 09:27:11.166] startIndex=2000000 retrievedItems=100 timeSpent=19955 ms.
[INF      2015-05-15 09:27:36.400] startIndex=3000000 retrievedItems=100 timeSpent=25234 ms.
[INF      2015-05-15 09:28:22.744] startIndex=4000000 retrievedItems=100 timeSpent=46344 ms.
[INF      2015-05-15 09:29:36.057] startIndex=5000000 retrievedItems=100 timeSpent=73314 ms.
[INF      2015-05-15 09:30:33.153] startIndex=6000000 retrievedItems=100 timeSpent=57095 ms.
[INF      2015-05-15 09:31:44.062] startIndex=7000000 retrievedItems=100 timeSpent=70910 ms.

My table has 14M rows and I failed to get result for starting index 8M (exception described above was thrown).

我的表有14M行,我没有得到开始索引8M的结果(抛出上面描述的异常)。

1 个解决方案

#1

Where does your pageToken come from? There is a setMaxLimit (https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/Bigquery.Tabledata.List.html#setMaxResults(java.lang.Long)) to set the max number of results to return. Maybe you can try to issue smaller requests to see if it would success.

你的pageToken来自哪里?有一个setMaxLimit(https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/Bigquery.Tabledata.List.html#setMaxResults (java.lang.Long))设置要返回的最大结果数。也许您可以尝试发出较小的请求,看它是否会成功。