在Python中的数据流中写入动态目标到云存储

时间:2021-04-30 15:38:27

I was trying to read from a big file in cloud storage and shard them according to a given field.

我试图从云存储中的大文件中读取并根据给定字段对它们进行分片。

I'm planning to Read | Map(lambda x: (x[key field], x)) | GroupByKey | Write to file with the name of the key field.

我打算阅读| Map(lambda x:(x [key field],x))| GroupByKey |使用键字段的名称写入文件。

However I couldn't find a way to write dynamically to cloud storage. Is this functionality supported?

但是我找不到动态写入云存储的方法。是否支持此功能?

Thank you, Yiqing

谢谢你,易清

1 个解决方案

#1


1  

Yes, you can use the FileSystems API to create the files.

是的,您可以使用FileSystems API来创建文件。

#1


1  

Yes, you can use the FileSystems API to create the files.

是的,您可以使用FileSystems API来创建文件。