I'm using go in order to interact with cloud storage .
我正在使用go来与云存储进行交互。
I can't use gsutil from the app engine and delete with the rm command ?
我不能使用app引擎中的gsutil并使用rm命令删除?
I can delete one object with DeleteObject or iterate over a range of objects and delete each one , but i`m looking for another solution something like DeleteMulti in Datastor.
我可以使用DeleteObject删除一个对象或迭代一系列对象并删除每个对象,但我正在寻找另一个解决方案,如Datastor中的DeleteMulti。
Do you have a better solution for multi deletion ?
你有更好的多删除解决方案吗?
1 个解决方案
#1
Each object that is deleted requires one call to GCS. Iterating over each object and calling delete is the easiest and likely best solution. If you need faster performance, you may want to send multiple delete requests to GCS at a time using multiple threads.
删除的每个对象都需要一次调用GCS。迭代每个对象并调用delete是最简单且最可能的最佳解决方案。如果您需要更快的性能,您可能希望使用多个线程一次向GCS发送多个删除请求。
If this is a significant performance issue for your app, there is, however, another way, which I hesitate to mention because it adds significant complexity and doesn't buy much extra performance. GCS supports batching calls together into a single connection. It likely won't be much faster than sending delete requests over several threads, but it does behave more like a DeleteMulti call.
如果这对您的应用程序来说是一个重要的性能问题,那么还有另一种方法,我不愿提及,因为它增加了显着的复杂性并且不会产生太多额外的性能。 GCS支持将呼叫一起批处理为单个连接。它可能不会比通过多个线程发送删除请求快得多,但它的行为更像是DeleteMulti调用。
Effectively, batch calls work by sending a multipart HTTP request to the /batch path, each part of which representing an HTTP call. A request to delete several objects would look like this:
实际上,批量调用通过向/ batch路径发送多部分HTTP请求来工作,每个部分代表一个HTTP调用。删除多个对象的请求如下所示:
POST /batch HTTP/1.1
Host: www.googleapis.com
Content-Length: content_length
Content-Type: multipart/mixed; boundary="===============7330845974216740156=="
Authorization: Bearer oauth2_token
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+1>
DELETE /storage/v1/b/example-bucket/o/obj1 HTTP/1.1
accept: application/json
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+2>
DELETE /storage/v1/b/example-bucket/o/obj2 HTTP/1.1
accept: application/json
--===============7330845974216740156==--
There's more documentation on it here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch
这里有更多文档:https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch
But, again, I recommend just sending individual delete requests. Batch calls are not atomic, which means some deletes might succeed while others fail. In the event one of the batch delete operations fails, you'll need to parse the batch response message to figure out which call failed so that you can retry it, which is very likely not worth the effort for the gain you'll get back.
但是,我再次建议只发送个别删除请求。批量调用不是原子的,这意味着某些删除可能会成功,而其他删除则会失败。如果其中一个批量删除操作失败,您将需要解析批处理响应消息以确定哪个调用失败,以便您可以重试它,这很可能不值得您获得的收益。
#1
Each object that is deleted requires one call to GCS. Iterating over each object and calling delete is the easiest and likely best solution. If you need faster performance, you may want to send multiple delete requests to GCS at a time using multiple threads.
删除的每个对象都需要一次调用GCS。迭代每个对象并调用delete是最简单且最可能的最佳解决方案。如果您需要更快的性能,您可能希望使用多个线程一次向GCS发送多个删除请求。
If this is a significant performance issue for your app, there is, however, another way, which I hesitate to mention because it adds significant complexity and doesn't buy much extra performance. GCS supports batching calls together into a single connection. It likely won't be much faster than sending delete requests over several threads, but it does behave more like a DeleteMulti call.
如果这对您的应用程序来说是一个重要的性能问题,那么还有另一种方法,我不愿提及,因为它增加了显着的复杂性并且不会产生太多额外的性能。 GCS支持将呼叫一起批处理为单个连接。它可能不会比通过多个线程发送删除请求快得多,但它的行为更像是DeleteMulti调用。
Effectively, batch calls work by sending a multipart HTTP request to the /batch path, each part of which representing an HTTP call. A request to delete several objects would look like this:
实际上,批量调用通过向/ batch路径发送多部分HTTP请求来工作,每个部分代表一个HTTP调用。删除多个对象的请求如下所示:
POST /batch HTTP/1.1
Host: www.googleapis.com
Content-Length: content_length
Content-Type: multipart/mixed; boundary="===============7330845974216740156=="
Authorization: Bearer oauth2_token
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+1>
DELETE /storage/v1/b/example-bucket/o/obj1 HTTP/1.1
accept: application/json
--===============7330845974216740156==
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <b29c5de2-0db4-490b-b421-6a51b598bd22+2>
DELETE /storage/v1/b/example-bucket/o/obj2 HTTP/1.1
accept: application/json
--===============7330845974216740156==--
There's more documentation on it here: https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch
这里有更多文档:https://cloud.google.com/storage/docs/json_api/v1/how-tos/batch
But, again, I recommend just sending individual delete requests. Batch calls are not atomic, which means some deletes might succeed while others fail. In the event one of the batch delete operations fails, you'll need to parse the batch response message to figure out which call failed so that you can retry it, which is very likely not worth the effort for the gain you'll get back.
但是,我再次建议只发送个别删除请求。批量调用不是原子的,这意味着某些删除可能会成功,而其他删除则会失败。如果其中一个批量删除操作失败,您将需要解析批处理响应消息以确定哪个调用失败,以便您可以重试它,这很可能不值得您获得的收益。