如何使用CLI删除AWS S3中的版本化存储桶？

I have tried both s3cmd:

我试过两个s3cmd：

$ s3cmd -r -f -v del s3://my-versioned-bucket/

And the AWS CLI:

和AWS CLI：

$ aws s3 rm s3://my-versioned-bucket/ --recursive

But both of these commands simply add DELETE markers to S3. The command for removing a bucket also doesn't work (from the AWS CLI):

但是这两个命令都只是将DELETE标记添加到S3。删除存储桶的命令也不起作用（来自AWS CLI）：

$ aws s3 rb s3://my-versioned-bucket/ --force
Cleaning up. Please wait...
Completed 1 part(s) with ... file(s) remaining
remove_bucket failed: s3://my-versioned-bucket/ A client error (BucketNotEmpty) occurred when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.

Ok... how? There's no information in their documentation for this. S3Cmd says it's a 'fully-featured' S3 command-line tool, but it makes no reference to versions other than its own. Is there any way to do this without using the web interface, which will take forever and requires me to keep my laptop on?

好吧怎么办？他们的文档中没有关于此的信息。 S3Cmd称它是一个“全功能”的S3命令行工具，但它没有引用除自己以外的版本。有没有办法在不使用网络界面的情况下做到这一点，这需要永远，并要求我保持我的笔记本电脑？

7 个解决方案

#1

One way to do it is iterate through the versions and delete them. A bit tricky on the CLI, but as you mentioned Java, that would be more straightforward:

一种方法是遍历版本并删除它们。在CLI上有点棘手，但正如你提到的Java那样，那会更直接：

AmazonS3Client s3 = new AmazonS3Client();
String bucketName = "deleteversions-"+UUID.randomUUID();

//Creates Bucket
s3.createBucket(bucketName);

//Enable Versioning
BucketVersioningConfiguration configuration = new BucketVersioningConfiguration(ENABLED);
s3.setBucketVersioningConfiguration(new SetBucketVersioningConfigurationRequest(bucketName, configuration ));

//Puts versions
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("some-bytes".getBytes()), null);
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("other-bytes".getBytes()), null);

//Removes all versions
for ( S3VersionSummary version : S3Versions.inBucket(s3, bucketName) ) {
    String key = version.getKey();
    String versionId = version.getVersionId();          
    s3.deleteVersion(bucketName, key, versionId);
}

//Removes the bucket
s3.deleteBucket(bucketName);
System.out.println("Done!");

You can also batch delete calls for efficiency if needed.

如果需要，您还可以批量删除调用以提高效率。

#2

I ran into the same limitation of the AWS CLI. I found the easiest solution to be to use Python and boto3:

我遇到了AWS CLI的相同限制。我发现最简单的解决方案是使用Python和boto3：

BUCKET = 'your-bucket-here'

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET)
bucket.object_versions.delete()

# if you want to delete the now-empty bucket as well, uncomment this line:
#bucket.delete()

A previous version of this answer used boto but that solution had performance issues with large numbers of keys as Chuckles pointed out.

这个答案的先前版本使用了boto但是该解决方案存在大量密钥的性能问题，正如Chuckles指出的那样。

#3

Using boto3 it's even easier than with the proposed boto solution to delete all object versions in an S3 bucket:

使用boto3比使用建议的boto解决方案更容易删除S3存储桶中的所有对象版本：

#!/usr/bin/env python
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
bucket.object_versions.all().delete()

Works fine also for very large amounts of object versions, although it might take some time in that case.

对于非常大量的对象版本也可以正常工作，尽管在这种情况下可能需要一些时间。

#4

You can delete all the objects in the versioned s3 bucket. But I don't know how to delete specify objects. aws s3api delete-objects --bucket <value> --delete "$(aws s3api list-object-versions --bucket <value> | jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

您可以删除版本化s3存储桶中的所有对象。但我不知道如何删除指定对象。 aws s3api delete-objects --bucket --delete“$（aws s3api list-object-versions --bucket | jq'{Objects：[.Versions [] | {Key：.Key，VersionId： .VersionId}]，安静：假}'）“

Hope it is helpful to you.

希望它对你有所帮助。

#5

I ran into issues with Abe's solution as the list_buckets generator is used to create a massive list called all_keys and I spent an hour without it ever completing. This tweak seems to work better for me, I had close to a million objects in my bucket and counting!

我遇到了Abe解决方案的问题，因为list_buckets生成器用于创建一个名为all_keys的大量列表，我花了一个小时没有完成。这个调整似乎对我来说效果更好，我的桶里有近百万个物品并且还在数！

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket("your-bucket-name-here")

chunk_counter = 0 #this is simply a nice to have
keys = []
for key in bucket.list_versions():
    keys.append(key)
    if len(keys) > 1000:
        bucket.delete_keys(keys)
        chunk_counter += 1
        keys = []
        print("Another 1000 done.... {n} chunks so far".format(n=chunk_counter))

#bucket.delete() #as per usual uncomment if you're sure!

Hopefully this helps anyone else encountering this S3 nightmare!

希望这有助于其他人遇到这个S3噩梦！

#6

For deleting specify object(s), using jq filter.
要删除指定对象，请使用jq过滤器。
You may need cleanup the 'DeleteMarkers' not just 'Versions'.
您可能需要清理'DeleteMarkers'而不仅仅是'版本'。
Using $() instead of ``, you may embed variables for bucket-name and key-value.
使用$（）代替``，可以为bucket-name和key-value嵌入变量。

aws s3api delete-objects --bucket bucket-name --delete "$(aws s3api list-object-versions --bucket bucket-name | jq -M '{Objects: [.["Versions","DeleteMarkers"][]|select(.Key == "key-value")| {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

#7

Here is a one liner you can just cut and paste into the command line to delete all versions and delete markers (it requires aws tools, replace yourbucket-name-backup with your bucket name)

这是一个单行程，您只需剪切并粘贴到命令行中删除所有版本并删除标记（它需要aws工具，用您的存储桶名称替换yourbucket-name-backup）

echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket yourbucket-name-backup | grep -E "^VERSIONS" | awk '{print "aws s3api delete-object --bucket yourbucket-name-backup --key "$4" --version-id "$8";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh; echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket yourbucket-name-backup | grep -E "^DELETEMARKERS" | grep -v "null" | awk '{print "aws s3api delete-object --bucket yourbucket-name-backup --key "$3" --version-id "$5";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;

then you could use:

然后你可以使用：

aws s3 rb s3://bucket-name --force

aws s3 rb s3：// bucket-name --force

#1