Ceph object Gateway 之 S3 API

时间：2022-11-17 00:58:35

s3配置

创建用户

radosgw-admin user create --uid cjid --display-name cj

返回： 

"user": "cjid",
"access_key": "77SH7KHBF310F4NTIA3A",
"secret_key": "THxUb5kcWBObET3LwkR0ObVjgzzzf28czqp161Nj"

安装s3cmd

yum install s3cmd

修改s3cmd 配置

~/.s3cfg

[default]
access_key = 77SH7KHBF310F4NTIA3A
secret_key = THxUb5kcWBObET3LwkR0ObVjgzzzf28czqp161Nj
host_base = 192.168.164.221:5588
host_bucket = 192.168.164.221:5588/%(bucket)
cloudfront_host = 192.168.164.221:5588
use_https = False

运行s3cmd ls, 可显示结果说明配置成功

注：端口配置需要查看 /etc/ceph/ceph.conf, rgw frontends = "beast port=5588", 确认rgw服务启动

List Bucket

描述：列出桶

运行命令：s3cmd ls [s3://BUCKET[/PREFIX]]

示例：

Ceph object Gateway 之 S3 API


Delete Bucket

描述：删除桶

运行命令：s3cmd rb s3://BUCKET

示例：

Ceph object Gateway 之 S3 API

Create Bucket

描述：创建桶

运行命令：s3cmd mb s3://BUCKET

示例：

Ceph object Gateway 之 S3 API


Bucket Lifecycle

设置桶生命周期

运行命令：s3cmd setlifecycle FILE s3://BUCKET

获取桶生命周期策略

运行命令：s3cmd getlifecycle s3://BUCKET

删除桶生命周期策略

运行命令：s3cmd dellifecycle s3://BUCKET

示例xml：

<?xml version="1.0"?>
<LifecycleConfiguration>
<Rule>
<ID>prefix expiration</ID>
<Filter>
<Prefix>dir1</Prefix>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>120</Days>
</Expiration>
<Filter>
<Prefix>dir2</Prefix>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>

示例：

Ceph object Gateway 之 S3 API


Bucket Replication

描述：桶复制

运行命令：s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

示例：

Ceph object Gateway 之 S3 API


Policy (Buckets, Objects)

描述：桶，对象策略 (ACLs(访问控制列表)支持)

添加,修改桶策略

运行命令：s3cmd setpolicy FILE s3://BUCKET

删除桶策略

运行命令：s3cmd delpolicy s3://BUCKET

示例：

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred:subuser"]},
    "Action": "s3:PutObjectAcl",
    "Resource": [
      "arn:aws:s3:::happybucket/*"
    ]
  }]
}

获取桶策略

运行命令: 

GET /{bucket}/?policy= HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

示例：

Ceph object Gateway 之 S3 API

注：Principal字段见官网说明：

​​https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-bucket-user-policy-specifying-principal-intro.html​​

IAM user policies：IAM user policies for controlling user access to Amazon S3.

Bucket Website

描述：桶网站

创建website

运行命令：s3cmd ws-create s3://BUCKET

删除website

运行命令：s3cmd ws-delete s3://BUCKET

website信息

运行命令：s3cmd ws-info s3://BUCKET


Bucket ACLs (Get, Put)

获取桶访问控制列表

GET /?acl HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

例子：

Ceph object Gateway 之 S3 API

设置桶访问控制列表

s3cmd setacl s3://BUCKET[/OBJECT]

PUT /?acl HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write: GrantWrite
x-amz-grant-write-acp: GrantWriteACP
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml versinotallow="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<AccessControlList>
<Grant>
<Grantee>
<DisplayName>string</DisplayName>
<EmailAddress>string</EmailAddress>
<ID>string</ID>
<xsi:type>string</xsi:type>
<URI>string</URI>
</Grantee>
<Permission>string</Permission>
</Grant>
</AccessControlList>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
</AccessControlPolicy>


Bucket Location

描述：获取桶的位置

运行命令：

GET /?location HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

存储URL: protocol://service-code.region-code.amazonaws.com

region-code对应的桶的位置

示例：

Ceph object Gateway 之 S3 API

Bucket Notification

描述：桶通知

兼容性：AWS支持（SNS,SQS,Lambda）, HTTP/S, Kafka and AMQP

s3消息格式：(其中requestParameters.sourceIPAddress要为空 )

{  
   "Records":[  
      {  
         "eventVersion":"2.2",
         "eventSource":"aws:s3",
         "awsRegion":"us-west-2",
         "eventTime":"The time, in ISO-8601 format, for example, 1970-01-01T00:00:00.000Z, when Amazon S3 finished processing the request",
         "eventName":"event-type",
         "userIdentity":{  
            "principalId":"Amazon-customer-ID-of-the-user-who-caused-the-event"
         },
         "requestParameters":{  
            "sourceIPAddress":"ip-address-where-request-came-from"
         },
         "responseElements":{  
            "x-amz-request-id":"Amazon S3 generated request ID",
            "x-amz-id-2":"Amazon S3 host that processed the request"
         },
         "s3":{  
            "s3SchemaVersion":"1.0",
            "configurationId":"ID found in the bucket notification configuration",
            "bucket":{  
               "name":"bucket-name",
               "ownerIdentity":{  
                  "principalId":"Amazon-customer-ID-of-the-bucket-owner"
               },
               "arn":"bucket-ARN"
            },
            "object":{  
               "key":"object-key",
               "size":"object-size in bytes",
               "eTag":"object eTag",
               "versionId":"object version if bucket is versioning-enabled, otherwise null",
               "sequencer": "a string representation of a hexadecimal value used to determine event sequence, only used with PUTs and DELETEs"
            }
         },
         "glacierEventData": {
            "restoreEventData": {
               "lifecycleRestorationExpiryTime": "The time, in ISO-8601 format, for example, 1970-01-01T00:00:00.000Z, of Restore Expiry",
               "lifecycleRestoreStorageClass": "Source storage class for restore"
            }
         }
      }
   ]
}

事件不支持列表：

s3:ObjectSynced:Delete
s3:ObjectSynced:DeletionMarkerCreated
s3:ObjectLifecycle:Expiration:AbortMultipartUpload
s3:ObjectRestore:Post
s3:ObjectRestore:Complete
s3:ReducedRedundancyLostObject

消息队列TOPIC 行为仅支持:

CreateTopic
DeleteTopic
ListTopics


Bucket Object Versions

描述：桶对象版本

运行命令：

GET /?versions HTTP/1.1
Host: BucketName.s3.<Region>.amazonaws.com
Date: Wed, 28 Oct 2009 22:32:00 +0000
Authorization: authorization string (see Authenticating Requests (AWS Signature Version
    4))

运行结果：

Ceph object Gateway 之 S3 API


Get Bucket Info (HEAD)

描述：获取桶信息

运行命令：s3cmd info s3://BUCKET

例子：

Ceph object Gateway 之 S3 API


Bucket Request Payment

描述：获取桶请求支付配置

运行命令：

GET /?requestPayment HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner




Ceph object Gateway 之 S3 API

Put Object

描述：写入对象

运行命令：s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]

示例：

Ceph object Gateway 之 S3 API


Delete Object

描述：删除对象

运行命令：

  • s3cmd del s3://BUCKET/OBJECT
  • s3cmd rm s3://BUCKET/OBJECT

示例：

Ceph object Gateway 之 S3 API


Get Object

描述：获取对象

运行命令：s3cmd get s3://BUCKET/OBJECT LOCAL_FILE


Object ACLs (Get, Put)

得到对象访问控制列表

运行命令：

GET /{Key+}?acl&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-request-payer: RequestPayer
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置对象访问控制列表

运行命令：

s3cmd setacl s3://BUCKET[/OBJECT]

PUT /{Key+}?acl&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-acl: ACL
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-grant-full-control: GrantFullControl
x-amz-grant-read: GrantRead
x-amz-grant-read-acp: GrantReadACP
x-amz-grant-write: GrantWrite
x-amz-grant-write-acp: GrantWriteACP
x-amz-request-payer: RequestPayer
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml version="1.0" encoding="UTF-8"?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<AccessControlList>
<Grant>
<Grantee>
<DisplayName>string</DisplayName>
<EmailAddress>string</EmailAddress>
<ID>string</ID>
<xsi:type>string</xsi:type>
<URI>string</URI>
</Grantee>
<Permission>string</Permission>
</Grant>
</AccessControlList>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
</AccessControlPolicy>

Get Object Info (HEAD)

描述：获取桶对象信息

运行命令：s3cmd info s3://BUCKET[/OBJECT]

Ceph object Gateway 之 S3 API


Copy Object

描述：拷贝桶对象信息

运行命令：s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]


Ceph object Gateway 之 S3 API


Multipart Uploads

描述：多分片上传

运行命令：s3cmd multipart s3://BUCKET [Id]

使用场景：对象超过100m的时候

好处：

  • 通过并发提高吞吐量
  • 在有网络问题的时候可以快速恢复
  • 能暂停与恢复上传
  • 可以在接受对象的同时上传对象


Object Tagging

获取对象标签

运行命令：

GET /{Key+}?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner
x-amz-request-payer: RequestPayer

删除对象标签

运行命令：

DELETE /{Key+}?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置对象标签

运行命令：

PUT /{Key+}?tagging&versionId=VersionId HTTP/1.1
Host: Bucket.s3.amazonaws.com
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-expected-bucket-owner: ExpectedBucketOwner
x-amz-request-payer: RequestPayer
<?xml versinotallow="1.0" encoding="UTF-8"?>
<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<TagSet>
<Tag>
<Key>string</Key>
<Value>string</Value>
</Tag>
</TagSet>

例子：

Ceph object Gateway 之 S3 API


Bucket Tagging

获取桶标签

GET /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

删除桶标签

DELETE /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-expected-bucket-owner: ExpectedBucketOwner

设置桶标签

PUT /?tagging HTTP/1.1
Host: Bucket.s3.amazonaws.com
Content-MD5: ContentMD5
x-amz-sdk-checksum-algorithm: ChecksumAlgorithm
x-amz-expected-bucket-owner: ExpectedBucketOwner
<?xml versinotallow="1.0" encoding="UTF-8"?>
<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<TagSet>
<Tag>
<Key>string</Key>
<Value>string</Value>
</Tag>
</TagSet>
</Tagging>

例子：

Ceph object Gateway 之 S3 API


Storage Class

描述：用来定制对象数据的放置(placement)，s3的桶生命周期的规则可以在storage classes之间自动进行对象转换

例子：

zonegroup的placement target 列举了一个初始化名字为STANDARD类

$radosgw-admin zonegroup get

{
    "id": "ab01123f-e0df-4f29-9d71-b44888d67cd5",
    "name": "default",
    "api_name": "default",
    ...
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": [],
            "storage_classes": [
                "STANDARD"
            ]
        }
    ],
    "default_placement": "default-placement",
    ...
}

zone 配置负责提供一个data_pool 的名字为每个zonegroup 的存储类

$ radosgw-admin zone get
{
    "id": "557cdcee-3aae-4e9e-85c7-2f86f5eddb1f",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    ...
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "inline_data": true
            }
        }
    ],
    ...
}


S3cmd 不支持的命令

参考该文档，用postman 进行调用，参数已经配置好，需要修改部分参数即可

​​​​​​https://documenter.getpostman.com/view/10394726/SzYbxHEf​​

例子（红框里的内容是需要修改的）：

Ceph object Gateway 之 S3 API

特殊名词

SNS (Simple Notification Service)

是一种完全托管的发布/订阅消息收发和移动通知服务，用于协调向订阅终端节点和客户端的消息分发。

SQS (Simple Queue Service)

可以轻松分离和扩展微服务，分布式系统和无服务应用程序，对程序进行解耦

Lambda

Lambda 是一种高度可用、无服务器、事件驱动的计算服务



标签：

相关文章