当s3配置文件设置enable_multipart = True时,put mutipart功能被启动
在用户上传一个object时,S3会按照配置文件要求对文件进行分块,参数如下:
multipart_chunk_size_mb = 15-----------------15M一块
multipart_max_chunks = 10000-----------------最大块数10000
运行流程
1、S3下发一个post消息,向radosgw申请upload id,后面分块上传都才有这个upload id做标示
2、radosgw返回一个upload id给s3使用
3、S3开始按照配置好的分块大小,将文件分块上传(本例分三块,总文件大小45M),这里下发的是PUT消息,并且代入了partNmuber和uploadId
4、S3下发POST消息,通知服务端上传文件完成
以上是mutipart上传文件S3部分下发的消息流程
下面是几个常用的命令
上传完毕后upload Id被删除(如果中途出现失败的情况,可采用./s3cmd multiparts3://ZHOU 的命令形式查询upload id)
手动删除未被销毁的upload Id 命令为./s3cmd abortmp s3://ZHOU/pydev.tar.gz upload id
下面是我修改了分块大小,将原来的15M修改为5M,而且在分块上传时,手动暂停了后面的几个上传,这样就可以发现upload id 保存在缓存中
采用命令./s3cmd multipart s3://ZHOU 来查询bucket中的upload Id
采用命令s3cmd listmp s3://ZHOU/pydev.tar.gz 2~Vz0wGLS0veZf2gXV1eCcogaz_3kmgfg 来查询upload Id的详细信息,可以看到上传了7个部分,每部分都是5M
修改配置参数put_continue = True,可以在原来的基础之上续传,续传是从以前的最后一块继续上传,图中上传了剩余的第8和第9块,这时候upload id被删除了
后台采用4M切片
上传一个文件
查询结果如下:
root>/home/ceph/src>./rados -p default.rgw.buckets.data ls
2016-06-24 18:12:41.215614 7fe61260aa40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-24 18:12:41.216001 7fe61260aa40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-24 18:12:41.241489 7fe61260aa40 -1 WARNING: the following dangerous and experimental features are enabled: *
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_4
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1_pydev.tar.gz
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_4
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3_1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_1
整理后可以看到
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1_pydev.tar.gz----------------------manifest
-------------每20M切片成1个mutipart+N个shadow
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.1_4
--------------下一个20M
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_1
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_2
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.2_4
------------下一个mutipart
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__multipart_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3
5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3_1
查看最后一个切片的大小
root>/home/ceph/src>./rados -p default.rgw.buckets.data stat5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3_1
2016-06-24 18:35:04.201558 7fc8cff71a40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-24 18:35:04.201797 7fc8cff71a40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-24 18:35:04.222031 7fc8cff71a40 -1 WARNING: the following dangerous and experimental features are enabled: *
default.rgw.buckets.data/5c4bca48-8a5e-43f0-81f1-1fcf7990a180.24109.1__shadow_pydev.tar.gz.2~sgJQ5EHwh4O5EPaDIR4woskDm9XrUYE.3_1 mtime 2016-06-24 18:07:07.000000,size 719778