I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.
我试图从命令行将文件放入Google云端存储(GCS)存储桶。在稍后阶段,这将在用户端的部署脚本中使用,而不使用任何类型的用户可见身份验证。
So far I generate a signed url like this:
到目前为止,我生成了一个这样的签名网址:
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile
which will generate something like
这会产生类似的东西
The next step (at the user end) would be curl uploading the file with a PUT request. Like so:
下一步(在用户端)将使用PUT请求卷曲上传文件。像这样:
curl -X PUT --data-binary @testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'
I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl throws the server's response with error messages like this at me:
我可以使用它来处理存储桶中的现有文件和GET请求(用于下载),但它似乎不适用于上载。 curl抛出服务器的响应,并在此处显示错误消息:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
Check your Google secret key and signing method.</Message>
<StringToSign>PUT
application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>
And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.
这对我来说很有意义,因为很明显,我不只是提出一个PUT请求,而是一个特定大小的特定文件,而'gsutil signurl'计算的签名在这个时候不会知道这些细节。计算。
Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl request?
不知何故,我受到了这种印象(例如,基于gsutil signurl文档中描述的最后一个使用案例以及帖子如何允许匿名上传到云存储),应该可以生成用于上载目的的通用签名URL,然后以后再用我只是弄错了这一点,还是有办法解决卷曲请求?
Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.
对此有任何想法表示赞赏。但是,我希望这可以使用“最小工具”,即理想情况下只有shell和curl,但没有其他编程语言。
EDIT: Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that
编辑:通过制定确切的问题来组织一个人的想法是迈向解决方案的第一步。我现在意识到了
curl -X PUT -T - [request-url] < testfile
does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use
确实解决了眼前的问题。但是,这意味着如果多个用户使用相同的签名URL,则会写入同一文件。文档建议您可以在创建签名URL时省略对象名称,即使用
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/
This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.
据推测,这将允许任何拥有最终签名网址的人将任何类型的任何对象放入我的桶中。只有我没有得到这个工作,因为我没有看到你怎么能告诉GCS你实际写的是哪个对象。
3 个解决方案
#1
If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:
如果资源未指定单个对象,则可以通过向请求添加URL参数以及对象的名称来单独执行此操作。例如:
curl -X PUT -T - [request-url]?name=[object-name] < testfile
curl -X PUT -T - [request-url]?name = [object-name]
This surely works with storage/v1, although I have not tried myself with a signed URL yet.
这肯定适用于storage / v1,虽然我还没有尝试使用签名的URL。
#2
This was driving me mad too. Turns out it was the binary-file part curl command. Try:
这让我很生气。原来这是二进制文件part curl命令。尝试:
curl -X PUT --upload-file me.jpeg $SIGNED_URL
#3
I encountered the similar problem(403 forbidden).
It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side
So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.
我遇到了类似的问题(403禁止)。事实证明,我使用它来编组每个响应的json库将替换为安全问题。所以url在程序中可能是正确的但在客户端是无效的所以我猜你的url的Signature查询字符串中可能存在一些字符串编码错误,因为与我的相比,签名字符串更难以检测错误。
#1
If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:
如果资源未指定单个对象,则可以通过向请求添加URL参数以及对象的名称来单独执行此操作。例如:
curl -X PUT -T - [request-url]?name=[object-name] < testfile
curl -X PUT -T - [request-url]?name = [object-name]
This surely works with storage/v1, although I have not tried myself with a signed URL yet.
这肯定适用于storage / v1,虽然我还没有尝试使用签名的URL。
#2
This was driving me mad too. Turns out it was the binary-file part curl command. Try:
这让我很生气。原来这是二进制文件part curl命令。尝试:
curl -X PUT --upload-file me.jpeg $SIGNED_URL
#3
I encountered the similar problem(403 forbidden).
It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side
So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.
我遇到了类似的问题(403禁止)。事实证明,我使用它来编组每个响应的json库将替换为安全问题。所以url在程序中可能是正确的但在客户端是无效的所以我猜你的url的Signature查询字符串中可能存在一些字符串编码错误,因为与我的相比,签名字符串更难以检测错误。