使用Ruby aws-sdk跟踪上传文件到S3的进度

Firstly, I am aware that there are quite a few questions that are similar to this one in SO. I have read most, if not all of them, over the past week. But I still can't make this work for me.

首先,我知道在SO中有很多类似于这个的问题。在过去的一周里,我读过大多数(如果不是全部的话)。但我仍然不能为我做这项工作。

I am developing a Ruby on Rails app that allows users to upload mp3 files to Amazon S3. The upload itself works perfectly, but a progress bar would greatly improve user experience on the website.

我正在开发一个Ruby on Rails应用程序,允许用户将mp3文件上传到Amazon S3。上传本身效果很好,但进度条可以极大地改善网站上的用户体验。

I am using the aws-sdk gem which is the official one from Amazon. I have looked everywhere in its documentation for callbacks during the upload process, but I couldn't find anything.

我正在使用aws-sdk gem,这是来自亚马逊的官方宝石。在上传过程中,我在其文档中随处可见回调,但我找不到任何内容。

The files are uploaded one at a time directly to S3 so it doesn't need to load it into memory. No multiple file upload necessary either.

文件一次一个地上传到S3,因此不需要将其加载到内存中。也不需要多文件上传。

I figured that I may need to use JQuery to make this work and I am fine with that. I found this that looked very promising: https://github.com/blueimp/jQuery-File-Upload And I even tried following the example here: https://github.com/ncri/s3_uploader_example

我想我可能需要使用JQuery来完成这项工作,我很好。我发现这看起来很有前途:https://github.com/blueimp/jQuery-File-Upload我甚至试过这里的例子:https://github.com/ncri/s3_uploader_example

But I just could not make it work for me.

但我无法让它对我有用。

The documentation for aws-sdk also BRIEFLY describes streaming uploads with a block:

aws-sdk的文档还简要描述了使用块的流上传:

  obj.write do |buffer, bytes|
     # writing fewer than the requested number of bytes to the buffer
     # will cause write to stop yielding to the block
  end

But this is barely helpful. How does one "write to the buffer"? I tried a few intuitive options that would always result in timeouts. And how would I even update the browser based on the buffering?

但这几乎没有帮助。如何“写入缓冲区”?我尝试了一些总是会导致超时的直观选项。我怎么能基于缓冲更新浏览器?

Is there a better or simpler solution to this?

有没有更好或更简单的解决方案?

Thank you in advance. I would appreciate any help on this subject.

先谢谢你。我很感激有关这个问题的任何帮助。

2 个解决方案

#1

The "buffer" object yielded when passing a block to #write is an instance of StringIO. You can write to the buffer using #write or #<<. Here is an example that uses the block form to upload a file.

将块传递给#write时产生的“缓冲区”对象是StringIO的一个实例。您可以使用#write或#< <写入缓冲区。以下是使用块表单上传文件的示例。< p>

file = File.open('/path/to/file', 'r')

obj = s3.buckets['my-bucket'].objects['object-key']
obj.write(:content_length => file.size) do |buffer, bytes|
  buffer.write(file.read(bytes))
  # you could do some interesting things here to track progress
end

file.close

#2

After read the source code of the AWS gem, I've adapted (or mostly copy) the multipart upload method to yield the current progress based on how many chunks have been uploaded

在阅读了AWS gem的源代码之后,我已经调整(或主要是复制)了分段上传方法,以根据已上传的块数生成当前进度

s3 = AWS::S3.new.buckets['your_bucket']

file = File.open(filepath, 'r', encoding: 'BINARY')
file_to_upload = "#{s3_dir}/#{filename}"
upload_progress = 0

opts = {
  content_type: mime_type,
  cache_control: 'max-age=31536000',
  estimated_content_length: file.size,
}

part_size = self.compute_part_size(opts)

parts_number = (file.size.to_f / part_size).ceil.to_i
obj          = s3.objects[file_to_upload]

begin
    obj.multipart_upload(opts) do |upload|
      until file.eof? do
        break if (abort_upload = upload.aborted?)

        upload.add_part(file.read(part_size))
        upload_progress += 1.0/parts_number

        # Yields the Float progress and the String filepath from the
        # current file that's being uploaded
        yield(upload_progress, upload) if block_given?
      end
    end
end

The compute_part_size method is defined here and I've modified it to this:

这里定义了compute_part_size方法,我将其修改为:

def compute_part_size options

  max_parts = 10000
  min_size  = 5242880 #5 MB
  estimated_size = options[:estimated_content_length]

  [(estimated_size.to_f / max_parts).ceil, min_size].max.to_i

end

This code was tested on Ruby 2.0.0p0

此代码在Ruby 2.0.0p0上进行了测试

#1