如何使用node.js，Express和knox将文件从浏览器上传到Amazon S3？

I'm trying to find some example code that utilizes node.js, Express, and knox.

我试图找到一些利用node.js，Express和knox的示例代码。

The docs for Knox only give clear examples of how to upload a file already stored in the file system. https://github.com/learnboost/knox#readme

Knox的文档仅提供了如何上传已存储在文件系统中的文件的明确示例。 https://github.com/learnboost/knox#readme

Additionally, there a number of simple tutorials (even in Express itself) on how to upload files directly to express and save to the file system.

此外，还有许多简单的教程（即使在Express本身中），如何直接上传文件以表达和保存到文件系统。

What I'm having trouble finding is an example that lets you upload a client upload to a node server and have the data streamed directly to S3 rather than storing in the local file system first.

我找不到的是一个示例，它允许您将客户端上载上传到节点服务器，并将数据直接流式传输到S3，而不是先存储在本地文件系统中。

Can someone point me to a gist or other example that contains this kind of information?

有人能指出我要包含这类信息的要点或其他例子吗？

6 个解决方案

#1

All of the previous answers involve having the upload pass through your node.js server which is inefficient and unnecessary. Your node server does not have to handle the bandwidth or processing of uploaded files whatsoever because Amazon S3 allows uploads direct from the browser.

所有以前的答案都涉及让上传通过您的node.js服务器，这是低效且不必要的。您的节点服务器无需处理上传文件的带宽或处理，因为Amazon S3允许直接从浏览器上传。

Have a look at this blog post: http://blog.tcs.de/post-file-to-s3-using-node/

看看这篇博文：http：//blog.tcs.de/post-file-to-s3-using-node/

I have not tried the code listed there, but having looked over it, it appears solid and I will be attempting an implementation of it shortly ad will update this answer with my findings.

我没有尝试过那里列出的代码，但是看了之后，它看起来很稳固，我将尝试实施它很快就会用我的发现更新这个答案。

#2

Here is an example of streaming directly to s3 without ever touching your hard drive, using multiparty and knox:

这是一个使用multiparty和knox直接流式传输到s3而无需触摸硬盘的示例：

var http = require('http')
  , util = require('util')
  , multiparty = require('multiparty')
  , knox = require('knox')
  , Batch = require('batch')
  , PORT = process.env.PORT || 27372

var s3Client = knox.createClient({
  secure: false,
  key: process.env.S3_KEY,
  secret: process.env.S3_SECRET,
  bucket: process.env.S3_BUCKET,
});

var Writable = require('readable-stream').Writable;
util.inherits(ByteCounter, Writable);
function ByteCounter(options) {
  Writable.call(this, options);
  this.bytes = 0;
}

ByteCounter.prototype._write = function(chunk, encoding, cb) {
  this.bytes += chunk.length;
  cb();
};

var server = http.createServer(function(req, res) {
  if (req.url === '/') {
    res.writeHead(200, {'content-type': 'text/html'});
    res.end(
      '<form action="/upload" enctype="multipart/form-data" method="post">'+
      '<input type="text" name="path"><br>'+
      '<input type="file" name="upload"><br>'+
      '<input type="submit" value="Upload">'+
      '</form>'
    );
  } else if (req.url === '/upload') {
    var headers = {
      'x-amz-acl': 'public-read',
    };
    var form = new multiparty.Form();
    var batch = new Batch();
    batch.push(function(cb) {
      form.on('field', function(name, value) {
        if (name === 'path') {
          var destPath = value;
          if (destPath[0] !== '/') destPath = '/' + destPath;
          cb(null, destPath);
        }
      });
    });
    batch.push(function(cb) {
      form.on('part', function(part) {
        if (! part.filename) return;
        cb(null, part);
      });
    });
    batch.end(function(err, results) {
      if (err) throw err;
      form.removeListener('close', onEnd);
      var destPath = results[0]
        , part = results[1];

      var counter = new ByteCounter();
      part.pipe(counter); // need this until knox upgrades to streams2
      headers['Content-Length'] = part.byteCount;
      s3Client.putStream(part, destPath, headers, function(err, s3Response) {
        if (err) throw err;
        res.statusCode = s3Response.statusCode;
        s3Response.pipe(res);
        console.log("https://s3.amazonaws.com/" + process.env.S3_BUCKET + destPath);
      });
      part.on('end', function() {
        console.log("part end");
        console.log("size", counter.bytes);
      });
    });
    form.on('close', onEnd);
    form.parse(req);

  } else {
    res.writeHead(404, {'content-type': 'text/plain'});
    res.end('404');
  }

  function onEnd() {
    throw new Error("no uploaded file");
  }
});
server.listen(PORT, function() {
  console.info('listening on http://0.0.0.0:'+PORT+'/');
});

example taken from https://github.com/superjoe30/node-multiparty/blob/master/examples/s3.js

例子来自https://github.com/superjoe30/node-multiparty/blob/master/examples/s3.js

#3

the node/express code doesn't work with nodejs v0.4.7

node / express代码不适用于nodejs v0.4.7

here is the updated code for nodejs v0.4.7

这是nodejs v0.4.7的更新代码

app.post('/upload', function (req, res) {
  // connect-form additions
  req.form.complete(function (err, fields, files) {
    // here lies your uploaded file:
    var path = files['upload-file']['path'];
    // do knox stuff here
  });
});

#4

* update *

*更新*

as of mid 2009 amazon supports CORS and the upload via your node.js server isn't needed anymore. you can directly upload the file to S3.

截至2009年中期，亚马逊支持CORS，并且不再需要通过您的node.js服务器上传。您可以直接将文件上传到S3。

with the help of the "connect-form" module you could just upload the file to your server (through normal multipart FORM) and then handle the S3 stuff afterwards ...

在“连接表单”模块的帮助下，您可以将文件上传到服务器（通过普通的多部分FORM），然后处理S3内容...

<form action="/upload" method="POST" id="addContentForm" enctype="multipart/form-data">
  <p><label for="media">File<br/><input type="file" name="media" /></label></p>
  <p><button type="submit">upload</button></p>
</form>

node/express code:

节点/快递代码：

app.post('/upload', function (req, res) {
  // connect-form additions
  req.form.complete(function (err, fields, files) {
    // here lies your uploaded file:
    var path = files['media']['path'];
    // do knox stuff here
  });
});

you have to add the following line to the app configuration:

您必须将以下行添加到应用程序配置：

app.configure(function(){
  // rest of the config stuff ...
  app.use(form({ keepExtensions: true }));
  // ...
});

#5

-1

The connect-stream-s3 library can upload all of your forms files to S3 as part of middleware so you don't have to do any logic yourself. It needs express.bodyParser() for it to work at the moment, but I'm working on a version that will stream files direct to Amazon S3 prior to being written to disk:

connect-stream-s3库可以将所有表单文件作为中间件的一部分上传到S3，因此您不必自己做任何逻辑。它需要express.bodyParser（）才能使它正常工作，但我正在开发一个版本，在将文件写入磁盘之前直接将文件流式传输到Amazon S3：

https://github.com/appsattic/connect-stream-s3
https://github.com/appsattic/connect-stream-s3

Please let me know how you get on. Hopefully it's a lot less hassle than doing it yourself once you're in your page handler. :)

请告诉我你是怎么过的。一旦你进入你的页面处理程序，希望它比你自己做的麻烦少得多。 :)

#6

-1

I made this to upload directly from the Jquery File Upload plugin to S3 with file being public - it should point you in the right direction.

我把它直接从Jquery文件上传插件上传到S3，文件是公共的 - 它应该指向正确的方向。

https://gist.github.com/3995819

#1