在node.js中读取csv文件的内容

I am trying to implement a module in nodejs(just started working in nodejs) which has requirement below as

我正在尝试在nodejs中实现一个模块（刚刚开始在nodejs中工作），该模块的要求如下

Upload .csv file.
上传.csv文件。
Read content of the csv file.
阅读csv文件的内容。

Frameworks currently being used for restful api is "express": "~4.2.0" and multer for file upload.

当前用于restful api的框架是“express”：“~4.2.0”和文件上传的multer。

Now I have configured multer like below in my app.js

现在我在app.js中配置了如下所示的multer

app.use(multer({
  onFileUploadData : function(file, data){
    console.log('onFileUploadData Called with data - '+ data);
  }
}));

In my route file, I have a post endpoint like below

在我的路径文件中，我有一个如下所示的帖子端点

app.post('/sample.csv',lead.processCSV);

This route is being called from an ajax call below as

这条路线是从下面的ajax调用中调用的

$.ajax({
            xhrFields: {withCredentials: true},
            url: '/sample.csv',
            type: 'POST',
            success: function (data) {
                $scope.handleResponse(data);
            },
            error: function (error, xhr) {
                angular.element('#csvUploadBusyIcon').hide();
                alert('Oops! Upload failed');
            },
            data: formData,
            cache: false,
            contentType: false,
            processData: false
        });

Now I want to get the content of the csv file, i.e. when all the content has been loaded then I should handle my lead.processCSV method.

现在我想获取csv文件的内容，即当所有内容都已加载后，我应该处理我的lead.processCSV方法。

Also do I need any other module for csv files, or multer is sufficient in my case?

我还需要csv文件的任何其他模块，或者multer在我的情况下是否足够？

Any suggestion/guidance in right direction will be helpful. Thanks in Advance.

任何正确方向的建议/指导都会有所帮助。提前致谢。

1 个解决方案

#1

There is an awesome node project which helped me a lot. You should check it out What we're going to use is their csv-parse module. It is able to get a stream as input and read it line by line without blocking the event loop, so basically while you are processing a file your server won't be stuck and other requests can still be processed normally.

有一个很棒的节点项目给了我很多帮助。你应该检查一下我们将要使用的是他们的csv-parse模块。它能够将流作为输入并逐行读取而不会阻塞事件循环，因此基本上在处理文件时，您的服务器不会被卡住，其他请求仍然可以正常处理。

Since you said you are just starting with nodejs, you should make a quick search and understand how midlewares work in the request handling process. As a simplification for request handling,a middleware is a function(req, res, next). With req you get request data. With res you can send the response, and next you send your req and res objects to the next middleware. This way you can process a request in parts and the last middleware of the flow will send response to the client (res.send(200) for example)

既然你说你刚开始使用nodejs，你应该快速搜索并理解中间件在请求处理过程中的工作方式。作为请求处理的简化，中间件是一个函数（req，res，next）。使用req，您可以获得请求数据。使用res，您可以发送响应，然后将req和res对象发送到下一个中间件。这样您就可以部分处理请求，并且流的最后一个中间件将向客户端发送响应（例如res.send（200））

the Multer({...}) call returns a middleware function. When a request gets to this Middleware, multer will try to download any files the user send in the post request. When u say app.use(Multer({...})), you are asking multer to try and download files from ANY post requests that contains files. This is a security risk if not all your routes are expecting files to be uploaded.

Multer（{...}）调用返回一个中间件函数。当请求到达此中间件时，multer将尝试下载用户在发布请求中发送的任何文件。当你说app.use（Multer（{...}））时，你要求multer尝试从任何包含文件的帖子请求中下载文件。如果并非所有路由都希望上载文件，则存在安全风险。

Ok, that being said, here's a sample code I wrote to handle your use case:

好的，这就是说，这是我为处理您的用例而编写的示例代码：

//Important Security advice: 
//don't add multer as a middleware to all requests. 
//If you do this, people will be able to upload files
//in ALL YOUR 'post' handlers!!! 

var Multer = require('multer');
var Parse = require('csv-parse');
var fs = require('fs')

function parseCSVFile(sourceFilePath, columns, onNewRecord, handleError, done){
    var source = fs.createReadStream(sourceFilePath);

    var linesRead = 0;

    var parser = Parse({
        delimiter: ',', 
        columns:columns
    });

    parser.on("readable", function(){
        var record;
        while (record = parser.read()) {
            linesRead++;
            onNewRecord(record);
        }
    });

    parser.on("error", function(error){
        handleError(error)
    });

    parser.on("end", function(){
        done(linesRead);
    });

    source.pipe(parser);
}

//We will call this once Multer's middleware processed the request
//and stored file in req.files.fileFormFieldName

function parseFile(req, res, next){
    var filePath = req.files.file.path;
    console.log(filePath);
    function onNewRecord(record){
        console.log(record)
    }

    function onError(error){
        console.log(error)
    }

    function done(linesRead){
        res.send(200, linesRead)
    }

    var columns = true; 
    parseCSVFile(filePath, columns, onNewRecord, onError, done);

}

//this is the route handler with two middlewares. 
//First:  Multer middleware to download file. At some point,
//this middleware calls next() so process continues on to next middleware
//Second: use the file as you need

app.post('/upload', [Multer({dest:'./uploads'}), parseFile]);

I hope this helped. Make sure to understand how routes middlewares work in node: they are a key to good quality code.

我希望这有帮助。确保了解路由中间件如何在节点中工作：它们是高质量代码的关键。

Marcel

马塞尔

#1