Node.js的stdout child_process exec被缩短了

In Node.js I'm using the exec command of the child_process module to call an algorithm in Java that returns a large amount of text to standard out which I then parse and use. I'm able to capture it mostly, but when it exceeds a certain number of lines, the content is cutoff.

在Node.js中，我使用了child_process模块的exec命令来调用Java中的算法，该算法将大量文本返回到标准输出，然后我将其解析并使用。我能够捕获它，但是当它超过一定数量的行时，内容就会被截止。

exec("sh target/bin/solver "+fields.dimx+" "+fields.dimy, function(error, stdout, stderr){
    //do stuff with stdout
}

I've tried using setTimeouts and callbacks but haven't succeeded but I do feel this is occurring because I'm referencing stdout in my code before it can be retrieved completely. I have tested that stdout is in-fact where the data loss first occurs. It's not an asynchronous issue further down the line. I've also tested this on my local machine and Heroku, and the exact same issue occurs, truncating at the exact same line number every time.

我已经尝试过使用setTimeouts和回调但没有成功，但我确实感觉到这种情况正在发生，因为我在我的代码中引用了stdout，然后才能完全检索它。我已经测试过stdout实际上是数据丢失首先发生的地方。这不是一个异步问题。我也在我的本地机器和Heroku上测试了这个，并且出现了完全相同的问题，每次都截断完全相同的行号。

Any ideas or suggestions as to what might help with this?

关于什么可能对此有所帮助的任何想法或建议？

3 个解决方案

#1

Edited: I have tried with dir /s on my computer (windows) and got the same problem( it look like a bug), this code solve that problem for me:

编辑：我在我的计算机（Windows）上尝试使用dir / s并遇到同样的问题（看起来像一个bug），这段代码为我解决了这个问题：

var exec = require('child_process').exec;

function my_exec(command, callback) {
    var proc = exec(command);

    var list = [];
    proc.stdout.setEncoding('utf8');

    proc.stdout.on('data', function (chunk) {
        list.push(chunk);
    });

    proc.stdout.on('end', function () {
        callback(list.join());
    });
}

my_exec('dir /s', function (stdout) {
    console.log(stdout);
})

#2

I had exec.stdout.on('end') callbacks hung forever with @damphat solution.

我用@damphat解决方案永远挂了exec.stdout.on（'end'）回调。

Another solution is to increase the buffer size in the options of exec: see the documentation here

另一种解决方案是在exec的选项中增加缓冲区大小：请参阅此处的文档

{ encoding: 'utf8',
  timeout: 0,
  maxBuffer: 200*1024, //increase here
  killSignal: 'SIGTERM',
  cwd: null,
  env: null }

To quote: maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed. I now use the following: this does not require handling the separated parts of the chunks separated by commas in stdout, as opposed to the accepted solution.

引用：maxBuffer指定stdout或stderr上允许的最大数据量 - 如果超过此值，则子进程将被终止。我现在使用以下内容：这不需要处理stdout中用逗号分隔的块的分隔部分，而不是接受的解决方案。

exec('dir /b /O-D ^2014*', {
    maxBuffer: 2000 * 1024 //quick fix
    }, function(error, stdout, stderr) {
        list_of_filenames = stdout.split('\r\n'); //adapt to your line ending char
        console.log("Found %s files in the replay folder", list_of_filenames.length)
    }
);

#3

The real (and best) solution to this problem is to use spawn instead of exec. As stated in this article, spawn is more suited for handling large volumes of data :

这个问题的真正（和最好）解决方案是使用spawn而不是exec。如本文所述，spawn更适合处理大量数据：

child_process.exec returns the whole buffer output from the child process. By default the buffer size is set at 200k. If the child process returns anything more than that, you program will crash with the error message "Error: maxBuffer exceeded". You can fix that problem by setting a bigger buffer size in the exec options. But you should not do it because exec is not meant for processes that return HUGE buffers to Node. You should use spawn for that. So what do you use exec for? Use it to run programs that return result statuses, instead of data.

child_process.exec返回子进程的整个缓冲区输出。默认情况下，缓冲区大小设置为200k。如果子进程返回的内容不止于此，则程序将崩溃，并显示错误消息“Error：maxBuffer exceeded”。您可以通过在exec选项中设置更大的缓冲区大小来解决该问题。但是你不应该这样做，因为exec不适用于将HUGE缓冲区返回给Node的进程。你应该使用spawn。那么你用exec做什么？使用它来运行返回结果状态而不是数据的程序。

spawn requires a different syntax than exec :

spawn需要与exec不同的语法：

var proc = spawn('sh', ['target/bin/solver', 'fields.dimx', 'fields.dimy']);

proc.on("exit", function(exitCode) {
    console.log('process exited with code ' + exitCode);
});

proc.stdout.on("data", function(chunk) {
    console.log('received chunk ' + chunk);
});

proc.stdout.on("end", function() {
    console.log("finished collecting data chunks from stdout");
});

#1