我可以提高效率吗？使用NodeJS，Socket.io在Javascript中创建缓冲区

I'm getting a lot of data on my socket.io client side that may or may not be complete data and its also potentially BIG data, so I need to make it efficient. So I have tried my hand at creating a buffer and parsing that into my app. I know there are some streaming/buffering modules available, and I might consider using those if they achieve the goal of being more efficient. Looking forward to seeing your answers and possible arguments on how to best do this.

我在socket.io客户端获得了大量数据,可能是也可能不是完整的数据,也可能是BIG数据,所以我需要提高效率。所以我尝试创建缓冲区并将其解析到我的应用程序中。我知道有一些流媒体/缓冲模块可用,如果他们达到提高效率的目标,我可能会考虑使用它们。期待看到您的答案和可能的争论如何最好地做到这一点。

Note, bandwidth is not my concern as much as how quickly the client side Javascript can render the data into a browser friendly format.

请注意,带宽并不是我所关心的,因为客户端Javascript可以多快地将数据呈现为浏览器友好格式。

Here is what I've got so far.

这是我到目前为止所得到的。

function extended_split(str, separator, max) {
        var out = [],
                index = 0,
                next;

        while (!max || out.length < max - 1 ) {
                next = str.indexOf(separator, index);
                if (next === -1) {
                        break;
                }
                out.push(str.substring(index, next));
                index = next + separator.length;
        }
        out.push(str.substring(index));
        return out;
};  

var buffer = ''; 

// data format "\nOP:ARGS:DATA(could be base64 or 'other' depending on OP)\0";
socket.on('ioSend', function(data) {
    data = String.fromCharCode.apply(null, new Uint16Array(data));
    buffer = buffer + data; 
    while(buffer.indexOf('\n') != -1 && extended_split(buffer, '\n', 2)[1].indexOf('\0') != -1)
    {
        splitted = extended_split(extended_split(buffer, '\n', 2)[1], '\0', 2);  
        parse = splitted[0];
        buffer = splitted[1];  
        parse = parse.split(':');
        // Do stuff with parse here                     
    } 
});

2 个解决方案

#1

Rolling your own buffer builder/parser its ok, but you can spent double the time working and maintaining it, from just getting a production ready script.

滚动你自己的缓冲区构建器/解析器就可以了,但是你可以花费两倍的时间来处理和维护它,而不仅仅是获得一个生产就绪的脚本。

Now from my point of view, i would first drop socket.io for your case, since it just doesn't transmit binary as it should , there are other modules that transmit binary https://github.com/binaryjs/binaryjs which are better suited for binary transmissions over websocket protocol.

现在从我的角度来看,我会首先为你的情况删除socket.io,因为它只是不传输二进制文件,还有其他模块传输二进制https://github.com/binaryjs/binaryjs这是更适合于websocket协议的二进制传输。

i would also try http://bsonspec.org/ (check implementations for node modules), which encodes your json into binary, this way you could skip the whole problem with building and maintaining the buffer parser/builder.

我也会尝试http://bsonspec.org/(检查节点模块的实现),它将你的json编码为二进制,这样你就可以跳过构建和维护缓冲区解析器/构建器的整个问题。

#2

I took another crack at this and dropped the extended split idea and came up with this.

我对此采取了另一个裂缝并放弃了扩展分裂的想法,并想出了这个。

socket.on('ioSend', function(data) {                                    // receive command from socket.io
    if (safeBrowser) {                                                  // IE < 10 doesn't support Uint16Array
        var xdata = new Uint16Array(data);
        data = String.fromCharCode.apply(null, xdata);
        buffer = buffer + data;                                         // Update the buffer with most recent ioSend data  
    }
    else {                                                              // So we have to kludge this in for IE < 10 
        var xdata = '';
        for (var i = 0; i < data.length; i++) {
            xdata += String.fromCharCode(data[i]);
        }
        buffer = buffer + xdata;                                         // Update the buffer with most recent ioSend data   
    }
    var splitA = [];
    var splitB = [];
    while(buffer.indexOf(d1) != -1 && buffer.indexOf(d2) != -1)         // While loop reads buffer until there are no commands left to issue
    {
        splitA = buffer.split(d2);                  // Array with rear delimiter
        splitB = splitA[0].split(d1);           
        doParse.call(null, splitB[1]);              // This should be an @command
        splitB = null;
        splitA.shift();                             // Shift array
        buffer = splitA.join(d2);                   // Update buffer from shifted array with rear delimiter
    }     
});

It's really fast in all my unit tests and does the job really well. I'm working on an implementation that doesn't use socket.io as @GeoPhoenix suggested but until then this works good.

在我的所有单元测试中它都非常快,并且工作得非常好。我正在开发一个不使用socket.io的实现,因为@GeoPhoenix建议但在此之前这很好用。

#1