如何将大数据传递给Web worker

I am working on web workers and I am passing large amount of data to web worker, which takes a lot of time. I want to know the efficient way to send the data.

我正在研究Web工作者,我正在将大量数据传递给Web worker,这需要花费大量时间。我想知道发送数据的有效方法。

I have tried the following code:

我试过以下代码:

var worker = new Worker('js2.js');
worker.postMessage( buffer,[ buffer]);
worker.postMessage(obj,[obj.mat2]);
if (buffer.byteLength) {
  alert('Transferables are not supported in your browser!');
}

2 个解决方案

#1

UPDATE:

According to Mozilla the SharedArrayBuffer has been disabled in all major browsers, thus the option described in the following EDIT does no longer apply.

根据Mozilla的说法,SharedArrayBuffer已在所有主流浏览器中禁用,因此以下编辑中描述的选项不再适用。

Note that SharedArrayBuffer was disabled by default in all major browsers on 5 January, 2018 in response to Spectre.

请注意,默认情况下,在2018年1月5日,所有主流浏览器都禁用了SharedArrayBuffer以响应Spectre。

EDIT: There is now another option and it is sending a sharedArray buffer. This is part of ES2017 under shared memory and atomics and is now supported in FireFox 54 Nightly. If you want to read about it you can look here. I will probably write up something some time and add it to my answer. I will try and add to the performance benchmark as well.

编辑:现在有另一个选项,它发送一个sharedArray缓冲区。这是ES2017在共享内存和原子下的一部分,现在在FireFox 54 Nightly中得到支持。如果你想阅读它,你可以看看这里。我可能会写一些时间并将其添加到我的答案中。我也会尝试添加性能基准测试。

To answer the original question:

I am working on web workers and I am passing large amount of data to web worker, which takes a lot of time. I want to know the efficient way to send the data.

我正在研究Web工作者,我正在将大量数据传递给Web worker,这需要花费大量时间。我想知道发送数据的有效方法。

The alternative to @MichaelDibbets answer, his sends a copy of the object to the webworker, is using a transferrable object which is zero-copy.

作为@MichaelDibbets的替代方案,他将对象的副本发送给webworker,使用的是一个零拷贝的可转移对象。

It shows that you were intending to make your data transferrable, but I'm guessing it didn't work out. So I will explain what it means for some data to be transferrable for you and future readers.

它表明你打算让你的数据可以转移,但我猜它没有成功。因此,我将解释一些数据对于您和未来的读者可以转移的意义。

Transferring objects "by reference" (although that isn't the perfect term for it as explained in the next quote) doesn't just work on any JavaScript Object. It has to be a transferrable data-type.

通过引用“转移”对象(虽然这不是它的完美术语,如下一篇引言中所述)不仅适用于任何JavaScript对象。它必须是可传输的数据类型。

[With Web Workers] Most browsers implement the structured cloning algorithm, which allows you to pass more complex types in/out of Workers such as File, Blob, ArrayBuffer, and JSON objects. However, when passing these types of data using postMessage(), a copy is still made. Therefore, if you're passing a large 50MB file (for example), there's a noticeable overhead in getting that file between the worker and the main thread.

[使用Web Workers]大多数浏览器都实现了结构化克隆算法,该算法允许您将更复杂的类型传入/传出Worker,例如File,Blob,ArrayBuffer和JSON对象。但是,使用postMessage()传递这些类型的数据时,仍会生成副本。因此,如果你传递一个大的50MB文件(例如),那么在工作者和主线程之间获取该文件会有明显的开销。

Structured cloning is great, but a copy can take hundreds of milliseconds. To combat the perf hit, you can use Transferable Objects.

结构化克隆很棒,但副本可能需要数百毫秒。要对抗性能,您可以使用可转移对象。

With Transferable Objects, data is transferred from one context to another. It is zero-copy, which vastly improves the performance of sending data to a Worker. Think of it as pass-by-reference if you're from the C/C++ world. However, unlike pass-by-reference, the 'version' from the calling context is no longer available once transferred to the new context. For example, when transferring an ArrayBuffer from your main app to Worker, the original ArrayBuffer is cleared and no longer usable. Its contents are (quiet literally) transferred to the Worker context.

使用可转移对象,数据从一个上下文传输到另一个上下文。它是零拷贝,极大地提高了向Worker发送数据的性能。如果您来自C / C ++世界,请将其视为传递参考。但是,与传递引用不同,一旦传输到新上下文,调用上下文中的“版本”就不再可用。例如,将ArrayBuffer从主应用程序传输到Worker时,原始的ArrayBuffer将被清除,不再可用。它的内容(字面上很安静)转移到Worker上下文。

- Eric Biddleman Developer at Google, source: html5rocks

- Google的Eric Biddleman开发人员,来源:html5rocks

The only problem is there are only two things that are transferrable as of now. ArrayBuffer, and MessagePort. (Canvas Proxies are hopefully coming later). ArrayBuffers cannot be manipulated directly through their API and should be used to create a typed array object or a DataView to give a particular view into the buffer and be able to read and write to it.

唯一的问题是,目前只有两件事可以转让。 ArrayBuffer和MessagePort。 (Canvas Proxies希望稍后出现)。 ArrayBuffers不能通过其API直接操作,应该用于创建类型化数组对象或DataView,以便为缓冲区提供特定视图,并能够读取和写入它。

From the html5rocks link

来自html5rocks链接

To use transferrable objects, use a slightly different signature of postMessage():

要使用可传输对象,请使用稍微不同的postMessage()签名:

worker.postMessage(arrayBuffer, [arrayBuffer]);

window.postMessage(arrayBuffer, targetOrigin, [arrayBuffer]);

window.postMessage(arrayBuffer,targetOrigin,[arrayBuffer]);

The worker case, the first argument is the data and the second is the list of items that should be transferred. The first argument doesn't have to be an ArrayBuffer by the way. For example, it can be a JSON object:

工作者案例,第一个参数是数据,第二个参数是应该传输的项目列表。顺便说一句,第一个参数不一定是ArrayBuffer。例如,它可以是JSON对象:

worker.postMessage({data: int8View, moreData: anotherBuffer}, [int8View.buffer, anotherBuffer]);

worker.postMessage({data:int8View,moreData:anotherBuffer},[int8View.buffer,anotherBuffer]);

So according to that your

所以根据你的意思

var worker = new Worker('js2.js');
worker.postMessage(buffer, [ buffer]);
worker.postMessage(obj, [obj.mat2]);

should be performing at great speeds and should be being transferred zero-copy. The only problem would be if your buffer or obj.mat2 is not an ArrayBuffer or transferrable. You may be confusing ArrayBuffers with a view of a typed array instead of what you should be using its buffer.

应该以极快的速度运行,并且应该被转移为零拷贝。唯一的问题是如果你的缓冲区或obj.mat2不是ArrayBuffer或者是可传输的。您可能会将ArrayBuffers与类型化数组的视图混淆,而不是使用其缓冲区。

So if you have this ArrayBuffer and it's Int32 representation. (though the variable is titled view it is not a DataView, but DataView's do have a property buffer just as typed arrays do. Also at the time this was written the MDN use the name 'view' for the result of calling a typed arrays constructor so I assumed it was a good way to define it.)

所以如果你有这个ArrayBuffer并且它是Int32表示。 (虽然变量标题为view,但它不是DataView,但DataView确实有一个属性缓冲区,就像类型化数组一样。当编写MDN时,MDN使用名称'view'来调用类型化数组构造函数的结果所以我认为这是定义它的好方法。)

var buffer = new ArrayBuffer(90000000);
var view = new Int32Array(buffer);
for(var c=0;c<view.length;c++) {
    view[c]=42;
}

This is what you should not do (send the view)

这是你不应该做的(发送视图)

worker.postMessage(view);

This is what you should do (send the ArrayBuffer)

这是你应该做的(发送ArrayBuffer)

worker.postMessage(buffer, [buffer]);

These are the results after running this test on plnkr.

这些是在plnkr上运行此测试后的结果。

Average for sending views is 144.12690000608563
Average for sending ArrayBuffers is 0.3522000042721629

EDIT: As stated by @Bergi in the comments you don't need the buffer variable at all if you have the view, because you can just send view.buffer like so

编辑:正如@Bergi在评论中所述,如果您有视图,则根本不需要缓冲变量,因为您可以像这样发送view.buffer

worker.postMessage(view.buffer, [view.buffer]);

Just as a side note to future readers just sending an ArrayBuffer without the last argument specifying what the ArrayBuffers are you will not send the ArrayBuffer transferrably

正如对未来读者的旁注,只是发送一个ArrayBuffer而没有指定ArrayBuffers的最后一个参数,你将不会发送ArrayBuffer

In other words when sending transferrables you want this:

换句话说,当您发送可转让物品时,您需要这样:

worker.postMessage(buffer, [buffer]);

Not this:

worker.postMessage(buffer);

EDIT: And one last note since you are sending a buffer don't forget to turn your buffer back into a view once it's received by the webworker. Once it's a view you can manipulate it (read and write from it) again.

编辑:最后一个注意事项,因为你发送一个缓冲区,不要忘记一旦webworker收到你的缓冲区就把它变回一个视图。一旦它成为一个视图,你就可以再次操作它(从中读取和写入)。

And for the bounty:

I am also interested in official size limits for firefox/chrome (not only time limit). However answer the original question qualifies for the bounty (;

我也对firefox / chrome的官方大小限制感兴趣(不仅限时)。然而,回答原始问题有资格获得赏金(;

As to a webbrowsers limit to send something of a certain size I am not completeley sure, but from that quote that entry on html5rocks by Eric Bidelman when talking about workers he did bring up a 50 mb file being transferred without using a transferrable data-type in hundreds of milliseconds and as shown through my test in a only around a millisecond using a transferrable data-type. Which 50 mb is honestly pretty large.

至于webbrowsers限制发送一定大小的东西,我并不完全确定,但是从引用html5rocks上的条目来看,Eric Bidelman在谈论工作时他确实调出了一个50 mb的文件而没有使用可传输的数据类型在数百毫秒内,通过我的测试显示,使用可传输数据类型仅在大约一毫秒内。其中50 MB实际上相当大。

Purely my own opinion, but I don't believe there to be a limit on the size of the file you send on a transferrable or non-transferrable data-type other than the limits of the data type itself. Of course your biggest worry would probably be for the browser stopping long running scripts if it has to copy the whole thing and is not zero-copy and transferrable.

纯粹是我自己的看法,但我不认为您在可传输或不可传输的数据类型上发送的文件大小有限制,而不是数据类型本身的限制。当然,您最担心的可能是浏览器停止长时间运行的脚本,如果它必须复制整个脚本并且不是零拷贝和可转移的。

Hope this post helps. Honestly I knew nothing about transferrables before this, but it was fun figuring out them through some tests and through that blog post by Eric Biddleman.

希望这篇文章有所帮助老实说,在此之前,我对可转移物一无所知,但通过一些测试以及Eric Biddleman撰写的博客文章很有趣。

#2

I had issues with webworkers too, until I just passed a single argument to the webworker.

我也遇到过webworkers的问题,直到我向webworker传递了一个参数。

So instead of

而不是

worker.postMessage( buffer,[ buffer]);
worker.postMessage(obj,[obj.mat2]);

Try

var myobj = {buffer:buffer,obj:obj};
worker.postMessage(myobj);

This way I found it gets passed by reference and its insanely fast. I post back and forth over 20.000 dataelements in a single push per 5 seconds without me noticing the datatransfer. I've been exclusively working with chrome though, so I don't know how it'll hold up in other browsers.

通过这种方式,我发现它通过引用传递,并且它的速度非常快。我每隔5秒就在一次推送中来回发送超过20,000个数据元素,而我没有注意到数据传输。我一直专注于使用chrome,所以我不知道它会如何在其他浏览器中保留。

Update

I've done some testing for some stats.

我已经对一些统计数据进行了一些测试。

tmp = new ArrayBuffer(90000000);
test = new Int32Array(tmp);
for(c=0;c<test.length;c++) {
    test[c]=42;
}
for(c=0;c<4;c++) {
    window.setTimeout(function(){
        // Cloning the Array. "We" will have lost the array once its sent to the webworker. 
        // This is to make sure we dont have to repopulate it.
        testsend = new Int32Array(test);
        // marking time. sister mark is in webworker
        console.log("sending at at  "+window.performance.now());
        // post the clone to the thread.
        FieldValueCommunicator.worker.postMessage(testsend);
    },1000*c);
}

results of the tests. I don't know if this falls in your category of slow or not since you did not define "slow"

测试结果。我不知道这是否属于你的慢类别,因为你没有定义“慢”

• sending at at 28837.418999988586

发送电子邮件地址:28837.418999988586

• recieved at 28923.06199995801

收到28923.06199995801

• 86 ms

  ---

• sending at at 212387.9840001464

  发送电子邮件地址为212387.9840001464

• recieved at 212504.72499988973

收到212504.72499988973

• 117 ms

  ---

• sending at at 247635.6210000813

  发送电子邮件地址:247635.6210000813

• recieved at 247760.1259998046

收到247760.1259998046

• 125 ms

  ---

• sending at at 288194.15999995545

  发送电子邮件地址:288194.15999995545

• recieved at 288304.4079998508

收到288304.4079998508

• 110 ms