如何从分叉进程发送大量数据？

I have a ctypes wrapper for a library. Unfortunately, this library is not 100% reliable (occasional segfaults, etc.). Because of how it's used, I want the wrapper to be reasonably resilient to the library crashing.

我有一个库的ctypes包装器。不幸的是,这个库不是100%可靠(偶尔会出现段错误等)。由于它的使用方式,我希望包装器对库崩溃具有相当的弹性。

The best way to do this seems to be forking a process and sending the results back from the child. I'd like to do something along these lines:

这样做的最佳方法似乎是分配一个过程并从孩子那里发回结果。我想沿着这些方向做点什么:

r, w = os.pipe()
pid = os.fork()

if pid == 0:
    # child
    result = ctypes_fn()
    os.write(w, pickle.dumps(result))
    os.close(w)
else:
    # parent
    os.waitpid(pid, 0)
    result = os.read(r, 524288) # can be this big
    os.close(r)

    return pickle.loads(result)

This doesn't quite work, though. The forked process hangs on the write. Am I trying to send too much at once? Is there a simpler solution to this problem?

但这并不是很有效。分叉的进程在写入时挂起。我想一次发送太多吗?这个问题有更简单的解决方案吗?

3 个解决方案

#1

Probably you are trying to write more data than can fit into the pipe, so it is blocking until someone comes along and reads some of that info out of there. That will never happen, because the only reader is the parent process, which you appear to have written to wait until the child terminates before it reads anything. This is what we call a deadlock.

可能你正在尝试编写的数据超出了管道的数量,因此它会阻塞,直到有人出现并从那里读取一些信息。这种情况永远不会发生,因为唯一的读者是父进程,您似乎已经编写该进程以等待子进程在读取任何内容之前终止。这就是我们所说的僵局。

You might consider taking out that os.waitpid call and see what happens. Another option would be to see if os.pipe has any methods that give it a bigger buffer (I don't know your environment enough to say).

您可以考虑取出os.waitpid调用,看看会发生什么。另一个选择是看看os.pipe是否有任何方法可以给它一个更大的缓冲区(我不知道你的环境足以说明)。

#2

The basic problem is that there's a 64kB limit on the pipe. A few possible solutions, from the simple to the complex:

基本问题是管道有64kB的限制。一些可能的解决方案,从简单到复杂:

Send less data. zlib.compress could help in getting under the limit.

发送较少的数据。 zlib.compress可以帮助你达到极限。

Store the actual data somewhere else (file, mmap, memcache), only using the pipe to send control information.

将实际数据存储在其他位置(文件,mmap,memcache),仅使用管道发送控制信息。

Continue using the pipe, but chunk the output. Use two sets of pipes so the processes can talk to each other and synchronize their communication. The code is more complex, but is otherwise very effective.

继续使用管道,但输出块。使用两组管道,以便进程可以相互通信并同步它们的通信。代码更复杂,但在其他方面非常有效。

#3

One solution to the deadlock that ted.dennison mentioned is the following pseudocode:

ted.dennison提到的死锁的一个解决方案是以下伪代码:

#parent
while waitpid(pid, WNOHANG) == (0, 0):
    result = os.read(r, 1024)
    #sleep for a short time
#at this point the child process has ended 
#and you need the last bit of data from the pipe
result = os.read(r, 1024)
os.close(r)

Waitpid with the WNOHANG option causes waitpid to return immediately when the child process hasn't exited yet. In this case it returns (0,0). You'll need to make sure not to overwrite the result variable each time through the loop like the above code does.

具有WNOHANG选项的Waitpid会导致waitpid在子进程尚未退出时立即返回。在这种情况下,它返回(0,0)。您需要确保不像上面的代码那样每次都通过循环覆盖结果变量。

#1