太多的打开文件错误与Popen的子进程。

I'm using Python's subprocess module to call a command to write values from a file to memory. It looks like:

我正在使用Python的子进程模块调用一个命令来从一个文件写到内存。它看起来像:

import subprocess

f = open('memdump', 'r')
content = [line.split()[1] for line in f]
f.close()
tbl_pt0 = 0x4400
tbl_pt1 = 0x4800
process = 0
i = 0
for value in content:
    p1 = subprocess.Popen("echo \"jaguar instance=0; jaguar wr offset=0x%x value=%s\" | pdt" \
% (tbl_pt0, value), shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    p2 = subprocess.Popen("echo \"jaguar instance=0; jaguar wr offset=0x%x value=%s\" | pdt" \
% (tbl_pt1, value), shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    tbl_pt0 += 0x4
    tbl_pt1 += 0x4

Here the pdt is a program that writes the value. There're 256 values in the file, so the for loop runs 256 loops. But when I get to loop 253, I got too many open files error in subprocess.py. Any one has any idea? Thanks.

这里pdt是一个写值的程序。文件中有256个值，因此for循环运行256个循环。但是当我进入第253循环时，在子过程中有太多的打开文件错误。有人知道吗?谢谢。

2 个解决方案

#1

The reason is that subprocess.Popen call is asynchronous in that it returns immediately without waiting for the spawned process to exit. That said, you are quickly creating 2x256 processes. Each of them has 2 pipes each taking up 1 file descriptor. A single process can have a limited amount of file descriptors opened at any one time, and you are reaching it because you don't wait for the processes and pipes to close. You can either wait for them to exit, e.g. p.communicate() where p is the return value of subprocess.Popen, or increase the maximum file descriptors opened at once:

原因是子过程。Popen调用是异步的，它可以立即返回，而无需等待生成的进程退出。也就是说，您正在快速创建2x256进程。每个都有2个管道，每个都占用1个文件描述符。单个进程可以在任何时候打开有限数量的文件描述符，而您正在到达它，因为您不等待进程和管道关闭。您可以等待它们退出，例如p通信()，其中p是子进程的返回值。Popen，或增加最大文件描述符同时打开:

• permamently - add fs.file-max = 100000 to /etc/sysctl.conf
• 后,添加fs。文件-max = 100000到/etc/sysctl.conf。
• temporarily (until reboot) - sysctl -w fs.file-max=100000
• 暂时(直到重新启动)- sysctl -w fs.文件-max=100000。

#2

With every Popen you create 6 new filehandles. Perhaps you can run the processes one by one, and use communicate instread of echo:

使用每个Popen创建6个新的文件句柄。也许你可以一个一个地运行这些过程，并且使用echo的通信方式:

import subprocess
from itertools import count

JAGUAR = "jaguar instance=0; jaguar wr offset=0x%x value=%s"

with open('memdump', 'r') as f:
    content = [line.split()[1] for line in f]

for tbl_pt0, tpl_pt1, value in zip(count(0x4400,4), count(0x4800,4), content):
    subprocess.Popen(["pdt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        close_fds=True).communicate(JAGUAR % (tbl_pt0, value))
    subprocess.Popen(["pdt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        close_fds=True).communicate(JAGUAR % (tbl_pt1, value))