捕获输出时，使用带有select和pty的子进程挂起

I'm trying to write a python program that is able to interact with other programs. That means sending stdin and receiving stdout data. I cannot use pexpect (although it definitely inspired some of the design). The process I'm using right now is this:

我正在尝试编写一个能够与其他程序交互的python程序。这意味着发送stdin并接收stdout数据。我不能使用pexpect(尽管它确实激发了一些设计)。我现在正在使用的过程是这样的:

Attach a pty to the subprocess's stdout

将pty附加到子进程的stdout

Loop until the subprocess exits by checking subprocess.poll
- When there is data available in the stdout write that data immediately to the current stdout.

通过检查subprocess.poll循环直到子进程退出当stdout中有数据可用时立即将数据写入当前stdout。

Finish!

I've been prototyping some code (below) which works but seems to have one flaw that is bugging me. After the child process has completed, the parent process hangs if I do not specify a timeout when using select.select. I would really prefer not to set a timeout. It just seems a bit dirty. However, all the other ways I've tried to get around the issue don't seem to work. Pexpect seems to get around it by using os.execv and pty.fork instead of subprocess.Popen and pty.openpty a solution I do not prefer. Am I doing something wrong with how I check for the life of the subprocess? Is my approach incorrect?

我一直在制作一些代码(下面)的原型,但它似乎有一个漏洞让我烦恼。子进程完成后,如果在使用select.select时未指定超时,则父进程将挂起。我真的不想设置超时。它看起来有点脏。但是,我尝试解决这个问题的所有其他方法似乎都不起作用。 Pexpect似乎通过使用os.execv和pty.fork而不是subprocess.Popen和pty.openpty来解决它,这是我不喜欢的解决方案。我是如何检查子流程的生命周期的?我的方法不正确吗?

The code I'm using is below. I'm using this on a Mac OS X 10.6.8, but I need it to work on Ubuntu 12.04 as well.

我正在使用的代码如下。我在Mac OS X 10.6.8上使用它,但我也需要它在Ubuntu 12.04上工作。

This is the subprocess runner runner.py:

这是subprocess runner runner.py:

import subprocess
import select
import pty
import os
import sys

def main():
    master, slave = pty.openpty()

    process = subprocess.Popen(['python', 'outputter.py'], 
            stdin=subprocess.PIPE, 
            stdout=slave, stderr=slave, close_fds=True)

    while process.poll() is None:
        # Just FYI timeout is the last argument to select.select
        rlist, wlist, xlist = select.select([master], [], [])
        for f in rlist:
            output = os.read(f, 1000) # This is used because it doesn't block
            sys.stdout.write(output)
            sys.stdout.flush()
    print "**ALL COMPLETED**"

if __name__ == '__main__':
    main()

This is the subprocess code outputter.py. The strange random parts are just to simulate a program outputting data at random intervals. You can remove it if you wish. It shouldn't matter:

这是子进程代码outputter.py。奇怪的随机部分只是模拟以随机间隔输出数据的程序。如果您愿意,可以将其删除。没关系:

import time
import sys
import random

def main():
    lines = ['hello', 'there', 'what', 'are', 'you', 'doing']
    for line in lines:
        sys.stdout.write(line + random.choice(['', '\n']))
        sys.stdout.flush()
        time.sleep(random.choice([1,2,3,4,5])/20.0)
    sys.stdout.write("\ndone\n")
    sys.stdout.flush()

if __name__ == '__main__':
    main()

Thanks for any help you all can provide!

感谢您提供的任何帮助!

Extra note

pty is used because I want to ensure that stdout isn't buffered.

使用pty是因为我想确保不缓冲stdout。

4 个解决方案

#1

First of all, os.read does block, contrary to what you state. However, it does not block after select. Also os.read on a closed file descriptor always returns an empty string, that you might want to check for.

首先,os.read会阻塞,与你声明的相反。但是,选择后它不会阻止。关闭文件描述符上的os.read也总是返回一个空字符串,您可能需要检查它。

The real problem however is that the master device descriptor is never closed, thus the final select is the one that will block. In a rare race condition, the child process has exited between select and process.poll() and your program exits nicely. Most of the time however the select blocks forever.

然而,真正的问题是主设备描述符永远不会关闭,因此最终选择是将阻止的。在罕见的竞争条件下,子进程已退出select和process.poll()之间,并且您的程序退出很好。大多数情况下,选择块永远存在。

If you install the signal handler as proposed by izhak all hell breaks loose; whenever a child process is terminated, the signal handler is run. After the signal handler is run, the original system call in that thread cannot be continued, so that syscall invocation returns nonzero errno, which often results in some random exception being thrown in python. Now, if elsewhere in your program you use some library with any blocking system calls that do not know how to handle such exceptions, you are in a big trouble (any os.read for example anywhere can now throw an exception, even after a successful select).

如果按照izhak的建议安装信号处理程序,那么一切都会破裂;每当子进程终止时,都会运行信号处理程序。运行信号处理程序后,该线程中的原始系统调用无法继续,因此syscall调用返回非零errno,这通常会导致在python中抛出一些随机异常。现在,如果您的程序中的其他位置使用某个库以及任何不知道如何处理此类异常的阻塞系统调用,那么您就会遇到大麻烦(例如任何os.read都可以在任何地方抛出异常,即使在成功之后也是如此选择)。

Weighing having random exceptions thrown anywhere against polling a bit, I don't think the timeout on select does not sound that bad idea. Your process would still hardly be the only (slow) polling process on the system anyway.

称重在任何地方抛出随机异常以防止轮询,我不认为select上的超时听起来不是坏主意。无论如何,您的过程仍然不是系统上唯一(慢)的轮询过程。

#2

There are a number of things you can change to make your code correct. The simplest thing I can think of is just to close your parent process's copy of the slave fd after forking, so that when the child exits and closes its own slave fd, the parent's select.select() will mark the master as available for read, and the subsequent os.read() will give an empty result and your program will complete. (The pty master won't see the slave end as being closed until both copies of the slave fd are closed.)

您可以更改许多内容以使代码正确无误。我能想到的最简单的事情就是在分叉后关闭父进程的slave fd副本,这样当子进出并关闭自己的slave fd时,父进程select.select()会将master标记为可读,后续的os.read()将给出一个空结果,你的程序将完成。 (在从属fd的两个副本都关闭之前,pty主站不会看到从属端关闭。)

So, just one line:

所以,只需一行:

os.close(slave)

..placed immediately after the subprocess.Popen call, ought to fix your problem.

..在subprocess.Popen调用之后立即放置,应该解决你的问题。

However, there are possibly better answers, depending on exactly what your requirements are. As someone else noted, you don't need a pty just to avoid buffering. You could use a bare os.pipe() in place of pty.openpty() (and treat the return value exactly the same). A bare OS pipe will never buffer; if the child process isn't buffering its output, then your select() and os.read() calls won't see buffering either. You would still need the os.close(slave) line, though.

但是,根据您的要求,可能会有更好的答案。正如其他人所说,你不需要pty只是为了避免缓冲。您可以使用裸os.pipe()代替pty.openpty()(并将返回值完全相同)。裸露的OS管道永远不会缓冲;如果子进程没有缓冲其输出,那么select()和os.read()调用也不会看到缓冲。但是你仍然需要os.close(slave)行。

But it's possible that you do need a pty for different reasons. If some of your child programs expect to be run interactively much of the time, then they might be checking to see if their stdin is a pty and behaving differently depending on the answer (lots of common utilities do this). If you really do want the child to think it has a terminal allocated for it, then the pty module is the way to go. Depending on how you'll run runner.py, you may need to switch from using subprocess to pty.fork(), so that the child has its session ID set and the pty pre-opened (or see the source for pty.py to see what it does and duplicate the appropriate parts in your subprocess object's preexec_fn callable).

但是你可能因为不同的原因需要一个pty。如果你的一些子程序希望在大多数情况下以交互方式运行,那么他们可能会检查他们的stdin是否是一个pty并且根据答案行为不同(很多常见的实用程序都这样做)。如果你真的希望孩子认为它有一个分配给它的终端,那么pty模块就是你要走的路。根据您将如何运行runner.py,您可能需要从使用子进程切换到pty.fork(),以便子进程设置其会话ID并预先打开pty(或查看pty.py的源代码)查看它的作用并复制子进程对象的preexec_fn可调用的相应部分。

#3

From what I understand, you do not need to use pty. runner.py can be modified as

据我所知,你不需要使用pty。 runner.py可以修改为

import subprocess
import sys

def main():
        process = subprocess.Popen(['python', 'outputter.py'],
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        while process.poll() is None:
                output = process.stdout.readline()
                sys.stdout.write(output)
                sys.stdout.flush()
        print "**ALL COMPLETED**"

if __name__ == '__main__':
        main()

process.stdout.read(1) can be used instead of process.stdout.readline() for real-time output per character from the subprocess.

process.stdout.read(1)可以代替process.stdout.readline()用于从子进程中每个字符的实时输出。

Note: If you do not require real-time output from the subprocess, use Popen.communicate to avoid the polling loop.

注意:如果您不需要子进程的实时输出,请使用Popen.communicate来避免轮询循环。

#4

When your child process exits - your parent process gets SIGCHLD signal. By default this signal is ignored but you can intercept it:

当您的子进程退出时 - 您的父进程会收到SIGCHLD信号。默认情况下,此信号被忽略但您可以拦截它:

import sys
import signal

def handler(signum, frame):
    print 'Child has exited!'
    sys.exit(0)

signal.signal(signal.SIGCHLD, handler)

The signal should also break the blocking syscall to 'select' or 'read' (or whatever you are in) and let you do whatever you have to (cleanup, exit, etc.) in handler function.

该信号还应该将阻塞系统调用分解为“选择”或“读取”(或者您所处的任何内容),并让您在处理函数中执行任何操作(清理,退出等)。

#1