Python Popen：同时写入stdout和日志文件

I am using Popen to call a shell script that is continuously writing its stdout and stderr to a log file. Is there any way to simultaneously output the log file continuously (to the screen), or alternatively, make the shell script write to both the log file and stdout at the same time?

我正在使用Popen调用一个shell脚本,该脚本不断将其stdout和stderr写入日志文件。有没有办法连续同时输出日志文件(到屏幕),或者让shell脚本同时写入日志文件和标准输出?

I basically want to do something like this in Python:

我基本上想在Python中做这样的事情:

cat file 2>&1 | tee -a logfile #"cat file" will be replaced with some script

Again, this pipes stderr/stdout together to tee, which writes it both to stdout and my logfile.

再次,这将stderr / stdout连接到tee,将它写入stdout和我的日志文件。

I know how to write stdout and stderr to a logfile in Python. Where I'm stuck is how to duplicate these back to the screen:

我知道如何在Python中将stdout和stderr写入日志文件。我被困在哪里是如何将这些复制回屏幕:

subprocess.Popen("cat file", shell=True, stdout=logfile, stderr=logfile)

Of course I could just do something like this, but is there any way to do this without tee and shell file descriptor redirection?:

当然我可以做这样的事情,但有没有办法在没有tee和shell文件描述符重定向的情况下做到这一点?:

subprocess.Popen("cat file 2>&1 | tee -a logfile", shell=True)

3 个解决方案

#1

You can use a pipe to read the data from the program's stdout and write it to all the places you want:

您可以使用管道从程序的标准输出读取数据并将其写入您想要的所有位置:

import sys
import subprocess

logfile = open('logfile', 'w')
proc=subprocess.Popen(['cat', 'file'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout:
    sys.stdout.write(line)
    logfile.write(line)
proc.wait()

UPDATE

In python 3, the universal_newlines parameter controls how pipes are used. If False, pipe reads return bytes objects and may need to be decoded (e.g., line.decode('utf-8')) to get a string. If True, python does the decode for you

在python 3中,universal_newlines参数控制管道的使用方式。如果为False,管道读取返回字节对象并且可能需要被解码(例如,line.decode('utf-8'))以获得字符串。如果为True,python会为您解码

Changed in version 3.3: When universal_newlines is True, the class uses the encoding locale.getpreferredencoding(False) instead of locale.getpreferredencoding(). See the io.TextIOWrapper class for more information on this change.

版本3.3中更改:当universal_newlines为True时,类使用编码locale.getpreferredencoding(False)而不是locale.getpreferredencoding()。有关此更改的更多信息,请参阅io.TextIOWrapper类。

#2

To emulate: subprocess.call("command 2>&1 | tee -a logfile", shell=True) without invoking the tee command:

要模拟:subprocess.call(“command 2>&1 | tee -a logfile”,shell = True)而不调用tee命令:

#!/usr/bin/env python2
from subprocess import Popen, PIPE, STDOUT

p = Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout, open('logfile', 'ab') as file:
    for line in iter(p.stdout.readline, b''):
        print line,  #NOTE: the comma prevents duplicate newlines (softspace hack)
        file.write(line)
p.wait()

To fix possible buffering issues (if the output is delayed), see links in Python: read streaming input from subprocess.communicate().

要解决可能的缓冲问题(如果输出延迟),请参阅Python中的链接:从subprocess.communicate()读取流输入。

Here's Python 3 version:

这是Python 3版本:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE, STDOUT

with Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1) as p, \
     open('logfile', 'ab') as file:
    for line in p.stdout: # b'\n'-separated lines
        sys.stdout.buffer.write(line) # pass bytes as is
        file.write(line)

#3

Write to terminal byte by byte for interactive applications

为交互式应用程序逐字节写入终端

I have found that this simulates the behavior of tee more closely for interactive applications.

我发现这可以更加密切地模拟tee对交互式应用程序的行为。

test.py:

#!/usr/bin/env python3
import os
import subprocess
import sys
with subprocess.Popen(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as proc, \
        open('logfile.txt', 'bw') as logfile:
    while True:
        byte = proc.stdout.read(1)
        if byte:
            sys.stdout.buffer.write(byte)
            sys.stdout.flush()
            logfile.write(byte)
        else:
            break
# In case you need it.
exit_status = proc.returncode

For example, if you run:

例如,如果您运行:

./test.py bash

then the characters you type appear immediately on the terminal as you type them, which is very important for interactive applications. This is what happens when you run:

然后,您键入的字符会在您键入时立即显示在终端上,这对于交互式应用程序非常重要。运行时会发生这种情况:

bash | tee logfile.txt

Also, if you want the output to show on the ouptut file immediately, then you can also add a:

此外,如果您希望输出立即显示在ouptut文件上,那么您还可以添加:

logfile.flush()

but tee does not do this, and I'm afraid it would kill performance. You can test this out easily with:

但是发球不会这样做,我担心它会扼杀表现。您可以使用以下方法轻松测试:

tail -f logfile.txt

#1