强制线程阻止所有其他线程执行

UPDATE:

更新:

This answer states that what I'm trying to do is impossible as of April 2013. This, however, seems to contradict what Alex Martelli says in Python Cookbook (p. 624, 3rd ed.):

这个答案表明，从2013年4月开始，我所做的一切都是不可能的。然而，这似乎与Alex Martelli在Python Cookbook(第624页，第三版)中所说的相反:

Upon return, PyGILState_Ensure() always guarantees that the calling thread has exclusive access to the Python interpreter. This is true even if the calling C code is running a different thread that is unknown to the interpreter.

返回时，PyGILState_Ensure()始终确保调用线程具有对Python解释器的独占访问。即使调用C代码运行的是解释器未知的另一个线程，这也是正确的。

The docs also seem to suggest GIL can be acquired, which would give me hope (except I don't think I can call PyGILState_Ensure() from pure python code, and if I create a C extension to call it, I'm not sure how to embed my memory_daemon() in that).

文档似乎还建议可以获得GIL，这给了我希望(除了我不认为可以从纯python代码调用PyGILState_Ensure()，如果我创建一个C扩展来调用它，我不确定如何将memory_daemon()嵌入其中)。

Perhaps I'm misreading either the answer or Python Cookbook and the docs.

也许我误读了答案或Python食谱和文档。

ORIGINAL QUESTION:

最初的问题:

I want a given thread (from threading module) to prevent any other thread from running while a certain segment of its code is executing. What's the easiest way to achieve it?

我想要一个给定的线程(来自线程模块)来阻止任何其他线程在其代码的某个部分执行时运行。最简单的方法是什么?

Obviously, it would be great to minimize code changes in the other threads, to avoid using C and direct OS calls, and to make it cross-platform for windows and linux. But realistically, I'll be happy to just have any solution whatsoever for my actual environment (see below).

显然，最好将其他线程中的代码更改最小化，避免使用C和直接操作系统调用，并使其成为windows和linux的跨平台。但实际上，我很乐意为我的实际环境提供任何解决方案(参见下面)。

Environment:

环境:

• CPython
• CPython的
• python 3.4 (but can upgrade to 3.5 if it helps)
• python 3.4(如果有帮助，可以升级到3.5)
• Ubuntu 14.04
• Ubuntu 14.04

Use case:

用例:

For debugging purposes, I calculate memory used by all the objects (as reported by gc.get_objects()), and print some summary report to sys.stderr. I do this in a separate thread, because I want this summary delivered asynchronously from other threads; I put time.sleep(10) at the end of the while True loop that does the actual memory usage calculation. However, the memory reporting thread takes a while to complete each report, and I don't want all the other threads to move ahead before the memory calculation is finished (otherwise, the memory snapshot will be really hard to interpret).

出于调试目的，我计算了所有对象使用的内存(如gc.get_objects()所报告的)，并将一些摘要报告打印到sys.stderr。我在一个单独的线程中执行此操作，因为我希望这个摘要从其他线程异步传递;我将time.sleep(10)放在执行实际内存使用计算的while True循环的末尾。但是，内存报告线程需要一段时间才能完成每个报告，而且我不希望所有其他线程在内存计算完成之前就开始运行(否则，内存快照将非常难以解释)。

Example (to clarify the question):

例子(澄清问题):

import threading as th
import time

def report_memory_consumption():
  # go through `gc.get_objects()`, check their size and print a summary
  # takes ~5 min to run

def memory_daemon():
  while True:
    # all other threads should not do anything until this call is complete
    report_memory_consumption()
    # sleep for 10 sec, then update memory summary
    # this sleep is the only time when other threads should be executed
    time.sleep(10)


def f1():
  # do something, including calling many other functions
  # takes ~3 min to run

def f2():
  # do something, including calling many other functions
  # takes ~3 min to run


def main():
  t_mem = th.Thread(target = memory_daemon)
  t1 = th.Thread(target = f1)
  t2 = th.Thread(target = f2)
  t_mem.start()
  t1.start()
  t2.start()

# requirement: no other thread is running while t_mem is not sleeping

4 个解决方案

#1

The Python Cookbook is correct. You have exclusive access to the Python interpreter at the point when PyGILState_Ensure() returns. Exclusive access means that you can safely call all CPython functions. And it means the current C thread is also the current active Python thread. If the current C thread did not have a corresponding Python thread before, PyGILState_Ensure() will have created one for you automatically.

Python食谱是正确的。当PyGILState_Ensure()返回时，您可以对Python解释器进行独占访问。独占访问意味着您可以安全地调用所有CPython函数。这意味着当前C线程也是当前活动的Python线程。如果当前的C线程以前没有相应的Python线程，那么PyGILState_Ensure()将自动为您创建一个。

That is the state right after PyGILState_Ensure(). And you also have the GIL acquired at that point.

这就是PyGILState_Ensure()之后的状态。你也得到了GIL在那个点上得到的。

However, when you call other CPython functions now, such as PyEval_EvalCode() or any other, they can implicitly make that the GIL gets released meanwhile. For example, that is the case if implicitly the Python statement time.sleep(0.1) gets called somewhere as a result. And while the GIL is released from this thread, other Python threads can run.

但是，当您现在调用其他CPython函数(如PyEval_EvalCode()或其他函数)时，它们可以隐式地使GIL同时被释放。例如，如果隐式地调用Python语句time.sleep(0.1)，结果就会被调用。当GIL从这个线程释放时，其他Python线程可以运行。

You only have the guarantee that when PyEval_EvalCode() (or whatever other CPython function you called) returns, you will again have the same state as before - i.e. you are on the same active Python thread and you again have the GIL.

您只能保证，当PyEval_EvalCode()(或您调用的任何其他CPython函数)返回时，您将再次拥有与以前相同的状态——也就是说，您在相同的活动Python线程上，并且再次拥有GIL。

---

About your original question: There currently is no way to achieve this, i.e. to call Python code and avoid that the GIL gets released as a result somewhere meanwhile. And this is a good thing, otherwise you could easily be end up in deadlocks, e.g. if you don't allow some other thread to release some lock which it currently holds.

关于您最初的问题:目前还没有实现这个目标的方法，即调用Python代码，同时避免在某个地方发布GIL。这是一件好事，否则很容易导致死锁，例如，如果不允许其他线程释放当前持有的锁。

About how to implement your use case: The only real way to do that is in C. You would call PyGILState_Ensure() to get the GIL. And at that point, you must only call those CPython functions which cannot have the side effect of calling other Python code. Be very careful. Even PyObj_DecRef() could call __del__. The best thing would be to avoid calling any CPython functions and manually traversing the CPython objects. Note that you probably don't have to do it as complicated as you outlined it: There is the underlying CPython memory allocator and I think you can just get the information from there.

关于如何实现您的用例:惟一真正的实现方法是在c中，您可以调用PyGILState_Ensure()来获取GIL。此时，您必须只调用那些不能调用其他Python代码的CPython函数。很小心。甚至pyobj_orderf()也可以调用__del__。最好的方法是避免调用任何CPython函数并手动遍历CPython对象。注意，您可能不需要像您所描述的那样复杂:有底层的CPython内存分配器，我认为您可以从那里获得信息。

Read here about the memory management in CPython.

请阅读本文，了解CPython中的内存管理。

Related code is in pymem.h, obmalloc.c and pyarena.c. See the function _PyObject_DebugMallocStats(), although that might not be compiled into your CPython.

相关代码在pymem中。h,obmalloc。c和pyarena.c。请参见函数_PyObject_DebugMallocStats()，尽管这可能不会编译到您的CPython中。

There is also the tracemalloc module which however will add some overhead. Maybe its underlying C code (file _tracemalloc.c) is helpful however to understand the internals a bit better.

不过，tracemalloc模块也会增加一些开销。也许它的底层C代码(file _tracemallocation . C)对更好地理解内部内容有帮助。

---

About sys.setswitchinterval(1000): That is related only for going through the Python bytecode and handling it. That is basically the main loop of CPython in PyEval_EvalFrameEx in the file ceval.c. There you'll find such part:

关于sys.setswitchinterval(1000):这只与通过Python字节码并处理它有关。这基本上是cev .c文件中PyEval_EvalFrameEx中的CPython的主循环。在那里你会发现这样的部分:

if (_Py_atomic_load_relaxed(&gil_drop_request))
    ...

All the logic with the switch interval is covered in the file ceval_gil.h.

在ceval_gil.h文件中涵盖了具有开关间隔的所有逻辑。

Setting a high switch interval just means that the main loop in PyEval_EvalFrameEx will not be interrupted for a longer time. That does not mean that there aren't other possibilities that the GIL could get released meanwhile and that another thread could run.

设置一个高开关间隔仅仅意味着PyEval_EvalFrameEx中的主循环在更长的时间内不会被中断。这并不意味着GIL可以同时被释放，并且另一个线程可以运行。

PyEval_EvalFrameEx will execute the Python bytecode. Let's assume that this calls time.sleep(1). That will call the native C implementation of the function. You'll find that in time_sleep() in the file timemodule.c. If you follow that code, you'll find this:

PyEval_EvalFrameEx将执行Python字节码。假设这调用了time.sleep(1)。这将调用函数的本机C实现。您将在文件timemodule.c中发现time_sleep()。如果你遵循这个代码，你会发现:

Py_BEGIN_ALLOW_THREADS
err = select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &timeout);
Py_END_ALLOW_THREADS

Thus, the GIL gets released meanwhile. Now, any other thread which is waiting for the GIL could pick it up and run other Python code.

因此，GIL被释放了。现在，任何其他等待GIL的线程都可以选择它并运行其他Python代码。

Theoretically, you could think, if you set a high switch interval and never call any Python code which in turn could release the GIL at some point, you would be safe. Note that this is almost impossible, though. E.g. the GC will get called from time to time and any __del__ of some objects could have various side effects.

理论上，你可以认为，如果你设置了一个高的开关间隔，并且从不调用任何Python代码，而这反过来又能在某个时刻释放GIL，那么你就安全了。注意，这几乎是不可能的。例如，GC会不时被调用，而某些对象的任何__del__都可能产生各种副作用。

#2

You should use threading locks to execute code synchronously between threads. The answer given is somewhat correct but I would use reentrant locals to check again to see if you indeed have the lock.

您应该使用线程锁在线程之间同步执行代码。给出的答案多少是正确的，但我将使用reentrant当地人再次检查，看看你是否真的有锁。

Do not use variables as described in another answer to check for lock possession. The variables can get corrupted between multiple threads. Reentrant locks were meant to solve this problem.

不要像在另一个答案中描述的那样使用变量来检查锁的占有。变量可能在多个线程之间被损坏。可重入锁旨在解决这个问题。

Also what's incorrect in that code is that lock is released assuming the code between doesn't throw exception. so always do in with context or try-catch-finally.

代码中不正确的是，如果代码之间没有抛出异常，锁就会被释放。所以，一定要在上下文中或试着抓住机会。

Here is an excellent article explaining synchronization in Python and threading docs.

这里有一篇优秀的文章，解释Python和线程文档中的同步。

Edit: Answering OP's update on embedding Python in C

编辑:回答OP关于在C中嵌入Python的更新

You misunderstood what he said in the cookbook. PyGILState_Ensure returns the GIL if a GIL is available in the current python interpreter but not C threads which is unknown to the python interpreter.

你误解了他在烹饪书上说的话。如果当前python解释器中有GIL可用，而不是python解释器未知的C线程，那么PyGILState_Ensure返回GIL。

You can't force to get GIL from other threads in the current interpreter. Imagine if you were able to, then basically you will cannibalize all other threads.

不能强制从当前解释器中的其他线程获取GIL。想象一下，如果您能够，那么基本上您将会吃掉所有其他线程。

#3

Python is always executing one thread at a time because of the Global Interpreter Lock. It doesn't do so when multiprocessing is involved. You can see this answer to learn more about the GIL in CPython.

由于全局解释器锁，Python每次总是执行一个线程。当涉及到多处理时，它不会这么做。您可以看到这个答案来了解更多关于CPython中的GIL的信息。

Note, that's pseudocode as I don't know how you're creating threads/using them/which code you're executing in threads.

注意，这是伪代码，因为我不知道如何创建线程/使用它们/在线程中执行哪个代码。

import threading, time

l=threading.Lock()
locked=False

def worker():
    l.acquire()
    locked=True
    #do something
    l.release()

def test():
    while locked:
        time.sleep(10)
    #do something

threads = []
t = threading.Thread(target=worker)
threads.append(t)
t = threading.Thread(target=test)
threads.append(t)
for th in threads:
    th.start()
for th in threads:
    th.join()

Certainly, it may be written better and can be optimized.

当然，它可以写得更好，也可以优化。

#4

As a stop-gap solution (for obvious reasons), the following worked for me:

作为一种权宜之计(原因显而易见)，下面的方法对我起了作用:

def report_memory_consumption():
  sys.setswitchinterval(1000) # longer than the expected run time
  # go through `gc.get_objects()`, check their size and print a summary
  # takes ~5 min to run
  sys.setswitchinterval(0.005) # the default value

If anyone has a better answer, please post it.

如果有人有更好的答案，请贴出来。