如何使内置的容器(集合，dicts，列表)线程安全?

I understand from this question that if I want to have a set which is thread-safe I have to implement the thread-safety part on my own.

我从这个问题中了解到，如果我想要一个线程安全的集合，我必须自己实现线程安全部分。

Therefore I could come up with:

因此我可以提出:

from threading import Lock

class LockedSet(set):
    """A set where add() and remove() are thread-safe"""

    def __init__(self, *args, **kwargs):
        # Create a lock
        self._lock = Lock()
        # Call the original __init__
        super(LockedSet, self).__init__(*args, **kwargs)

    def add(self, elem):
        self._lock.acquire()
        try:
            super(LockedSet, self).add(elem)
        finally:
            self._lock.release()

    def remove(self, elem):
        self._lock.acquire()
        try:
            super(LockedSet, self).remove(elem)
        finally:
            self._lock.release()

So, of course only add() and remove() are thread-safe in this implementation. The other methods are not because they were not overwritten in the subclass.

因此，当然只在这个实现中添加()和remove()是线程安全的。其他方法并不是因为它们没有在子类中被覆盖。

Now, the pattern is pretty simple: acquire lock, call original method, release lock. If I follow the logic above, I would have to overwrite all methods exposed by set in essentially the same way, e.g.:

现在，模式很简单:获取锁，调用原始方法，释放锁。如果我遵循上面的逻辑，我将不得不重写所有以相同的方式设置的方法，例如:

(pseudo-code)

(伪代码)

def <method>(<args>):
    1. acquire lock
    2. try:
    3.     call original method passing <args>
    4. finally:
    5.     release lock

(/pseudo-code)

(/伪代码)

This is not only tedious but also prone to errors. So, any ideas/suggestions on how to approach this in a better way?

这不仅乏味而且容易出错。那么，对于如何更好地处理这个问题，有什么想法/建议吗?

4 个解决方案

#1

You can use Python's metaprogramming facilities to accomplish this. (Note: written quickly and not thoroughly tested.) I prefer to use a class decorator.

您可以使用Python的元编程工具来实现这一点。(注:写得很快，没有经过彻底的测试。)我更喜欢使用类装饰器。

I also think you may need to lock more than add and remove to make a set thread-safe, but I'm not sure. I'll ignore that problem and just concentrate on your question.

我还认为您可能需要锁定更多的内容，而不是添加和删除，以使设置线程安全，但我不确定。我会忽略这个问题，专注于你的问题。

Also consider whether delegation (proxying) is a better fit than subclassing. Wrapping objects is the usual approach in Python.

还要考虑委托(代理)是否比子类化更合适。在Python中，包装对象是通常的方法。

Finally, there is no "magic wand" of metaprogramming that will magically add fine-grained locking to any mutable Python collection. The safest thing to do is to lock any method or attribute access using RLock, but this is very coarse-grained and slow and probably still not a guarantee that your object will be thread-safe in all cases. (For example, you may have a collection that manipulates another non-threadsafe object accessible to other threads.) You really do need to examine each and every data structure and think about what operations are atomic or require locks and which methods might call other methods using the same lock (i.e., deadlock itself).

最后，没有元编程的“魔棒”，它会神奇地为任何可变的Python集合添加细粒度的锁定。最安全的做法是使用RLock锁定任何方法或属性访问，但这是非常粗粒度和缓慢的，而且可能仍然不能保证您的对象在所有情况下都是线程安全的。(例如，您可能有一个收集来操作其他线程可以访问的另一个非线程安全对象)。您确实需要检查每个数据结构，并考虑哪些操作是原子性的，或者需要锁，哪些方法可以使用相同的锁调用其他方法(例如:、死锁本身)。

That said, here are some techniques at your disposal in increasing order of abstraction:

也就是说，这里有一些技巧可以在你的处理中增加抽象的顺序:

Delegation

class LockProxy(object):
    def __init__(self, obj):
        self.__obj = obj
        self.__lock = RLock()
        # RLock because object methods may call own methods
    def __getattr__(self, name):
        def wrapped(*a, **k):
            with self.__lock:
                getattr(self.__obj, name)(*a, **k)
        return wrapped

lockedset = LockProxy(set([1,2,3]))

Context manager

class LockedSet(set):
    """A set where add(), remove(), and 'in' operator are thread-safe"""

    def __init__(self, *args, **kwargs):
        self._lock = Lock()
        super(LockedSet, self).__init__(*args, **kwargs)

    def add(self, elem):
        with self._lock:
            super(LockedSet, self).add(elem)

    def remove(self, elem):
        with self._lock:
            super(LockedSet, self).remove(elem)

    def __contains__(self, elem):
        with self._lock:
            super(LockedSet, self).__contains__(elem)

Decorator

def locked_method(method):
    """Method decorator. Requires a lock object at self._lock"""
    def newmethod(self, *args, **kwargs):
        with self._lock:
            return method(self, *args, **kwargs)
    return newmethod

class DecoratorLockedSet(set):
    def __init__(self, *args, **kwargs):
        self._lock = Lock()
        super(DecoratorLockedSet, self).__init__(*args, **kwargs)

    @locked_method
    def add(self, *args, **kwargs):
        return super(DecoratorLockedSet, self).add(elem)

    @locked_method
    def remove(self, *args, **kwargs):
        return super(DecoratorLockedSet, self).remove(elem)

Class Decorator

I think this is the cleanest and easiest-to-understand of the abstract methods, so I've expanded it to allow one to specify the methods to lock and a lock object factory.

我认为这是最干净、最容易理解的抽象方法，因此我将它扩展为允许指定锁和锁对象工厂的方法。

def lock_class(methodnames, lockfactory):
    return lambda cls: make_threadsafe(cls, methodnames, lockfactory)

def lock_method(method):
    if getattr(method, '__is_locked', False):
        raise TypeError("Method %r is already locked!" % method)
    def locked_method(self, *arg, **kwarg):
        with self._lock:
            return method(self, *arg, **kwarg)
    locked_method.__name__ = '%s(%s)' % ('lock_method', method.__name__)
    locked_method.__is_locked = True
    return locked_method


def make_threadsafe(cls, methodnames, lockfactory):
    init = cls.__init__
    def newinit(self, *arg, **kwarg):
        init(self, *arg, **kwarg)
        self._lock = lockfactory()
    cls.__init__ = newinit

    for methodname in methodnames:
        oldmethod = getattr(cls, methodname)
        newmethod = lock_method(oldmethod)
        setattr(cls, methodname, newmethod)

    return cls


@lock_class(['add','remove'], Lock)
class ClassDecoratorLockedSet(set):
    @lock_method # if you double-lock a method, a TypeError is raised
    def frobnify(self):
        pass

Override Attribute access with __getattribute__

class AttrLockedSet(set):
    def __init__(self, *args, **kwargs):
        self._lock = Lock()
        super(AttrLockedSet, self).__init__(*args, **kwargs)

    def __getattribute__(self, name):
        if name in ['add','remove']:
            # note: makes a new callable object "lockedmethod" on every call
            # best to add a layer of memoization
            lock = self._lock
            def lockedmethod(*args, **kwargs):
                with lock:
                    return super(AttrLockedSet, self).__getattribute__(name)(*args, **kwargs)
            return lockedmethod
        else:
            return super(AttrLockedSet, self).__getattribute__(name)

Dynamically-added wrapper methods with __new__

class NewLockedSet(set):
    def __new__(cls, *args, **kwargs):
        # modify the class by adding new unbound methods
        # you could also attach a single __getattribute__ like above
        for membername in ['add', 'remove']:
            def scoper(membername=membername):
                # You can also return the function or use a class
                def lockedmethod(self, *args, **kwargs):
                    with self._lock:
                        m = getattr(super(NewLockedSet, self), membername)
                        return m(*args, **kwargs)
                lockedmethod.__name__ = membername
                setattr(cls, membername, lockedmethod)
        self = super(NewLockedSet, cls).__new__(cls, *args, **kwargs)
        self._lock = Lock()
        return self

Dynamically-added wrapper methods with __metaclass__

def _lockname(classname):
    return '_%s__%s' % (classname, 'lock')

class LockedClass(type):
    def __new__(mcls, name, bases, dict_):
        # we'll bind these after we add the methods
        cls = None
        def lockmethodfactory(methodname, lockattr):
            def lockedmethod(self, *args, **kwargs):
                with getattr(self, lockattr):
                    m = getattr(super(cls, self), methodname)
                    return m(*args,**kwargs)
            lockedmethod.__name__ = methodname
            return lockedmethod
        lockattr = _lockname(name)
        for methodname in ['add','remove']:
            dict_[methodname] = lockmethodfactory(methodname, lockattr)
        cls = type.__new__(mcls, name, bases, dict_)
        return cls

    def __call__(self, *args, **kwargs):
        #self is a class--i.e. an "instance" of the LockedClass type
        instance = super(LockedClass, self).__call__(*args, **kwargs)
        setattr(instance, _lockname(self.__name__), Lock())
        return instance



class MetaLockedSet(set):
    __metaclass__ = LockedClass

Dynamically-created Metaclasses

def LockedClassMetaFactory(wrapmethods):
    class LockedClass(type):
        def __new__(mcls, name, bases, dict_):
            # we'll bind these after we add the methods
            cls = None
            def lockmethodfactory(methodname, lockattr):
                def lockedmethod(self, *args, **kwargs):
                    with getattr(self, lockattr):
                        m = getattr(super(cls, self), methodname)
                        return m(*args,**kwargs)
                lockedmethod.__name__ = methodname
                return lockedmethod
            lockattr = _lockname(name)
            for methodname in wrapmethods:
                dict_[methodname] = lockmethodfactory(methodname, lockattr)
            cls = type.__new__(mcls, name, bases, dict_)
            return cls

        def __call__(self, *args, **kwargs):
            #self is a class--i.e. an "instance" of the LockedClass type
            instance = super(LockedClass, self).__call__(*args, **kwargs)
            setattr(instance, _lockname(self.__name__), Lock())
            return instance
    return LockedClass

class MetaFactoryLockedSet(set):
    __metaclass__ = LockedClassMetaFactory(['add','remove'])

I'll bet using a simple, explicit try...finally doesn't look so bad now, right?

我打赌用一个简单的，明确的尝试…终于看起来不那么糟糕了，对吧?

Exercise for the reader: let the caller pass in their own Lock() object (dependency injection) using any of these methods.

练习给读者:让调用者通过他们自己的锁()对象(依赖注入)使用任何这些方法。

#2

This is my first attempt to play with decorators (although my code doesn't actually use the @decorate syntax), and I don't have much experience with multi-threading/multiprocessing. With that disclaimer, though, here's an attempt I made:

这是我第一次尝试使用decorator(尽管我的代码实际上并不使用@修饰语法)，而且我对多线程/多处理没有太多的经验。不过，有了这个免责声明，我尝试了一下:

from multiprocessing import Lock

def decorate_all(obj):
    lock = Lock()
    #you'll want to make this more robust:
    fnc_names = [fnctn for fnctn in dir(obj) if '__' not in fnctn]
    for name in fnc_names:
        print 'decorating ' + name
        fnc = getattr(obj, name)
        setattr(obj, name, decorate(fnc, lock))
    return obj

def decorate(fnctn, lock):
    def decorated(*args):
        print 'acquiring lock'
        lock.acquire()
        try:
            print 'calling decorated function'
            return fnctn(*args)
        finally:
            print 'releasing lock'
            lock.release()
    return decorated


def thread_safe(superclass):
    lock = Lock()
    class Thread_Safe(superclass):
        def __init__(self, *args, **kwargs):
            super(Thread_Safe, self).__init__(*args, **kwargs)
    return decorate_all(Thread_Safe)


>>> thread_safe_set = thread_safe(set)
decorating add
decorating clear
decorating copy
decorating difference
decorating difference_update
decorating discard
decorating intersection
decorating intersection_update
decorating isdisjoint
decorating issubset
decorating issuperset
decorating pop
decorating remove
decorating symmetric_difference
decorating symmetric_difference_update
decorating union
decorating update
>>> s = thread_safe_set()
>>> s.add(1)
acquiring lock
calling decorated function
releasing lock
>>> s.add(4)
acquiring lock
calling decorated function
releasing lock
>>> s.pop()
acquiring lock
calling decorated function
releasing lock
1
>>> s.pop()
acquiring lock
calling decorated function
releasing lock
4
>>>

#3

[Indeed, see the comments, it is not true]

[确实，看到评论了，这不是真的]

If you are running CPython you can see from the set source code that it doesn't release the GIL (http://hg.python.org/cpython/file/db20367b20de/Objects/setobject.c) so all its operations should be atomic.

如果您正在运行CPython，您可以从set源代码中看到它没有发布GIL (http://hg.python.org/cpython/file/db20367b20de/Objects/setobject.c)，所以它的所有操作都应该是原子的。

If it is all what you need and you are sure to run your code on CPython you can just use it directly.

如果这是您所需要的，并且您一定要在CPython上运行您的代码，您可以直接使用它。

#4

You can implement your own context manager:

您可以实现自己的上下文管理器:

class LockableSet:
    def __enter__(self):
        self.lock()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        #Do what you want with the error
        self.unlock()

with LockableSet() as s:
    s.whatever()
    raise Exception()

No matter what, the object's __exit__ method will be called at the end. More detailed informations are available here (python official docs).

无论如何，对象的__exit__方法将在最后调用。这里有更多的详细信息(python官方文档)。

Another use for this could be a lock decorator for methods, like this:

对于这种方法的另一个用途可能是用于方法的锁装饰器:

def lock(func):
    def safe_func(self, *args, **kwargs):
        with self:
            func(self, *args, **kwargs)
    return safe_func