Python's popular Requests library is said to be thread-safe on its home page, but no further details are given. If I call requests.session()
, can I then safely pass this object to multiple threads like so:
据称Python的流行Requests库在其主页上是线程安全的,但没有给出进一步的细节。如果我调用requests.session(),我可以安全地将此对象传递给多个线程,如下所示:
session = requests.session()
for i in xrange(thread_count):
threading.Thread(
target=target,
args=(session,),
kwargs={}
)
and make requests using the same connection pool in multiple threads?
并在多个线程中使用相同的连接池发出请求?
If so, is this the recommended approach, or should each thread be given its own connection pool? (Assuming the total size of all the individual connection pools summed to the size of what would be one big connection pool, like the one above.) What are the pros and cons of each approach?
如果是这样,这是推荐的方法,还是应该为每个线程提供自己的连接池? (假设所有单个连接池的总大小总计为一个大连接池的大小,如上所述。)每种方法的优缺点是什么?
2 个解决方案
#1
18
After reviewing the source of requests.session
, I'm going to say the session object might be thread-safe, depending on the implementation of CookieJar being used.
在查看了requests.session的源代码后,我将说会话对象可能是线程安全的,具体取决于所使用的CookieJar的实现。
Session.prepare_request
reads from self.cookies
, and Session.send
calls extract_cookies_to_jar(self.cookies, ...)
, and that calls jar.extract_cookies(...)
(jar
being self.cookies
in this case).
Session.prepare_request从self.cookies读取,Session.send调用extract_cookies_to_jar(self.cookies,...),并调用jar.extract_cookies(...)(在这种情况下jar是self.cookies)。
The source for Python 2.7's cookielib
acquires a lock (threading.RLock
) while it updates the jar, so it appears to be thread-safe. On the other hand, the documentation for cookielib
says nothing about thread-safety, so maybe this feature should not be depended on?
Python 2.7的cookielib的源代码在更新jar时获取了一个锁(threading.RLock),因此它看起来是线程安全的。另一方面,cookielib的文档没有说明线程安全,所以也许不应该依赖这个功能?
UPDATE
UPDATE
If your threads are mutating any attributes of the session object such as headers
, proxies
, stream
, etc. or calling the mount
method or using the session with the with
statement, etc. then it is not thread-safe.
如果您的线程正在改变会话对象的任何属性,例如头,代理,流等,或者调用mount方法或使用带有with语句的会话等,则它不是线程安全的。
#2
15
https://github.com/kennethreitz/requests/issues/1871 implies that Session is not thread-safe, and that at least one maintainer recommends one Session per thread.
https://github.com/kennethreitz/requests/issues/1871暗示Session不是线程安全的,并且至少有一个维护者建议每个线程使用一个Session。
I just opened https://github.com/kennethreitz/requests/issues/2766 to clarify the documentation.
我刚刚打开https://github.com/kennethreitz/requests/issues/2766来澄清文档。
#1
18
After reviewing the source of requests.session
, I'm going to say the session object might be thread-safe, depending on the implementation of CookieJar being used.
在查看了requests.session的源代码后,我将说会话对象可能是线程安全的,具体取决于所使用的CookieJar的实现。
Session.prepare_request
reads from self.cookies
, and Session.send
calls extract_cookies_to_jar(self.cookies, ...)
, and that calls jar.extract_cookies(...)
(jar
being self.cookies
in this case).
Session.prepare_request从self.cookies读取,Session.send调用extract_cookies_to_jar(self.cookies,...),并调用jar.extract_cookies(...)(在这种情况下jar是self.cookies)。
The source for Python 2.7's cookielib
acquires a lock (threading.RLock
) while it updates the jar, so it appears to be thread-safe. On the other hand, the documentation for cookielib
says nothing about thread-safety, so maybe this feature should not be depended on?
Python 2.7的cookielib的源代码在更新jar时获取了一个锁(threading.RLock),因此它看起来是线程安全的。另一方面,cookielib的文档没有说明线程安全,所以也许不应该依赖这个功能?
UPDATE
UPDATE
If your threads are mutating any attributes of the session object such as headers
, proxies
, stream
, etc. or calling the mount
method or using the session with the with
statement, etc. then it is not thread-safe.
如果您的线程正在改变会话对象的任何属性,例如头,代理,流等,或者调用mount方法或使用带有with语句的会话等,则它不是线程安全的。
#2
15
https://github.com/kennethreitz/requests/issues/1871 implies that Session is not thread-safe, and that at least one maintainer recommends one Session per thread.
https://github.com/kennethreitz/requests/issues/1871暗示Session不是线程安全的,并且至少有一个维护者建议每个线程使用一个Session。
I just opened https://github.com/kennethreitz/requests/issues/2766 to clarify the documentation.
我刚刚打开https://github.com/kennethreitz/requests/issues/2766来澄清文档。