为什么我不能捕获这个python异常呢?

I was writing some etcd modules for SaltStack and ran into this strange issue where it's somehow preventing me from catching an exception and I'm interested in how it's doing that. It seems specifically centered around urllib3.

我为SaltStack编写了一些etcd模块，遇到了一个奇怪的问题，它阻止我捕获一个异常，我对它是怎么做到的很感兴趣。它似乎特别集中在urllib3。

A small script ( not salt ):

一个小脚本(不是salt):

import etcd
c = etcd.Client('127.0.0.1', 4001)
print c.read('/test1', wait=True, timeout=2)

And when we run it:

当我们运行它时

[root@alpha utils]# /tmp/etcd_watch.py
Traceback (most recent call last):
  File "/tmp/etcd_watch.py", line 5, in <module>
    print c.read('/test1', wait=True, timeout=2)
  File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
    timeout=timeout)
  File "/usr/lib/python2.6/site-packages/etcd/client.py", line 788, in api_execute
    cause=e
etcd.EtcdConnectionFailed: Connection to etcd failed due to ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.",)

Ok, let's catch that bugger:

好吧，让我们抓住那个混蛋:

#!/usr/bin/python

import etcd
c = etcd.Client('127.0.0.1', 4001)

try:
  print c.read('/test1', wait=True, timeout=2)
except etcd.EtcdConnectionFailed:
  print 'connect failed'

Run it:

运行该程序:

[root@alpha _modules]# /tmp/etcd_watch.py
connect failed

Looks good - it's all working python. So what's the issue? I have this in the salt etcd module:

看起来不错——它都在运行python。那么问题是什么呢?我在salt etcd模块中有这个:

[root@alpha _modules]# cat sjmh.py
import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    return c.read('/test1', wait=True, timeout=2)
  except etcd.EtcdConnectionFailed:
    return False

And when we run that:

当我们运行时

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1173, in _thread_return
        return_data = func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/sjmh.py", line 5, in test
        c.read('/test1', wait=True, timeout=2)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
        timeout=timeout)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 769, in api_execute
        _ = response.data
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 150, in data
        return self.read(cache_content=True)
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 218, in read
        raise ReadTimeoutError(self._pool, None, 'Read timed out.')
    ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.

Hrm, that's weird. etcd's read should have returned etcd.EtcdConnectionFailed. So, let's look at it further. Our module is now this:

人力资源管理,这是奇怪的。etcd的读取应该返回etcd. etcdconnectionfailed。让我们进一步看一下。我们的模块是这样的:

import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    return c.read('/test1', wait=True, timeout=2)
  except Exception as e:
    return str(type(e))

And we get:

我们得到:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    <class 'urllib3.exceptions.ReadTimeoutError'>

Ok, so we know that we can catch this thing. And we now know it threw a ReadTimeoutError, so let's catch that. The newest version of our module:

我们知道我们可以抓住这个东西。现在我们知道它带来了恐惧，我们来看看。我们模块的最新版本:

import etcd
import urllib3.exceptions

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError as e:
    return 'caught ya!'
  except Exception as e:
    return str(type(e))

And our test..

和我们的测试。

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    <class 'urllib3.exceptions.ReadTimeoutError'>

Er, wait, what? Why didn't we catch that? Exceptions work, right.. ?

呃,等等,什么?为什么我们没抓住?异常工作吧. .吗?

How about if we try and catch the base class from urllib3..

如果我们尝试从urllib3获取基类怎么办?

[root@alpha _modules]# cat sjmh.py
import etcd
import urllib3.exceptions

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.HTTPError:
    return 'got you this time!'

Hope and pray..

希望和祈祷. .

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1173, in _thread_return
        return_data = func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/sjmh.py", line 7, in test
        c.read('/test1', wait=True, timeout=2)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
        timeout=timeout)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 769, in api_execute
        _ = response.data
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 150, in data
        return self.read(cache_content=True)
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 218, in read
        raise ReadTimeoutError(self._pool, None, 'Read timed out.')
    ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.

BLAST YE! Ok, let's try a different method that returns a different etcd Exception. Our module now looks like this:

爆炸你们!好的，让我们尝试一个不同的方法来返回一个不同的etcd异常。我们的模块现在是这样的:

import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.delete('/')
  except etcd.EtcdRootReadOnly:
    return 'got you this time!'

And our run:

和我们的运行:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    got you this time!

As a final test, I made this module, which I can run either from straight python, or as a salt module..

作为最后的测试，我制作了这个模块，我可以直接从python或者salt模块运行它。

import etcd
import urllib3

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError:
    return 'got you this time!'
  except etcd.EtcdConnectionFailed:
    return 'cant get away from me!'
  except etcd.EtcdException:
    return 'oh no you dont'
  except urllib3.exceptions.HTTPError:
    return 'get back here!'
  except Exception as e:
    return 'HOW DID YOU GET HERE? {0}'.format(type(e))

if __name__ == "__main__":
  print test()

Through python:

通过python:

[root@alpha _modules]# python ./sjmh.py
cant get away from me!

Through salt:

通过盐:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    HOW DID YOU GET HERE? <class 'urllib3.exceptions.ReadTimeoutError'>

So, we can catch exceptions from etcd that it throws. But, while we normally are able to catch the urllib3 ReadTimeoutError when we run python-etcd by its lonesome, when I run it through salt, nothing seems to be able to catch that urllib3 exception, except a blanket 'Exception' clause.

因此，我们可以捕获它抛出的etcd中的异常。但是，当我们运行python-etcd时，我们通常能够捕捉到urllib3 ReadTimeoutError，当我在salt中运行它时，似乎没有什么能够捕捉到urllib3异常，除了一个毯子“exception”子句。

I can do that, but I'm really curious as to what the heck salt is doing that's making it so that an exception is uncatchable. I've never seen this before when working with python, so I'd be curious as to how it's happening and how I can work around it.

我可以这么做，但我真的很好奇为什么盐会这么做，以至于一个异常是不可避免的。我以前在使用python时从未见过这种情况，所以我很好奇它是如何发生的，以及我如何处理它。

Edit:

编辑:

So I was finally able to catch it.

所以我终于抓住了它。

import etcd
import urllib3.exceptions
from urllib3.exceptions import ReadTimeoutError

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError:
    return 'caught 1'
  except urllib3.exceptions.HTTPError:
    return 'caught 2'
  except ReadTimeoutError:
    return 'caught 3'
  except etcd.EtcdConnectionFailed as ex:
    return 'cant get away from me!'
  except Exception as ex:
    return 'HOW DID YOU GET HERE? {0}'.format(type(ex))

if __name__ == "__main__":
  print test()

And when run:

和运行时:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    caught 3

It still doesn't make sense though. From what I know of exceptions, the return should be 'caught 1'. Why should I have to import the name of the exception directly, rather than just using full class name?

但这仍然没有意义。根据我所知道的例外情况，返回应该被“捕获1”。为什么我必须直接导入异常的名称，而不是只使用完整的类名?

MORE EDITS!

更多的编辑!

So, adding the comparison between the two classes produces 'False' - which is sorta obvious, because the except clause wasn't working, so those couldn't be the same.

因此，在两个类之间添加比较会产生“False”——这很明显，因为except子句不起作用，所以它们不可能是相同的。

I added the following to the script, right before I call the c.read().

在调用c.read()之前，我在脚本中添加了以下内容。

log.debug(urllib3.exceptions.ReadTimeoutError.__module__)
log.debug(ReadTimeoutError.__module__)

And now I get this in the log:

现在我把它写在对数里

[DEBUG   ] requests.packages.urllib3.exceptions
[DEBUG   ] urllib3.exceptions

So, that appears to be the reason that is getting caught the way it is. This is also reproducible by just downloading the etcd and requests library and doing something like this:

所以，这似乎就是为什么会被发现的原因。这也可以通过下载etcd和请求库进行复制

#!/usr/bin/python

#import requests
import etcd

c = etcd.Client('127.0.0.1', 4001)
c.read("/blah", wait=True, timeout=2)

You'll end up getting the 'right' exception raised - etcd.EtcdConnectionFailed. However, uncomment 'requests' and you'll end up with urllib3.exceptions.ReadTimeoutError, because etcd now no longer catches the exception.

您最终会得到一个“正确”的异常——etcd.EtcdConnectionFailed。但是，如果不注释“请求”，就会出现urllib3.exception。ReadTimeoutError，因为etcd现在不再捕获异常。

So it appears that when requests is imported, it rewrites the urllib3 exceptions, and any other module that is trying to catch those, fails. Also, it appears that newer versions of requests do not have this issue.

因此，当导入请求时，它会重写urllib3异常，而试图捕获这些异常的任何其他模块都将失败。而且，新版本的请求似乎没有这个问题。

1 个解决方案

#1

My answer below is a little of speculation, because I cannot prove it on practice with these exact libraries (to start with I cannot reproduce your error as it also depends on libraries versions and how they are installed), but nevertheless shows one of the possible ways of this happening:

下面我的回答是一些猜测，因为我无法用这些精确的库来证明它(首先，我不能复制您的错误，因为它也依赖于库版本和它们是如何安装的)，但是仍然显示了其中一个可能的方法:

The very last example gives a good clue: the point is indeed that at different moments in the time of the program execution, name urllib3.exceptions.ReadTimeoutError may refer to different classes. ReadTimeoutError is, just like for every other module in Python, is simply a name in the urllib3.exceptions namespace, and can be reassigned (but it doesn't mean it is a good idea to do so).

最后一个示例提供了一个很好的线索:关键是在程序执行的不同时刻，命名为urllib3.exception。ReadTimeoutError可能指不同的类。ReadTimeoutError是，就像Python中的其他模块一样，只是urllib3中的一个名称。异常名称空间，并且可以重新分配(但这并不意味着这样做是一个好主意)。

When referring to this name by its fully-qualified "path" - we are guaranteed to refer to the actual state of it by the time we refer to it. However, when we first import it like from urllib3.exceptions import ReadTimeoutError - it brings name ReadTimeoutError into the namespace which does the import, and this name is bound to the value of urllib3.exceptions.ReadTimeoutError by the time of this import. Now, if some other code reassigns later the value for urllib3.exceptions.ReadTimeoutError - the two (its "current"/"latest" value and the previously imported one) might be actually different - so technically you might end up having two different classes. Now, which exception class will be actually raised - this depends on how the code which raises the error uses it: if they previously imported the ReadTimeoutError into their namespace - then this one (the "original") will be raised.

当使用完全限定的“路径”来引用这个名称时，我们保证在引用它时引用它的实际状态。然而，当我们第一次像从urllib3导入时。异常导入readtimeou恐怖——它将名称ReadTimeoutError引入到执行导入的名称空间中，这个名称与urllib3.exception的值绑定在一起。在此输入时读取时间。现在，如果其他一些代码重新分配了urllib3.exception的值。ReadTimeoutError——这两个(它的“当前”/“最新”值和先前导入的值)实际上可能是不同的——因此从技术上来说，您可能最终拥有两个不同的类。现在，实际将引发哪个异常类——这取决于引起错误的代码如何使用它:如果他们之前将ReadTimeoutError导入到他们的名称空间中——那么这个(“原始”)将被引发。

To verify if this is the case you might add the following to the except ReadTimeoutError block:

要验证这种情况是否存在，您可以在ReadTimeoutError block中添加以下内容:

print(urllib3.exceptions.ReadTimeoutError == ReadTimeoutError)

If this prints False - it proves that by the time the exception is raised, the two "references" refer to different classes indeed.

如果这个输出为False——它证明在引发异常时，两个“引用”确实引用了不同的类。

---

A simplified example of a poor implementation which can yield similar result:

一个糟糕的实现的简化示例，可以产生类似的结果:

File api.py (properly designed and exists happily by itself):

文件的api。py(设计合理，自我快乐的存在):

class MyApiException(Exception):
    pass

def foo():
    raise MyApiException('BOOM!')

File apibreaker.py (the one to blame):

文件apibreaker。py(该责备的人):

import api

class MyVeryOwnException(Exception):
    # note, this doesn't extend MyApiException,
    # but creates a new "branch" in the hierarhcy
    pass

# DON'T DO THIS AT HOME!
api.MyApiException = MyVeryOwnException

File apiuser.py:

文件apiuser.py:

import api
from api import MyApiException, foo
import apibreaker

if __name__ == '__main__':
    try:
        foo()
    except MyApiException:
        print("Caught exception of an original class")
    except api.MyApiException:
        print("Caught exception of a reassigned class")

When executed:

当执行:

$ python apiuser.py
Caught exception of a reassigned class

If you remove the line import apibreaker - clearly then all goes back to their places as it should be.

如果您删除了行导入apibreaker——很明显，所有的都回到了它们应该的位置。

This is a very simplified example, yet illustrative enough to show that when a class is defined in some module - newly created type (object representing new class itself) is "added" under its declared class name to the module's namespace. As with any other variable - its value can be technically modified. Same thing happens to functions.

这是一个非常简单的示例，但足以说明在某些模块中定义类时——新创建的类型(对象表示新类本身)将在其声明的类名下“添加”到模块的名称空间。与任何其他变量一样，它的值可以在技术上修改。函数也是如此。