最近有个小需求想使用Scrapy库做抓取，但公司开发机操作系统版本老旧，导致系统默认的openssl库版本也很低（OpenSSL 0.9.7a Feb 19 2003），最终导致安装Scrapy非常麻烦。趁着元旦假期，决定用自己安装好的高版本openssl库作为依赖，重新编译安装开发机的Python环境。
安装及其后使用中遇到一些关于openssl的问题，或许对自己后续工作或别人有参考价值，作为笔记，记录于此。
源码编译安装流程如下：
1. Python官网下载源码包（version 2.7.8）并解压
2. 通过configure配置Makefile参数假定先前已在/home/slvher/tools/openssl-1.0.1j路径下成功安装好高版本openssl，那么，可以通过下面的环境变量来设置gcc的编译/链接参数：
export LDFLAGS="-lssl -lcrypto -Wl,-rpath=/home/slvher/tools/openssl-1.0.1j/lib/"然后运行configure工具：
export CPPFLAGS="-I/home/slvher/tools/openssl-1.0.1j/include"
./configure --prefix=/home/slvher/tools/Python-2.7.8configure运行完后，会生成Makefile，check无误后，执行make && make install就能完成Python安装。
3. 验证Python支持的ssl版本
Python安装成功且其bin路径加入PATH环境变量后，可通过下面的命令来验证是否正确支持高版本的openssl库：
python -c "import ssl; print ssl.OPENSSL_VERSION"例如我机器上的输出结果如下：
OpenSSL 1.0.1j 15 Oct 20144. 后续使用依赖openssl的包时，可能遇到的问题及解决思路
在依赖了高版本openssl库（普通用户自定义安装在非系统标准路径下）的Python环境下，通过easy_install或pip安装Python第三方包时，可能会报类似于下面的错误（以pip安装Scrapy为例）：
此处省略上文...受*这篇帖子 python easy_install fails with SSL certificate error for all packages的启发，结合pip的安装日志（本文最后会贴出报错日志），最终确定报错的原因是由于安装过程中setuptools会通过https访问PyPI来拉取某个库依赖的其它包，而通过https访问PyPI源时，它使用的是高版本的openssl，但其加载的certificate证书却是与系统默认的旧版openssl库配套的证书，所以会报出"certificate verify failed"的错误。
[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
此处省略下文...
关于这个错误，有人在github给pip提了issue ( pip --cert not working as expected )，但实际上这应该是setuptools的问题。
解决思路1：
为setuptools指定新的证书地址，具体方法可参考这篇资料OpenSSL Errors and Rails–Certificate Verify Failed，但我按帖中提到的export SSL_CERT_FILE的方法试验过，报错依旧。估计这个方法对Ruby on Rails环境奏效，Python环境未必有疗效。
setuptools是由于ssl证书认证失败而报错，且setuptools加载ca证书的地址是硬编码的，并未没有开放参数设置给使用者。因此，思路1的方法暂时无解。作为替代方案，根据github/pip关于这个问题的讨论帖，我们可以绕过setuptools，直接用pip去拉取依赖：
pip install --cert /home/slvher/tools/https-ca/https/cacert.pem -U -i https://pypi.python.org/simple six不出意外的话，依赖包（上面的pip install命令中以Scrapy依赖的six包为例）会拉取并编译成功，这表明依赖了自定义安装的高版本openssl库的Python环境其实是可以正常工作的，这个报错可能是setuptools的小bug而已。
下面是上文提到的运行"pip install Scrapy"会由于ssl certificate证书太旧导致报错的日志片段。
可以看到，出问题的地方从"Running setup.py install for cryptography"那行（第33行）开始，而setup.py install调用了setuptools，所以可以推断，认证失败的确是由setuptools引起。而用pip直接拉取依赖包不会报错也证实了这个推断。
$ pip install Scrapy【参考资料】
Requirement already satisfied (use --upgrade to upgrade): Scrapy in ./lib/python2.7/site-packages/Scrapy-0.24.4-py2.7.egg
Collecting Twisted>=10.0.0 (from Scrapy)
Using cached Twisted-14.0.2.tar.bz2
Collecting w3lib>=1.8.0 (from Scrapy)
Using cached w3lib-1.10.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy)
Using cached queuelib-1.2.2-py2.py3-none-any.whl
Collecting lxml (from Scrapy)
Using cached lxml-3.4.1.tar.gz
/home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
warnings.warn(msg)
Building lxml version 3.4.1.
Building without Cython.
Using build configuration of libxslt 1.1.26
Building against libxml2/libxslt in the following directory: /home/slvher/.jumbo/lib
Collecting pyOpenSSL (from Scrapy)
Using cached pyOpenSSL-0.14.tar.gz
Collecting cssselect>=0.9 (from Scrapy)
Using cached cssselect-0.9.1.tar.gz
Collecting six>=1.5.2 (from Scrapy)
Using cached six-1.8.0-py2.py3-none-any.whl
Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)
Using cached zope.interface-4.1.2.tar.gz
Collecting cryptography>=0.2.1 (from pyOpenSSL->Scrapy)
Using cached cryptography-0.7.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-10.1-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=0.8 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in ./lib/python2.7/site-packages (from cffi>=0.8->cryptography>=0.2.1->pyOpenSSL->Scrapy)
Installing collected packages: cryptography, zope.interface, six, cssselect, pyOpenSSL, lxml, queuelib, w3lib, Twisted
Running setup.py install for cryptography
Download error on https://pypi.python.org/simple/six/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
Couldn't find index page for 'six' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for six>=1.4.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-G9LiJL/cryptography/setup.py", line 337, in <module>
**keywords_with_side_effects(sys.argv)
File "/home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 266, in __init__
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 312, in fetch_build_eggs
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 753, in resolve
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1005, in best_match
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1017, in obtain
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 379, in fetch_build_egg
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 613, in easy_install
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six>=1.4.1')
Complete output from command /home/slvher/tools/Python-2.7.8/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-G9LiJL/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-eZ3kZg-record/install-record.txt --single-version-externally-managed --compile:
Download error on https://pypi.python.org/simple/six/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
Couldn't find index page for 'six' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for six>=1.4.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-G9LiJL/cryptography/setup.py", line 337, in <module>
**keywords_with_side_effects(sys.argv)
File "/home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 266, in __init__
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 312, in fetch_build_eggs
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 753, in resolve
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1005, in best_match
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1017, in obtain
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 379, in fetch_build_egg
File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 613, in easy_install
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six>=1.4.1')
----------------------------------------
Command "/home/slvher/tools/Python-2.7.8/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-G9LiJL/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-eZ3kZg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-G9LiJL/cryptography
1. python easy_install fails with SSL certificate error for all packages
2. pip --cert not working as expected
======================== EOF =======================