最近用Python写了些爬虫,在爬取一个gb2312的页面时,抛出异常:
- UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-23: ordinal not in range(128)
解决方案如下:
首先设置系统的默认编码为utf-8:
- import sys
- reload(sys)
- sys.setdefaultencoding('utf-8')
然后将网页以gbk解码后转为utf-8:
- result = urllib2.urlopen(req).read()
- result = unicode(result,'GBK').encode('UTF-8')
之后就正常了。