在做58同城爬二手房时，由于房产详情页内对价格进行了转码处理，所以只能从获取详情页url时同时获取该url对应房产的价格，并通过meta传递给下回调函数
现在问题是，在回调函数中找不到原函数meta信息：
Traceback (most recent call last):
File "c:\users\chen\python36\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "c:\users\chen\python36\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 30, in process_spider_output
for x in result:
File "c:\users\chen\python36\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "c:\users\chen\python36\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "c:\users\chen\python36\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "D:\oldHouse\oldHouse\spiders\old58House.py", line 69, in parse_detail
item = response.meta['item']
KeyError: 'item'
我第一猜想是由于请求经过各种retry重试，和rediret到jump_url、firewall上面，在这个过程中retry和redirect中间件是不是只拿到相应的url而没有保存原来的meta信息，这两个中间件对请求是怎么处理的
1.先看redirect中间件：scrapy.downloadermiddlewares.redirect.BaseRedirectMiddleware，重点代码位于_redirect方法
def _redirect(self, redirected, request, spider, reason):
ttl = request.meta.setdefault('redirect_ttl', self.max_redirect_times)
redirects = request.meta.get('redirect_times', 0) + 1 if ttl and redirects <= self.max_redirect_times:
redirected.meta['redirect_times'] = redirects
redirected.meta['redirect_ttl'] = ttl - 1
redirected.meta['redirect_urls'] = request.meta.get('redirect_urls', []) + \
[request.url]
redirected.dont_filter = request.dont_filter
redirected.priority = request.priority + self.priority_adjust
logger.debug("Redirecting (%(reason)s) to %(redirected)s from %(request)s",
{'reason': reason, 'redirected': redirected, 'request': request},
extra={'spider': spider})
return redirected
else:
logger.debug("Discarding %(request)s: max redirections reached",
{'request': request}, extra={'spider': spider})
raise IgnoreRequest("max redirections reached")
可以看到_redirect方法涉及到meta操作主要是刷新最大重试次数和已经重试次数，并没有丢失原有的meta信息
2.再看retry中间件：scrapy.downloadermiddlewares.retry.BaseRetryMiddleware，重点代码位于_redirect方法
def _retry(self, request, reason, spider):
retries = request.meta.get('retry_times', 0) + 1 retry_times = self.max_retry_times if 'max_retry_times' in request.meta:
retry_times = request.meta['max_retry_times'] stats = spider.crawler.stats
if retries <= retry_times:
logger.debug("Retrying %(request)s (failed %(retries)d times): %(reason)s",
{'request': request, 'retries': retries, 'reason': reason},
extra={'spider': spider})
retryreq = request.copy()
retryreq.meta['retry_times'] = retries
retryreq.dont_filter = True
retryreq.priority = request.priority + self.priority_adjust if isinstance(reason, Exception):
reason = global_object_name(reason.__class__) stats.inc_value('retry/count')
stats.inc_value('retry/reason_count/%s' % reason)
return retryreq
可以看到_retry方法涉及到meta操作主要是刷新重试次数，并未丢失原有meta信息
事实上，框架没有错，我开始的猜想也没错，错在我定制的edirect中间件修改了meta信息：
return Request(request.url, callback=spider.parse_detail, dont_filter=True)
我在real_url重定向到firewall上时，不允许它重定向，而是继续请求到real_url，重要的是没有携带real_url的meta信息，所以meta就是在这里丢失的！
第一次修改：
return Request(request.url, callback=spider.parse_detail, meta=response.meta, dont_filter=True)
由于我debug到redirect中间件中，response.url和request.url是一样的，所以我认为meta=response.meta和request.meta都是一样的效果，这是错误的，这样会报如下错误：
"Response.meta not available, this response is not tied to any request"
意思是这个response响应没有绑定给任何request，通过源码发现，response绑定给request是在引擎中发生的：
source:scrapy.core.engine.py line230~line241 in scrapy version 1.5.0
def _download(self, request, spider):
slot = self.slot
slot.add_request(request)
def _on_success(response):
assert isinstance(response, (Response, Request))
if isinstance(response, Response):
response.request = request # tie request to response received
logkws = self.logformatter.crawled(request, response, spider)
logger.log(*logformatter_adapter(logkws), extra={'spider': spider})
self.signals.send_catch_log(signal=signals.response_received, \
response=response, request=request, spider=spider)
return response
从请求到spider过程是这样的：
1)request --> 2)downloadmiddleware --> 3)downloader --> 4)downloadmiddleware --> 5)engine --> 6)spidermiddleware --> 7)spider
而当前在4）处，将response绑定给request的操作还未发生，自然就会报错了（ps:spider中使用response.meta是因为在位置7，所以可以拿到）
第二次修改：
return Request(request.url, callback=spider.parse_detail, meta=request.meta, dont_filter=True)
结果很顺利拿到meta信息。
这次也带给我一个教训，程序出现问题，首先从自己身上找问题，而不是找项目问题，scrapy还是很强大的