scrapy过滤重复数据和增量爬取

时间:2022-09-24 19:38:35

原文链接

前言

这篇笔记基于上上篇笔记的---《scrapy电影天堂实战(二)创建爬虫项目》,而这篇又涉及redis,所以又先熟悉了下redis,记录了下《redis基础笔记》,这篇为了节省篇幅所以只添加改动部分代码。

个人实现思路

  • 过滤重复数据

在pipeline写个redispipeline,要爬的内容hash后的键movie_hash通过pipeline时与从redis获取的movie_hash(set类型)比对,如果在redis里面则在pipeline里raise DropItem将这个item去掉,在通过pipeline的mysqlpipeline存入数据库时就不会有重复数据了。

  • 增量爬取

虽然没有重复数据了,但是不能增量爬取,一旦停止了爬虫,又会从头爬取,效率很低。想到在downloader middleware中增加对request和response的url进行处理将它放到redis,然后比对,有在redis里面就raise IgnoreRequest忽略掉这个请求,测试也的确忽略了,但其实还是从头开始爬取,只不过忽略了这个请求,效率并没有提升多少,好像并不是增量爬取。最后默默删掉自己的代码。。。

求助

折腾了许久,熬不住了,还是求助吧。其实之前google就搜到了一个scrapy-redis的开源项目就实现了这些功能,但自己试了下没有实现效果(我自己的问题),所以想找本书系统看看scrapy,然后就找了下python scrapy相关的pdf(穷逼买不起书)。找到了一本也引用了scrapy-redis这个项目的书,什么书就不说了,尊重别人的劳动,土豪另当别论。按照书里的配置了下,过滤重复数据是ok了,但增量爬取死活不行,一旦停止、重新爬取,它只会将redis里面的requests爬完就完了,之后再重启爬虫都不会有新的requests生成了。又折腾了许久,在某个群里有个大佬回答了,将爬取page的链接的yield那设置dont_filter = True,自己设置测试了下,的确实现了增量爬取。再次感谢那位大佬的帮助。

具体实现

创建个redis容器

具体查看《redis基础笔记》的使用redis镜像这里的方法,这里不再赘述。

引用scrapy-redis

修改settings.py添加配置,完整配置查看:https://github.com/rmax/scrapy-redis#usage

###### scrapy-redis settings start ######
# https://github.com/rmax/scrapy-redis
# Enables scheduling storing requests queue in redis.
SCHEDULER = "scrapy_redis.scheduler.Scheduler" # Ensure all spiders share same duplicates filter through redis.
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" # Specify the full Redis URL for connecting (optional).
# If set, this takes precedence over the REDIS_HOST and REDIS_PORT settings.
REDIS_URL = 'redis://:123123@192.168.229.128:8889' # Don't cleanup redis queues, allows to pause/resume crawls.
SCHEDULER_PERSIST = True ###### scrapy-redis settings end ######

修改spider

我一开始引用了scrapy-redis不成功的原因是我将这个参数设置成了dont_filter = False,还有就是爬取下一页和详情页的顺序反了,应该先爬页链接在爬详情页。

    def parse(self, response):
item = MovieHeavenBarItem()
domain = "https://www.dytt8.net" # 爬取下一页
last_page_num = response.xpath('//select[@name="sldd"]//option[last()]/text()').extract()[0]
last_page_url = 'list_23_' + last_page_num + '.html'
next_page_url = response.xpath('//div[@class="x"]//a[last() - 1]/@href').extract()[0]
next_page_num = next_page_url.split('_')[-1].split('.')[0]
if next_page_url != last_page_url:
url = 'https://www.dytt8.net/html/gndy/dyzz/' + next_page_url
logging.log(logging.INFO, f'***************** crawling page {next_page_num} ***************** ')
yield Request(url=url, callback=self.parse, meta={'item': item}, dont_filter = True) # 爬取详情页
urls = response.xpath('//b/a/@href').extract() # list type
#print('urls', urls)
for url in urls:
url = domain + url
yield Request(url=url, callback=self.parse_single_page, meta={'item': item}, dont_filter = False)

见证奇迹的时刻

日志输出

2019-07-24 09:45:31 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force 从中可以看到我按了两次ctrl+c强行停止了爬虫,在这之前是在爬取这个链接的2019-07-24 09:45:30 [root] INFO: movie_link: https://www.dytt8.net/html/gndy/dyzz/20180826/57328.html ,然后我重新启动爬虫看到接着爬取这个链接了2019-07-24 09:46:03 [root] INFO: crawling url: https://www.dytt8.net/html/gndy/dyzz/20180718/57146.html

忽略部分...
2019-07-24 09:45:30 [root] INFO: crawling url: https://www.dytt8.net/html/gndy/dyzz/20180826/57328.html
2019-07-24 09:45:30 [root] INFO: **************** movie detail log ****************
2019-07-24 09:45:30 [root] INFO: movie_link: https://www.dytt8.net/html/gndy/dyzz/20180826/57328.html
2019-07-24 09:45:30 [root] INFO: movie_name: 金蝉脱壳2/金蝉脱壳2:冥府
2019-07-24 09:45:30 [root] INFO: movie_publish_date: 2018-06-13(菲律宾)/2018-06-29(中国)/2018-06-29(美国)
2019-07-24 09:45:30 [root] INFO: movie_score: 4.0/10 from 1,180 users
2019-07-24 09:45:30 [root] INFO: movie_directors: 史蒂芬·C·米勒 Steven C. Miller
2019-07-24 09:45:30 [root] INFO: ***************** commit to mysql *****************
2019-07-24 09:45:31 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2019-07-24 09:45:31 [scrapy.core.engine] INFO: Closing spider (shutdown)
2019-07-24 09:45:58 [scrapy.extensions.telnet] INFO: Telnet Password: 4e5dbb60f52f81fe
2019-07-24 09:45:58 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2019-07-24 09:45:58 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'movie_heaven_bar.middlewares.MovieHeavenBarDownloaderMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-07-24 09:45:58 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-07-24 09:45:58 [scrapy.middleware] INFO: Enabled item pipelines:
['movie_heaven_bar.pipelines.MovieHeavenBarPipeline']
2019-07-24 09:45:58 [scrapy.core.engine] INFO: Spider opened
2019-07-24 09:45:58 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-07-24 09:45:58 [newest_movie] INFO: Spider opened: newest_movie
2019-07-24 09:45:58 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-07-24 09:46:03 [root] INFO: crawling url: https://www.dytt8.net/html/gndy/dyzz/20180718/57146.html
2019-07-24 09:46:03 [root] INFO: **************** movie detail log ****************
2019-07-24 09:46:03 [root] INFO: movie_link: https://www.dytt8.net/html/gndy/dyzz/20180718/57146.html
2019-07-24 09:46:03 [root] INFO: movie_name: Operation Red Sea
2019-07-24 09:46:03 [root] INFO: movie_publish_date: 2018-02-16(中国)
2019-07-24 09:46:03 [root] INFO: movie_score: 8.3/10 from 433,101 users
2019-07-24 09:46:03 [root] INFO: movie_directors: 林超贤 Dante Lam
2019-07-24 09:46:03 [root] INFO: movie_actors: 张译 Yi Zhang , 黄景瑜 Jingyu Huang , 海清 Hai-Qing , 杜江 Jiang Du , 蒋璐霞 Luxia Jiang , 尹昉 Fang Yin , 王强 Qiang Wang , 郭郁滨 Yubin Guo , 王雨甜 Yutian Wang , 麦亨利 Henry Mai , 张涵>予 Hanyu Zhang , 王彦霖 Yanlin Wang
2019-07-24 09:46:03 [root] INFO: movie_download_link: magnet:?xt=urn:btih:3c1188fdbec2f63ce1e30d2061913fcba15ebb90&dn=%e9%98%b3%e5%85%89%e7%94%b5%e5%bd%b1www.ygdy8.com.%e7%ba%a2%e6%b5%b7%e8%a1%8c%e5%8a%a8.BD.720p.%e5%9b%bd%e8%af%ad%e4%b8%ad%e5%ad%97.mkv&tr=udp%3a%2f%2ftracker.leechers-paradise.org%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2feddie4.nl%3a6969%2fannounce&tr=udp%3a%2f%2fshadowshq.eddie4.nl%3a6969%2fannounce
2019-07-24 09:46:03 [root] INFO: ***************** commit to mysql *****************
忽略部分...

查看redis key的变化

scrapy-redis 会在redis生成两个set,用于存储请求有序集合requests和过滤链接无序集合dupefilter,当请求消费完了,有序集合requests就会被干掉,直到有新请求时才重新生成,如果未消费完就一直追加。而无序集合dupefilter由于设置了SCHEDULER_PERSIST = True就不会被干掉,有新的请求就会追加进来。

root@fa2b076097e9:/data# redis-cli
127.0.0.1:6379> AUTH 123123
OK
127.0.0.1:6379> KEYS *
1) "newest_movie:requests"
2) "newest_movie:dupefilter"
127.0.0.1:6379> TYPE newest_movie:requests
zset
127.0.0.1:6379> TYPE newest_movie:dupefilter
set
127.0.0.1:6379> ZCARD newest_movie:requests
(integer) 47
127.0.0.1:6379> ZCARD newest_movie:requests
(integer) 45
127.0.0.1:6379> ZCARD newest_movie:requests
(integer) 0
忽略部分...
127.0.0.1:6379> KEYS *
1) "newest_movie:dupefilter"
127.0.0.1:6379> SCARD newest_movie:dupefilter
(integer) 775
127.0.0.1:6379> ZRANGE newest_movie:requests 0 -1
1) "\x80\x04\x95\x7f\x01\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03url\x94\x8c8https://www.dytt8.net/html/gndy/dyzz/20180107/56002.html\x94\x8c\bcallback\x94\x8c\x11parse_single_page\x94\x8c\aerrback\x94N\x8c\x06method\x94\x8c\x03GET\x94\x8c\aheaders\x94}\x94C\aReferer\x94]\x94C4https://www.dytt8.net/html/gndy/dyzz/list_23_31.html\x94as\x8c\x04body\x94C\x00\x94\x8c\acookies\x94}\x94\x8c\x04meta\x94}\x94(\x8c\x04item\x94\x8c\x16movie_heaven_bar.items\x94\x8c\x12MovieHeavenBarItem\x94\x93\x94)\x81\x94}\x94\x8c\a_values\x94}\x94sb\x8c\x05depth\x94K\x1fu\x8c\t_encoding\x94\x8c\x05utf-8\x94\x8c\bpriority\x94K\x00\x8c\x0bdont_filter\x94\x89\x8c\x05flags\x94]\x94u."
2) "\x80\x04\x95\x7f\x01\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03url\x94\x8c8https://www.dytt8.net/html/gndy/dyzz/20180108/56021.html\x94\x8c\bcallback\x94\x8c\x11parse_single_page\x94\x8c\aerrback\x94N\x8c\x06method\x94\x8c\x03GET\x94\x8c\aheaders\x94}\x94C\aReferer\x94]\x94C4https://www.dytt8.net/html/gndy/dyzz/list_23_31.html\x94as\x8c\x04body\x94C\x00\x94\x8c\acookies\x94}\x94\x8c\x04meta\x94}\x94(\x8c\x04item\x94\x8c\x16movie_heaven_bar.items\x94\x8c\x12MovieHeavenBarItem\x94\x93\x94)\x81\x94}\x94\x8c\a_values\x94}\x94sb\x8c\x05depth\x94K\x1fu\x8c\t_encoding\x94\x8c\x05utf-8\x94\x8c\bpriority\x94K\x00\x8c\x0bdont_filter\x94\x89\x8c\x05flags\x94]\x94u."
忽略部分...
127.0.0.1:6379> SMEMBERS newest_movie:dupefilter
1) "1bff65c147e71ea6d43b7e4a4ac86fc982375939"
2) "9d99491255ee83dd4ffb72c3c59c17c938dbe08f"
忽略部分...

检查数据

将数据库数据导出到excel发现500条记录还是有4条重复值,这可能是我多次测试强行停止导致,所以也算是基本实现了过滤重复数据,增量爬取数据的功能了。

scrapy过滤重复数据和增量爬取

结语

这个项目基本功能是实现了,但还是有写细节要处理下,而且scrapy-redis的原理还不了解,下一篇就了解下scrapy-redis的原理。