go_spider爬虫框架

【文件属性】：
文件名称：go_spider爬虫框架
文件大小：40KB
文件格式：ZIP
更新时间：2021-05-14 11:04:12
go [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to a Individualized crawler easily or can only use the default crawl components.

【文件预览】：
.gitignore
README.md
read.t
extension
----pipeline()
--------contribution_pipeline_extension_here(0B)
----other()
--------contribution_other_tool_here(0B)
----scheduler()
--------contribution_scheduler_extension_here(0B)
----downloader()
--------contribution_downloader_extension_here(0B)
LICENSE
example
----login_profile_page_processor()
--------main.go(4KB)
----sohu_gossip_page_json_processor()
--------main.go(3KB)
----baidu_baike_page_processor()
--------main.go(3KB)
----sina_stock_json_processor()
--------main.go(4KB)
----github_repo_page_processor()
--------main.go(2KB)
.travis.yml
core
----pipeline()
--------pipeline_file.go(978B)
--------pipeline_console.go(677B)
--------pipeline.go(818B)
--------collect_pipeline_page_items.go(689B)
----spider()
--------spider.go(9KB)
----page_processer()
--------page_processer.go(317B)
----common()
--------resource_manage()
--------request()
--------page()
--------com_interfaces()
--------etc_config()
--------page_items()
--------config()
--------mlog()
--------util()
----scheduler()
--------scheduler_queue.go(2KB)
--------scheduler_simple.go(800B)
--------scheduler.go(363B)
--------scheduler_test.go(1KB)
----downloader()
--------downloader.go(505B)
--------downloader_http.go(8KB)
--------downloader_test.go(3KB)