文件名称：protego:纯Python robots.txt解析器，支持现代约定
文件大小：4.04MB
文件格式：ZIP
更新时间：2024-06-13 07:40:31
python robots-txt robots-parser DIGITALCommandLanguage
Protego Protego是纯Python robots.txt解析器，支持现代约定。 安装 要安装Protego，只需使用pip： pip install protego 用法 >>> from protego import Protego >>> robotstxt = """ ... User - agent: * ... Disallow: / ... Allow: / about ... Allow: / account ... Disallow: / account / contact $ ... Disallow: / account /*/ profile ... Crawl - delay: 4 ... Request - rate: 10 / 1m # 10 requests every 1 minute ... ... Sit