爬虫代理方法的使用

时间:2021-08-03 17:00:06

1.urllib模块

设置代理的demo代码如下:

from urllib.error import URLError
from urllib import request

user_agent = r'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87'
# Keep-Alive功能使客户端到服务器端的连接持续有效
headers = {'User-Agent': user_agent}
proxy = '127.0.0.1:9734'
proxy_handler = request.ProxyHandler({
    'http': 'http://' + proxy,
    'https': 'http://' + proxy
})
opener = request.build_opener(proxy_handler)
try:
    req = request.Request('http://httpbin.org/get',headers=headers)
    resp = opener.open(req)
    print(resp.read().decode('utf-8'))
except Exception as e:
    print(e)

运行结果如下图所示:

{

  "args": {}, 

  "headers": {

    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 

    "Accept-Encoding": "gzip, deflate", 

    "Accept-Language": "zh-CN,zh;q=0.9", 

    "Connection": "close", 

    "Host": "httpbin.org", 

    "Upgrade-Insecure-Requests": "1", 

    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

  }, 

  "origin": "111.19.38.245", 

  "url": "http://httpbin.org/get"

}

 

在上面的代码中,先借助request的ProxyHandler方法设置代理,参数是字典类型,键的名称是协议类型,值是代理。需要注意的是代理值的前面要加上协议(http或者https)。当请求时http时,调用http代理,是HTTPS时调用https代理。

创建完ProxyHandler代理后,就可以用request.build_opener方法传入该对象创建一个opener,这一步完成后代理就创建好了。之后就可以访问我们需要访问的链接。

 
2.代理认证
如果遇到需要认证的代理,我们可以使用如下方法设置代理。
proxy = 'username:passwd@127.0.0.1:9734'  # 代理认证

proxy_handler = request.ProxyHandler({

    'http': 'http://' + proxy,

    'https': 'http://' + proxy

})
3.代理是socks5类型
代理是socks5类型,使用如下类型。
from urllib.error import URLError
from urllib import request
import socks,socket

user_agent = r'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87'
# Keep-Alive功能使客户端到服务器端的连接持续有效
headers = {'User-Agent': user_agent}
socks.set_default_proxy(socks.SOCKS5,'127.0.0.1','9742')
socket.socket = socks.socksocket
try:
    req = request.Request('http://httpbin.org/get',headers=headers)
    resp = request.urlopen(req)
    print(resp.read().decode('utf-8'))
except Exception as e:
    print(e)

4.requests方法设置代理

对于requests方法来说,设置代理的方法比较简单,只需要将代理传入proxies参数就可以了。其他的socks5和代理认证的方法和上面一样。代码如下:

import requests

user_agent = r'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87'
# Keep-Alive功能使客户端到服务器端的连接持续有效
headers = {'User-Agent': user_agent}
proxy = '127.0.0.1:9734'
proxy_handler = {
    'http': 'http://' + proxy,
    'https': 'http://' + proxy
}
try:
    req = requests.get('http://httpbin.org/get',proxies = proxy_handler,headers=headers)
    print(req.text)
except Exception as e:
    print(e)