Request.Request method
The previous section introduced the use of request.urlopen(), which is simply used without setting headers or cookies. The request.request () method further wraps the request.
1, source view parameters
class Request:
def __init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None):
pass
2, use request Baidu
from urllib import request
if __name__ == “__main__”:
headers = {
‘the user-agent’ : ‘Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 OPR/54.0.2952.64’
}
req = request.Request(url=’https://www.baidu.com’, headers=headers)
response = request.urlopen(req)
print(response.read())
Use a proxy server
1, common free proxy address
West stab agent
Secure proxy
Fast acting
89 Free Agency
2. Test site to detect the current IP information
3. Use proxies
from urllib import request
if __name__ == “__main__”:
headers = {
‘the user-agent’ : ‘Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 OPR/54.0.2952.64’
}
Create a proxy
Handler = request. ProxyHandler ({” HTTP “:” 186.103.173.189:58243 “})
opener = request.build_opener(handler)
Wrap the request object
req = request.Request(“http://httpbin.org/get”, headers=headers)
Use the proxy to open the site
response = opener.open(req)
print(response.read().decode(“utf8”))