Introduction to the Crawler Python Toolkit (2)

>>>payload = {'key1': 'value1'.'key2': 'value2'}
>>>r = requests.get("http://httpbin.org/get", params=payload)print(r.url)http://httpbin.org/get?key2=value2&key1=value1Copy the code

Example 2:

>>> param = 'httpparams'
>>> r = requests.get("http://httpbin.org/get",params=param)
>>> print r.urlhttp://httpbin.org/get?httpparamsCopy the code

>>> payload = {'key1': 'value1'.'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print(r.text)
{
...
    "form": {
    "key2": "value2"."key1": "value1"},... }Copy the code

Example 2:

>>> payload = (('key1'.'value1'), ('key1'.'value2'))
>>> r = requests.post('http://httpbin.org/post', data=payload)
>>> print(r.text)
{
...
    "form": {
        "key1": [
            "value1"."value2"]},... }Copy the code

r = requests.post(url, json={"key":"value"}})Copy the code

The captured header is content-Type: application/json

r = requests.post(url,headers={"user-agent":"test"})Copy the code

>>> url ='http://example.com/some/cookie/setting/url'
>>> r = requests.get(url)
>>>r.cookies['example_cookie_name']Copy the code

Example 2: Sending cookies to the server

>>> jar = requests.cookies.RequestsCookieJar()
>>> jar.set('tasty_cookie'.'yum', domain='httpbin.org', path='/cookies')
>>> url = ' '
>>> r = requests.get(url, cookies=jar)
>>> r.text
'{"cookies": {"tasty_cookie": "yum"}}'Copy the code

>>> url = 'http://httpbin.org/post'
>>> files = {'file': open('report.xls'.'rb')}
>>> r = requests.post(url, files=files)Copy the code

Example 2: Explicitly set the file name, file type, and request header

>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls'.'rb'),
           'application/vnd.ms-excel', {'Expires': '0'})}
>>> r = requests.post(url, files=files)Copy the code

>>>url = 'http://httpbin.org/digest-auth/auth/user/pass'
>>>requests.get(url, auth=HTTPDigestAuth('user'.'pass'))Copy the code

After packet capture, the HTTP header is as follows:

GET http://httpbin.org/digest-auth/auth/user/pass HTTP / 1.1 Host:httpbin.org Connection: keep alive - Accept - Encoding: Gzip, deflate the Accept: / the user-agent: python - requests / 2.12.4 Authorization: Basic dXNlcjpwYXNzCopy the code

>>> requests.get('http://github.com', timeout=0.001)
Traceback (most recent call last):
    File "", line 1, inrequests.exceptions.Timeout: 
    HTTPConnectionPool(host='github.com', port=80): 
        Request timed out. (timeout=0.001)Copy the code

>>>r = requests.get('http://github.com', allow_redirects=False)
>>> r.status_code 
    301 
>>> r.history
    []Copy the code

>>>proxies = {
    "http": "http://127.0.0.1:8888"."https": "https://127.0.0.1:8888",
}
>>>r=requests.get(url, headers=headers,proxies=proxies)Copy the code

 import requests
from contextlib import closing
def download_image_improve():
    url = (' '
    'image&quality=80&size=b9999_10000'
    '&sec=1504068152047&di=8b53bf6b8e5deb64c8ac726e260091aa&imgtype=0'
    '&src=http%3A%2F%2Fpic.baike.soso.com%2Fp%2F'
    '20140415%2Fbki-20140415104220-671149140.jpg')
    with closing(requests.get(url, stream=True,verify=False)) as response:
    # open an empty PNG file, equivalent to creating an empty TXT file,
    # wb indicates write file
        with open('selenium1.png'.'wb') as file:
        This file is finally written to selen.png
            file.write(data)Copy the code

r=requests.get(url, verify=False)Copy the code

Example 2: Borrow Fiddler’s converted PEM certificate to access Central Asia. Fiddlerroot.zip download address:

nos.netease.com/knowledge/

2b99aacb-e9bf-42f7-8edf-0f8ca0326533? download=FiddlerRoot.zip

Note: The certificate can be in cer or PEM format. You are advised to install a cer certificate first.

You can also install Fiddler yourself, trust the Fiddler certificate,

Export the cer format and use it again

Cer -out specifies the format of the fiddlerroot. pem conversion certificate

headers = {
    "Host": "www.amazon.com"."Connection": "keep-alive"."Cache-Control": "max-age=0"."Upgrade-Insecure-Requests": "1"."User-Agent": ("Mozilla / 5.0 (Windows NT 10.0; Win64; x64) "
                   AppleWebKit/537.36 (KHTML, like Gecko)
                   "Chrome / 68.0.3440.106 Safari / 537.36"),
    "Accept": ("text/html,application/xhtml+xml,"
               "application/xml; Q = 0.9, image/webp image/apng, /; Q = 0.8"),
    "Accept-Encoding": "gzip, deflate, br"."Accept-Language": "zh-CN,zh; Q = 0.9, en. Q = 0.8"
}
print requests.get('https://www.amazon.com/', 
verify=r"FiddlerRoot.pem", headers=headers).contentCopy the code