Basic use of crawler series (seven) requests

I. Introduction to Requests

Requests is a powerful, easy-to-use HTTP request library that can be installed using the PIP install requests command

We’ll cover common methods for Requests below, as detailed in the official documentation

For requests

1. Get method

This method is used to send a request to the target url and receive the response

This method returns a Response object, whose commonly used properties and methods are listed as follows:

• Response. url: Returns the URL of the requesting site
• Response. status_code: Returns the status code of the response
• Response. encoding: Indicates the encoding of the response
• Response. cookies: Returns the Cookie information of the response
• Response. headers: Returns the response header
• Response. content: Returns a response body of bytes
• Response. text: returns the response body of type STR, equivalent toresponse.content.decode('utf-8')
• Response.json () : Returns the body of a dict response, equivalent tojson.loads(response.text)

>>> import requests
>>> response = requests.get('http://www.httpbin.org/get')
>>> type(response)
# <class 'requests.models.Response'>
>>> print(response.url) Return the URL of the requested site
# http://www.httpbin.org/get
>>> print(response.status_code) # return the status code of the response
# 200
>>> print(response.encoding) # return the encoding of the response
# None
>>> print(response.cookies) Return Cookie information for the response
# <RequestsCookieJar[]>
>>> print(response.headers) Return the response header
# {'Connection': 'keep-alive', 'Server': 'Gunicorn /19.9.0', 'Date': 'Sat, 18 Aug 2018 02:00:23 GMT',' Content-type ': 'application/json', 'Content-Length': '275', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Via the' : 'vegur' 1.1}
>>> type(response.content) Return the body of the response as bytes
# <class 'bytes'>
>>> type(response.text) # return the response body of type STR
# <class 'str'>
>>> type(response.json()) Return the body of a dict response
# <class 'dict'>
Copy the code

The parameters of this method are described as follows:

• Url: Specifies the request URL. This parameter is mandatory

• Params: dictionary type that specifies the request parameters, often used when sending GET requests

  >>> import requests
  >>> url = 'http://www.httpbin.org/get'
  >>> params = {
      'key1':'value1'.'key2':'value2'
  }
  >>> response = requests.get(url=url,params=params)
  >>> print(response.text)
  # {
  # "args": {# request parameters we set
  # "key1": "value1",
  # "key2": "value2"
  #}.
  # "headers": {
  # "Accept": "*/*",
  # "Accept-Encoding": "gzip, deflate",
  # "Connection": "close",
  # "Host": "www.httpbin.org",
  # "the user-agent" : "python - requests / 2.19.1." "
  #}.
  # "origin" : "110.64.88.141",
  # "url": "http://www.httpbin.org/get?key1=value1&key2=value2"
  #}
  Copy the code

• Data: Dictionary type, specifying form information, often used when sending POST requests

  Note: Instead of using the POST method, simply replace GET with POST

  >>> import requests
  >>> url = 'http://www.httpbin.org/post'
  >>> data = {
      'key1':'value1'.'key2':'value2'
  }
  >>> response = requests.post(url=url,data=data)
  >>> print(response.text)
  # {
  # "args": {},
  # "data": "",
  # "files": {},
  # "form": {# We set the form data
  # 'key1': 'value1',
  # 'key2': 'value2'
  #}.
  # "headers": {
  # "Accept": "*/*",
  # "Accept-Encoding": "gzip, deflate",
  # "Connection": "close",
  # "Content-Length": "17",
  # "Content-Type": "application/x-www-form-urlencoded",
  # "Host": "www.httpbin.org",
  # "the user-agent" : "python - requests / 2.19.1." "
  #}.
  # "json": null,
  # "origin" : "116.16.107.178",
  # "url": "http://www.httpbin.org/post"
  #}
  Copy the code

• Headers: dictionary type, specifying the request header

  >>> import requests
  >>> url = 'http://www.httpbin.org/headers'
  >>> headers = {
      'USER-AGENT':'the Mozilla / 5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
  }
  >>> response = requests.get(url=url,headers=headers)
  >>> print(response.text)
  # {
  # "headers": {
  # "Accept": "*/*",
  # "Accept-Encoding": "gzip, deflate",
  # "Connection": "close",
  # "Host": "www.httpbin.org",
  # "user-agent ": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" # We set request header
  #}
  #}
  Copy the code

• Proxies: Dictionary type, specify proxies to use

  >>> import requests
  >>> url = 'http://www.httpbin.org/ip'
  >>> proxies = {
      'http':'182.88.178.128:8123'.'http':'61.135.217.7:80'
  }
  >>> response = requests.get(url=url,proxies=proxies)
  >>> print(response.text)
  # {
  # "origin" : "182.88.178.128"
  #}
  Copy the code

• Cookies: dictionary type, specifying cookies

  >>> import requests
  >>> url = 'http://www.httpbin.org/cookies'
  >>> cookies = {
      'name1':'value1'.'name2':'value2'
  }
  >>> response = requests.get(url=url,cookies=cookies)
  >>> print(response.text)
  # {
  # "cookies": {
  # "name1": "value1",
  # "name2": "value2"
  #}
  #}
  Copy the code

• Auth: indicates the tuple type, which specifies the login account and password

  >>> import requests
  >>> url = 'http://www.httpbin.org/basic-auth/user/password'
  >>> auth = ('user'.'password')
  >>> response = requests.get(url=url,auth=auth)
  >>> print(response.text)
  # {
  # "authenticated": true,
  # "user": "user"
  #}
  Copy the code

• Verify: A Boolean type that specifies whether certificate verification is required when a website is requested. The default value is True, indicating that certificate verification is required

  If you do not want certificate validation, set it to False

  >>> import requests
  >>> response = requests.get(url='https://www.httpbin.org/',verify=False)
  Copy the code

  But in this case, you will typically get a Warning because Python expects us to use certificate authentication

  If you do not want to see Warning information, you can use the following command to remove it

  >>> requests.packages.urllib3.disable_warnings()
  Copy the code

• Timeout: Specifies the timeout period. If no response is received after the specified time, an exception will be thrown

2, exceptions module

>>> import requests
>>> try:
		response = requests.get('http://www.httpbin.org/get', timeout=0.1)
except requests.exceptions.RequestException as e:
		if isinstance(e,requests.exceptions.Timeout):
			print("Time out")
        
# Time out
Copy the code