First, by the time you read this article, we assume that you are familiar with Python’s basic syntax and how to install Python third-party libraries. On this basis, we can continue to read.

Web page: web page is the basic element of the website, is the platform bearing various web applications.

Example: Baidu home page

URL: On the WWW, each information resource has a uniform and unique address on the web. Example: Baidu home page www.baidu.com/

Web source code: The HTML file content of a web page.

Example: Google Browser open baidu home page, right click to view the source code

Debug mode: Google Chrome’s own developer tools.

Example: Baidu home page

PS: There isn’t much detail here about developer tools.

1. Open a specific web page
2. Look at the source code of the web page and find (CTRL+F) whether the data you are looking for is in the web page.
   
   2.1. If yes, open developer mode and click Network to refresh. At this point you will find that the data you need is returned from the first url. The following
   
   
   
   Finally, code was written to crawl the web page and parse it using xpath
   
   2.2. If not, open developer mode and click Network to refresh. At this point you will notice that the data will not be returned in the first site as in 2.1. In this case, the data is hidden in other JS files. A little front-end development foundation is needed here. We need to find out which links the data resides in, and this is often done manually and empirically, with XHR data filtered first. As shown in figure:
   
   
   
   Through the above case, we screened the word “Baidu” on baidu’s home page as shown in the following figure:
   
   
   
   Finally, write code to crawl the web page and parse it using JSON (in most cases)

Demo2: Climbs the home page information of a specified bilibili user

Install the requests library if the requests library fails
import requests
Note: in developer tools this time is to find the corresponding URL of the data, not the first URL
url = "https://api.bilibili.com/x/space/arc/search"
# Imitate the browser headers
headers = {
    "user-agent": "Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
}
# required parameters
params = {
    "mid": 387694560."pn": 1."ps": 25."index": 1."jsonp": "jsonp"
}
Call the get method, pass in the parameter, and return the result set
resp = requests.get(url,headers=headers,params=params)
Convert the result to JS format
js = resp.json()
Get the data set we need in JS
infos = js['data'] ['list'] ['vlist']
The following code is traversal data
bli = []
for info in infos:
    li = []
    author = info['author']
    bvid = info['bvid']
    pic = info['pic']
    title = info['title']

    li.append(author)
    li.append(bvid)
    li.append(pic)
    li.append(title)
    bli.append(li)
Output complete data
for ll in bli:
    print(ll)
Copy the code

The running results are as follows:

The URL, HEADERS, and xpath matches in the above code are copied directly from the developer tools. Note: To copy the full path of xpath, first select the element you want from the developer tools arrow tool, right-click the element in Elements, and copy it. Note: Further adjustments will be made to the copied full xpath path

1. To get the text of the label, add it after the path/text()
2. If you want to get the tag value for a tag of the element, add to the path/@ Attribute name
3. If you want to get multiple copies of the same data (e.g., rank 1,2… You can concatenate strings in xpath and loop through them.
   
   
   
   

1. The two methods listed above are only the most common encountered when using Python crawlers, and not all cases are appropriate
2. These two cases are only applicable in general, but real business often involves handling cookies, handling anti-theft, multi-threading, multi-processing, and so on. However, since this article is for entry level only, this section is not included in this article.
3. Most of the requests we see on the web are GET and POST, so we should pay attention to the method we use when we climb the page, and check whether the get method needs params parameter and the POST method needs form parameter.

Github.com/zhizhangxue…