preface

• Python environment setup and basics
• Overview of crawler principle
• An overview of crawler technology
• Cat eye movie ranking data capture
• Ajax data to climb the cat’s eye box office
• More advanced, proxy, simulated login, APP crawl… .

Python environment setup and basics

Python Environment setup

Anaconda installation

IDE environment setup

1.1 Basic Python technologies

• The data type
• The list of
• Looping statements
• statement
• function

Python based

• | rookie Python3 tutorial tutorial
• Python tutorial | Liao Xuefeng’s official website
• Python tutorial | introductory tutorial
• Python tutorial ™
• Stupid to learn Python | see cloud
• The Python Tutorial | Python 3.6.6 documentation

The Python official documentation is absolutely authoritative and comprehensive, but the documentation itself is in English, so it is not very friendly for most beginners. The following is the Python version translated by Chinese people.

• Python start guide | Python tutorial 3.6.3 documentation some introductory books recommended
• Python Programming from Beginning to Practice
• For those who want to improve their skills, the next series of books and materials should be right up your street:
• The Python Standard Library
• Python CookBook 3
• Smooth Python
• Python Learning Manual (4th Edition)
• Python Core Programming (Version 3)
• Python Language Description of Data Structures
• High Performance Programming in Python

What is a reptile

The crawler principle

• The first column Name: the Name of the request, usually the last part of the URL.
• The second column Status: the Status code of the response, which is displayed as 200, indicating that the response is normal. Using the status code, we can determine whether the request received a normal response after sending it.
• The third column Type: the document Type requested. This is called document, which means what we’re asking for this time is an HTML document, and the content is just some HTML code.
• Column 4 initiator: request source. Used to mark the object or process from which the request originated.
• Column 5 Size: the Size of the file downloaded from the server and the requested resource. If the resource is fetched from the cache, this column displays from Cache.
• Column 6 Time: The total Time from the Time the request was initiated to the Time the response was received.
• 7th column Waterfall: Visual Waterfall flow with network requests. Next, let’s analyze the detailed composition of the request. For example, click the first request, which is the request whose Name is www.baidu.com, as shown in the figure below:

• Request URL indicates the URL of the Request
• Request Method indicates the Request Method
• Status Code is the response Status Code.
• Remote Address Indicates the Address and port number of the Remote server

• HTTP/1.1 Protocol and version number used for the response 200 (status code) OK(Description)
• Location indicates the path of the page that the server requires the client to access
• Server indicates the name of the Web Server on the Server
• Content-encoding Specifies the Encoding type that the server can send
• Content-length Specifies the Length of compressed data sent by the server
• Content-language Indicates the Language type sent by the server
• Content-type indicates the Type and encoding mode sent by the server
• Last-modified Indicates the time when the server Last Modified the resource
• Refresh for the server to ask the client to Refresh after one second and then access the specified page path
• Content-disposition for the server requires the client to open the file as a download
• Transfer-encoding Transfers data to the client in blocks
• Set-cookie is the temporary data sent by the server to the client
• Connection Maintains the Connection between the client and server

• GET(request mode) /newcoder/hello.html(request target resource) HTTP/1.1(request protocol and version)
• Accept indicates the type of resources that can be received by the client
• Accept-language indicates the Language type received by the client
• Connection Maintains the Connection between the client and server
• Host: localhost indicates the destination Host and port number
• Does Referer tell the server where I’m from
• User-agent indicates the version number of the client
• Accept-encoding Specifies the type of compressed data that can be received by the client
• If-modified-since indicates the cache time
• Cookie indicates that the client temporarily stores information about the server
• The Date for the time to serve the client requests And we need to do is to simulate the browser to submit Requests Headers for the server’s response information, and the data, we want to want to understand the reader, please visit HTTP | MDN documentation for more information.

What kind of data can a crawler capture

An overview of crawler technology

First request

Requests the library

Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated.

Import the Requests module
import requests
Make a Get request and return a Response object containing the server's Response to the HTTP request
response = requests.get('https://gitbook.cn/')
Print the response status code
print(response.status_code)
Print the response body of type STR, such as a normal HTML page, using text for further analysis of the text
print(response.text)

Copy the code

Import the Requests module
import requests
Form data to submit
data = {
    'name': 'ruo'.'age'22} :# Post request
response = requests.post("http://httpbin.org/post", data=data)
# Response body content
print(response.text)

Copy the code

# PUT requestRequests. The put (" http://httpbin.org/put ")# the DELETE requestRequests. The delete (" http://httpbin.org/delete ")# HEAD requestRequests. The head (" http://httpbin.org/get ")# the OPTIONS requestRequests. The options (" http://httpbin.org/get ")Copy the code

Import the Requests module
import requests
Make a Get request
response = requests.get("https://www.zhihu.com")
# status code
print(response.status_code)
# Response body content
print(r.text)

Copy the code

Import the Requests module
import requests
Add user-agent to Headers
headers = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
Make a Get request
response = requests.get("https://www.zhihu.com", headers=headers)
# status code
print(response.status_code)
# Response body content
print(response.text)

Copy the code

Extracting information

Regular expression

Import the re module
import re
Text to be matched
h1 = '
Chapter 3.2.1 - Regular Expressions 
Compile the regular string into a regular expression object for reuse in later matches
pat = re.compile('
(.*?) 
', re.S)
# re.search scans the entire string and returns the first successful match
result = re.search(pat, h1)
Group () can enter more than one group number at a time, in which case it will return a tuple containing the values corresponding to those groups.
print(result.group(0))
Group () can enter more than one group number at a time, in which case it will return a tuple containing the values corresponding to those groups.
print(result.group(1))

Copy the code

Xpath

import requests
Import the ETREE module of LXML library
from lxml import etree
headers = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
url = 'http://maoyan.com/board/4'
response = requests.get(url, headers=headers)
html = response.text
Call the HTML class for initialization
html = etree.HTML(html)
# Paste our copy xpath to extract the movie title "Farewell my Concubine"
result_bawangbieji = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd[1]/div/div/div[1]/p[1]/a')
Print the text content contained in the node tag
print(result_bawangbieji[0].text)
# Extract all movie names of this page, that is, select all movie names of the 'DD' tag
result_all = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd/div/div/div[1]/p[1]/a')
Print all extracted movie names
print('All movie names on this page:')
for one in result_all:
    print(one.text)

Copy the code

Store information

TEXT storage

import requests
from lxml import etree
headers = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
url = 'http://maoyan.com/board/4'
response = requests.get(url, headers=headers)
html = response.text
Call the HTML class for initialization
html = etree.HTML(html)
# Paste our copy xpath to extract the movie title "Farewell my Concubine"
result_bawangbieji = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd[1]/div/div/div[1]/p[1]/a')
Print the text content contained in the node tag
print(result_bawangbieji[0].text)
Extract all movie names from this page
result_all = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd/div/div/div[1]/p[1]/a')
Print all extracted movie names
print('All movie names on this page:')
for one in result_all:
    print(one.text)
# Store the movie name of this page in a TEXT file. 'A' means to open a file for append. If a file exists, the file pointer is at the end of the file. That is, the file is in append mode. If the file does not exist, it creates a new file to write to.
with open('film_name.text'.'a') as f:
    for one in result_all:
        f.write(one + '\n')

Copy the code

CSV storage

import requests
from lxml import etree
# Import CSV module
import csv

headers = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
url = 'http://maoyan.com/board/4'

response = requests.get(url, headers=headers)
html = response.text
html = etree.HTML(html)
result_bawangbieji = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd[1]/div/div/div[1]/p[1]/a')
print(result_bawangbieji[0].text)
result_all = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd/div/div/div[1]/p[1]/a')
print('All movie names on this page:')
for one in result_all:
    print(one.text)
# Store the movie name in a CSV file:
with open('film_name.csv'.'a', newline=' ') as f:
    csv_file = csv.writer(f)
    for one in result_all:
        csv_file.writerow([one.text])

Copy the code

MySQL storage

import requests
from lxml import etree
Import the pymysql module
import pymysql

Open a database connection
db = pymysql.connect(host='localhost', user='root', password='root', port=3306, db='spider', use_unicode=True, charset="utf8")
Select * from MySQL where cursor is used to execute SQL statement
cursor = db.cursor()

headers = {
    'User-Agent': 'the Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
url = 'http://maoyan.com/board/4'

response = requests.get(url, headers=headers)
html = response.text
html = etree.HTML(html)
result_bawangbieji = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd[1]/div/div/div[1]/p[1]/a')
print(result_bawangbieji[0].text)
result_all = html.xpath('//*[@id="app"]/div/div/div[1]/dl/dd/div/div/div[1]/p[1]/a')
print('All movie names on this page:')
for one in result_all:
    print(one.text)

    try:
        Insert data statement
        sql = 'INSERT INTO film_infor(film_name) values (%s)'
        cursor.execute(sql, (one.text))
        db.commit()

    except:
        db.rollback()

Copy the code