This is the 12th day of my participation in the August More Text Challenge. For details, see:August is more challenging

(1) Scrapy module installation

Scrapy supports Python2.7 and above.

Python packages can be installed globally (also known as system-wide) or in user space.

A Windows. Download the Twisted version file from https://www.lfd.uci.edu/~gohlke/pythonlibs/. 2. On the command line, go to the Twisted directory and run PIP install and Twisted name 3. Run PIP install scrapy

2. Installed under annaconda (official recommendation) 1. Conda conda docs.anaconda.com/anaconda/pa the old version… https://blog.csdn.net/ychgyyn/article/details/82119201 2 installation method. Install scrapy Conda install scrapy

(2) Scrapy framework introduction

What’s a Scrapy?

Scrapy is an application framework designed to extract structural data from web sites. Originally designed for page scraping (more specifically, Web scraping), it can also be applied to retrieving data returned by apis (such as Amazon Associates Web Services) or general Web crawlers. Scrapy is widely used for data mining, monitoring, and automated testing. It uses the Twisted asynchronous network library to handle network communications.

Why are we using this thing?

(3) Operation process

(As long as the framework is mentioned, it is important to focus on its flow/logical order)

Introduction:

1. Get to the point:

(

Spiders

The items project

Engine engine

The scheduler scheduler

Downloader downloader

Item Pipelines

A: That’s a nice piece of middleware

）

Data flow: The diagram above shows the architecture of the Scrapy framework and its components, as well as the data flow occurring within the system (shown by the red arrow). The data flow in Scrapy is controlled by the execution engine, and the process is as follows:

First of all get the initial request from web crawler Put the request in a scheduling module, and then retrieve the next need to crawl request scheduling module to return the next need to crawl to the engine, the engine will request sent to download, download all through the middleware in turn Downloader, once a page to load returns a response contains the page data, And then it goes through all the downloaded middleware in turn. Downloader to receive the response from the engine, and then sent to the crawler parsing, in turn, through all of the crawler middleware crawler processing the response received, and then parse out the item and generate a new request, and sent to the engine Engine will have a good item to send to piping components, will generate good new request sent to the scheduling module, The process repeats until there are no more requests from the scheduler.

Introduction to middleware:

2. Introduction to each component:

The Scrapy Engine controls the flow of data between all the components of the system and fires events when certain operations occur.

The scheduler receives requests from the engine and queues them so that the engine can request them later.

The Downloader takes the Web pages and feeds them to the engine, which feeds them to the spider.

A spider is a custom class written by a user that parses a response, extracts data from it, or other requests to fetch.

The Item Pipeline is responsible for the subsequent processing of the data after it is extracted by the crawler. Typical tasks include cleanup, validation, and persistence (such as storing data in a database)

(3) Simple use

1. Basic operation (simple project command)!

(1) Create project:

(2) Create crawler file

{create a bdSpider class that inherits scrapy.Spider, define the following three properties: name: the name of the Spider, which must be and only start_urls: The initial URL list parse(self, response) method: is called after each initial URL completes. This function does two things: parse the response, encapsulate it as an Item object and return the object to extract a new URL to download, and create a new request and return it

We can also create crawlers by command

The spiders will start to emit bd.py as soon as they emit more spiders. The spiders will start to emit bd.py as soon as they emit more spiders. Select * from ‘scrapy genspider’ where ‘scrapy genspider’ is created; select * from ‘scrapy genspider’ where ‘scrapy genspider’ is created; 1, scrapy genspider bd www.baidu.com 2, scrapy genspider bd www.baidu.com 3, scrapy genspider

# -*- coding: utf-8 -*-
import scrapy


class BdSpider(scrapy.Spider) :          # Inherit from scrapy.Spider
    name = 'bd'                         The # name is unique (not repeated) because when we start the project, we look for crawler files based on this name
    allowed_domains = ['www.baidu.com'] # Allow domain names without this restriction!
    start_urls = ['http://www.baidu.com/']  # First request (must have) otherwise start can not start, how to let the whole framework run!

    def parse(self, response) :           The parse function must not be renamed to receive data downloaded by the downloader
        print("* * * * * * *")		# For more intuitive observation of whether the framework is working properly!
        print("* * * * * * *")
        print("* * * * * * *")
        print("* * * * * * *")
        print("* * * * * * *")
        print("* * * * * * *")
        print(response)         # the response object
        # Two methods to obtain data:
        print(response.body.decode())   The byte code format is obtained
        # print(response.text)
Copy the code

Note: The engine finally sends the data to the spider module to the parse parameter response:

(3) Run the crawler file

But! After we run the crawler file, we find that the print function used for testing is not displayed. After checking the terminal output data, we know that Scrapy framework follows the robots protocol by default, so we must not get the data!!

How to solve this problem?

# Obey robots.txt rules
ROBOTSTXT_OBEY = True
Copy the code

cdRun the following command on spiders: Scrapy Runspider SpidersCopy the code

1Create a py file named main.py or manage.py in the project folder:Copy the code

2From scrapy.cmdline import execute import sys import OSSelect * from 'scrapy' where 'scrapy', 'crawl' and 'bd' are executed. (You don't have to!)
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
execute(["scrapy"."crawl"."bd"])
Copy the code

3Now, we can run the py file directly and see that it will run the same scrapy framework as the previous two methods. Also, you can debug this scrapy framework by debugging the py file.Copy the code

🔆 In The End!

This blogger will continue to update the basic crawler column and actual crawler column, carefully read this article friends, you can like the collection and comment out of your reading feeling. And can follow this blogger, in the days to read more reptilian!

If there are any mistakes or inappropriate words, please point them out in the comment section. Thank you! If reprint this article, please contact me to obtain my consent, and annotate the source and the name of this blogger, thank you!