This article is participating in Python Theme Month. See the link to the event for more details

This article has participated in the good article call order activity, click to see: back end, big front end double track submission, 20,000 yuan prize pool for you to challenge!

After reading this article, don’t say you don’t know how to use Feapder. What is a Feapder? How to create a Feapder entry project (actual combat: collect easy car network car data).

The actual part includes the crawler data and storage to the Mysql database, so that we can feel the data from the web page through the framework Feapder collection, directly stored to the database process.

We have already used Scrapy crawler framework to climb data (” B station “as a practical case! Today, try to use Feapder to write Scrapy crawler is a kind of experience, please go to ！！！！！

01. Feapder framework

1. Introduction to the Feapder framework

Feapder is a simple, powerful, fast, lightweight crawler framework for Python. Support lightweight crawler, distributed crawler, batch crawler, crawler integration, and perfect crawler alarm mechanism.

What is the function of each piece of the specific FeAPder project structure? How does it work? Then look down, the following actual combat has a detailed explanation.

2. Feapder installation

Feapder installation is very simple, with the following command to install!

pip install feapder
Copy the code

The following page appears indicating that feAPder has been installed successfully!

With the introduction and environment installation of FeAPder complete, it is time to actually use Fepader to climb the data of The Easy Car network and store it in the mysql database.

02, actual combat,

1. Create a Feapder project

Create a crawler project named chenge_yc_spider with the following command

feapder create -p chenge_yc_spider
Copy the code

With that created, let’s look at the project structure

2. Write a crawler

On the terminal, go to the spiders folder of the project (chenge_yc_spider) and create a target crawler file (target_spider) with the following command

feapder create -s  target_spider
Copy the code

The project structure at the moment is as follows:

Edit the target_spider.py file

Here practical case: collect data of easy car network. I’m just going to go ahead and execute the py file and see if the request is okay.

You can see that the request returned a response of 200, indicating that the request was successful. Next, we began to parse the web page data and set up the crawler framework to automatically collect the next page data.

3. Parse your web page

The results of the web page (data to be collected) are as follows:

By looking at the source code, analyze the data corresponding to the page label

Through the source code of the web page can be analyzed, the car list data is in the class of search-result-list. Each class is search-result-list-item, which represents a piece of data. Under each piece of data, there are attributes corresponding to the car (such as car name, price, etc.).

Here is only a real case to learn the Feapder crawler framework, so here is only to climb the name of the car, the price; These two field properties.

4. Create the Mysql database

The collected data needs to be stored in the database (mysql), so let’s define the database and tables first

Calvin elder brother here to create a database: chenge_yc, and built should be inside tables: t_yc, the table structure as shown above, if not here are not familiar with mysql how to build a database table can refer to Calvin elder brother of the article (actual combat | will teach you how to play the mysql in Python)

To configure the database in the crawler project, open the setting.py file in the root directory

Feapder supports the interconnection of multiple databases. In this case, mysql is used. The configuration is as follows:

Next, under the terminal, go to the items folder in the root directory and run the following command to generate the item for the database table

feapder create -i t_yc
Copy the code

Note that t_yc in the command is for t_yc in the database table

Finally, the t_yc_item.py file is generated:

The name and price are the corresponding database fields.

5. Extract web page fields

The above has obtained the source code of the web page, and also know the label where the data is located, now start to write the code to parse.

Execution result:

You can see that the data has been successfully pulled out, and the next step is to store the data in the database.

6. Store the data to the database

import feapder
Copy the code

Introduce t_yc_item.py and create the object TycItem. Initialize the name and price to the object. Finally, yieId TycItem, which is actually stored directly in the database (because the database table is connected to the item, so it is stored directly in the database).

It’s so convenient, it saves SQL statements, 6666666666

The execution result is as follows:

Check the database:

You can also see that the data is stored directly in the database. That’s it ！！！！！！

03, subtotal

I believe you have completely mastered the crawler framework: Feapder. You not only know what Feapder is, but also learn how to use Feapder.

In addition, the actual part includes the crawler data and storage to the Mysql database, so that you can feel the data from the web page through the framework Feapder collection, directly stored to the database process.

Do try it! Do try it! Do try it!