preface

The text and pictures in this article come from the network, only for learning, communication, do not have any commercial purposes, if you have any questions, please contact us to deal with.

Basic Environment Configuration

• Python 3.6
• pycharm
• requests
• parsel

The related module PIP can be installed

Define your site’s goals

Basic general operation, F12 open developer tools, analyze web pages

The data returned by the web page is a pile of garbled code, and then write code when you can turn the code is good static web pages are very simple, it is possible to directly grab the data you want

The request page

import requests import re url = 'http://www.sccnn.com/shiliangtuku/default({}).html'.format(page) headers = { 'user-agent ': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'} Response = requests. Get (url=url, headers=headers) response.encoding = response.apparent_encodingCopy the code

Analyze web pages and data

import parsel r = re.findall('<a href="(.*?) " target="_blank">', response.text) urls = r[2:] for i in urls: page_url = 'http://www.sccnn.com' + i response_2 = requests.get(url=page_url, headers=headers) response_2.encoding = response_2.apparent_encoding selector = parsel.Selector(response_2.text) title = selector.css('#LeftBox h2::text').get() img_url = selector.css('#LeftBox .PhotoDiv img::attr(src)').get()Copy the code

Save the data

def downlaod(title, url): Path = 'D:\ python\\demo\\ material website \\img\\' + title + '.jpg' response = requests. Get (url=url, Headers =headers) with open(path, mode='wb') as f: f.write(response.content) print(' downloading {}'. Format (title))Copy the code

Implementation effect

Complete project code background