Herpetology is a good place to live

Python3 Web crawler Development Combat PDF HD download address:

Extraction code: 1028

Content Introduction

This book describes how to develop a web crawler using Python 3. It first introduces the environment configuration and the basics, then discusses URllib, Requests, regular expressions, Beautiful Soup, XPath, PyQuery, data storage, Ajax data crawl, and more. Then through a number of cases introduced how to achieve data crawling in different scenarios, after the introduction of PySpider framework, Scrapy framework and distributed crawler.

This book is suitable for Python programmers.

• Catalogue
• Chapter 1 Development Environment Configuration 1
• 1.1Python 3 Installation 1
• 1.1.1 Installation in Windows 1
• 1.1.2 Installation in Linux 6
• 1.1.3 Installation on a Mac 8
• 1.2 Installation of the Request library 10
• Installation of 1.2.1 Requests 10
• 1.2.2Selenium Installation 11
• 1.2.3ChromeDriver Installation 12
• 1.2.4GeckoDriver Installation 15
• 1.2.5PhantomJS Installation 17
• 1.2.6 AiOHTTP Installation 18
• 1.3 Installation of parsing libraries 19
• 1.3.1 LXML Installation 19
• 1.3.2 Installation of Beautiful Soup 21
• 1.3.3 PyQuery Installation 22
• 1.3.4 Tesserocr Installation 22
• 1.4 Database Installation 26
• 1.4.1 installing MySQL 27
• 1.4.2MongoDB Installation 29
• 1.4.3Redis Installation 36
• 1.5 Repository Installation 39
• 1.5.1PyMySQL Installation 39
• 1.5.2PyMongo Installation 39
• 1.5.3 Redis-py installation 40
• 1.5.4 Installing RedisDump 40
• 1.6 Installation of the Web Library 41
• 1.6.1Flask installation 41
• 1.6.2Tornado installation 42
• 1.7App Crawl Related library installation 43
• 1.7.1Charles Installation 44
• 1.7.2 Installing mitmProxy 50
• 1.7.3Appium Installation 55
• 1.8 Installation of crawler frame 59
• 1.8.1 PySpider Installation 59
• Installation of 1.8.2Scrapy 61
• 1.8.3Scrapy-Splash installation 65
• 1.8.4Scrapy-Redis installation 66
• 1.9 Installation of deploy-related libraries 67
• 1.9.1Docker Installation 67
• Installation of Scrapyd 71
• 1.9.3 Installation of Scrapyd-Client
• Install 1.9.4Scrapyd API 75
• Install 1.9.5Scrapyrt 75
• 1.9.6Gerapy installation 76
• Chapter 2 Foundations of reptiles 77
• 2.1HTTP Fundamentals 77
• 2.1.1 URI and URL77
• 2.1.2 Hypertext 78
• 2.1.3 HTTP and HTTPS78
• 2.1.4HTTP Request Process 80
• 2.1.5 request 82
• 2.1.6 response 84
• 2.2 Web Basics 87
• 2.2.1 Composition of web pages 87
• 2.2.2 Structure of web page 88
• 2.2.3 Node tree and Relationships between Nodes 90
• 2.2.4 Selector 91
• 2.3 Basic Principles of crawlers 93
• 2.3.1 Overview of crawlers 93
• 2.3.2 What data can be captured 94
• 2.3.3JavaScript Rendering page 94
• 2.4 Conversations and Cookies95
• 2.4.1 Static and Dynamic Web pages 95
• 2.4.2 Stateless HTTP96
• 2.4.3 Common Misunderstandings 98
• 2.5 Basic Principles of Proxy 99
• 2.5.1 Basic Principles 99
• 2.5.2 Functions of Agents 99
• 2.5.3 Crawler agent 100
• 2.5.4 Agent Classification 100
• 2.5.5 Common Proxy Settings 101
• Chapter 3 Use of the Basic Library
• 3.1　使用urllib　　102
• 3.1.1 Sending a Request 102
• 3.1.2 Handling Exceptions 112
• 3.1.3 Parsing Link 114
• 3.1.4 Analyzing the Robots protocol 119
• 3.2　使用requests　　122
• 3.2.1 Basic Usage 122
• 3.2.2 Advanced Usage 130
• 3.3 Regular Expression 139
• 3.4 Cat eye movie ranking 150
• Chapter 4 Parsing the use of libraries
• 4.1　使用XPath　　158
• 4.2 Using Beautiful Soup168
• 4.3　使用pyquery　　184
• Chapter 5 Data Storage 197
• 5.1 File Storage 197
• 5.1.1TXT Storage 197
• 5.1.2JSON File Storage 199
• 5.1.3CSV File Storage 203
• 5.2 Relational database Storage 207
• 5.2.1MySQL Storage 207
• 5.3 Non-relational Database Storage 213
• 5.3.1 mongo store 214
• 5.3.2 Redis store 221
• Chapter 6 Ajax Data crawl 232
• 6.1 What is Ajax232
• 6.2Ajax analysis method 234
• 6.3Ajax result extraction 238
• 6.4 Analysis of Ajax to climb today’s headlines street photos 242
• Chapter 7 Dynamic Rendering page crawl 249
• 7.1 Use of Selenium 249
• 7.2 Use of Splash 262
• 7.3 Configuring Splash Load Balancing 286
• 7.4 Use Selenium to climb Taobao commodity 289
• Chapter 8 Identification of verification codes
• 8.1 Recognition of graphic verification code 298
• 8.2 Recognition of polar sliding verification code 301
• 8.3 Verification Code Identification 311
• 8.4 Recognition of Micro-blog palace verification code 318
• Chapter 9 Use of Agents 326
• 9.1 Proxy Settings 326
• 9.2 Agent Pool Maintenance 333
• 9.3 Use of paid agents 347
• 9.4ADSL Dial agent 351
• 9.5 Use an agent to access article 364 of wechat public account
• Chapter 10 Simulated login 379
• 10.1 Mock Login and climb to GitHub379
• 10.2 Setting up the Cookies pool 385
• Chapter 11 App crawl 398
• 11.1 Use of Charles 398
• 11.2 Use of MITmProxy 405
• 11.3 MITmdump climbs “Get” App ebook
• Information on 417
• 11.4 Basic use of Appium
• 11.5Appium climbs wechat moments 433
• 11.6Appium+ MitMDump climb jingdong commodity 437
• Chapter 12 uses the PySpider framework 443
• 12.1 PySpider Framework 443
• 12.2 Basic Use of pySpider 445
• 12.3 Pyspider usage details 459
• Chapter 13 Use of Scrapy frames
• Scrapy Framework468
• 13.2 Scrapy introduction of 470
• 13.3 The use of Selector 480
• 13.4 The use of Spider
• 13.5 Usage of Downloader Middleware
• 13.6 Usage of Spider Middleware
• 13.7 Usage of Item Pipeline
• 13.8 Scrapy Selenium506 docking
• 13.9 Scrapy Splash511 docking
• Scrapy Universal crawler 516
• 13.11Scrapyrt use 533
• 13.12 Scrapy Docker536 docking
• Scrapy Scrapy Scrapy
• Chapter 14 Distributed crawler 555
• 14.1 Principles of Distributed crawler 555
• Scrapy-redis scrapy-redis
• Scrapy distributed implementation 564
• 14.4Bloom Filter Interconnection 569
• Chapter 15 Deployment of distributed crawlers
• 15.1Scrapyd
• 15.2 Use of Scrapyd-client
• 15.3 Scrapyd Docker583 docking
• Scrapyd Scrapyd
• 15.5Gerapy Distributed Management 590