Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

🍖 extension a: verification code processing!

We frequently send requests when logging in to register, and the server pops up picture verification code for verification! But now most websites even if you send a request for the first time, it will pop up a picture verification code to ask you to verify!

🍮 (1) Solution:

  1. Manual input, but only login once, cannot last;
  2. Image recognition engine, which uses optical recognition engine to process the data in the image, is often used for image data extraction, less used for verification code processing;
  3. Play code platform, crawler commonly used to solve the verification code scheme.

🍚 (2) On the picture recognition engine (TESSERact) the specific operation of this processing scheme:

🍏 Step 1: Download and install

Address of the project: https://github.com/tesseract-ocr/tesseract engine installation: (1) MAC environment directly execute the command: Brew install -- with-train-tools Tesseract (2) Tesseract > tesseract > tesseract > Tesseract < path > Tesseract-ocr sudo apt-get install tesseract-OCr sudo apt-get install Tesseract-OCrCopy the code

🍒 Step 2: Use

The image recognition engine Tesseract is used in Python. (The following code does not do any processing to the image, the effect is not good!)

Pre-war preparations: (1PIP install pillow (PIL)2PIP install PytesserAct is used to parse data from imagesCopy the code


from PIL import Image
import pytesseract

im = Image.open('download.png')

result = pytesseract.image_to_string(im)
print(result)

Copy the code

🍑 extension: Dacheng image recognition engine (direct tuning interface, why not ~)

Other OCR platform (1) Microsoft Azure image recognition: https://azure.microsoft.com/zh-cn/services/cognitive-services/computer-vi (2) the youdao ZhiYun character recognition: http://aidemo.youdao.com/ocrdemo (3) ali cloud graphic recognition: https://www.aliyun.com/product/cdi/ (4) tencent OCR character recognition:  https://cloud.tencent.com/product/ocrCopy the code

🍉 (3) Common types of verification codes:

The first is that the URL address and verification code remain unchanged. Second: The URL address remains unchanged, but the verification code changes.

🐮 Regarding the part of verification code, students have always had a question: how does the server of the other party determine that the verification code I enter is the one displayed on my screen? 🐮

💘 is actually implemented through cookies. Correspondingly, in the request page, the request verification code needs to ensure the consistency of cookies, using requests. Session to solve the problem. 💘

🔞 Extension 2: About crawler operations in Chrome

(The use of stealth window, the first time to open the website, will not bring cookies, can observe the page to obtain the situation, including the other server how to set the cookie in the local!)

  1. Js 解析 :

    1. Determine the position of JS. ① Observe the binding JS event of the button (this method is not guaranteed to be found) ② Locate the JS file through initiator ③ locate the JS file through serach search keyword ④ Locate the JS file through the event listener function of element binding 2. In Chrome, check that the consle executes part of the JS code and resolves what part of the JS code represents 3. ① Using jS2py translation tool, Python program reproduction; Pyv8, ExecuteJS and Splash are similar to JS2PY. ③ A pure Python implementation;Copy the code

  2. Ideas:

    After the understanding of JS content and execution order, through Python to complete the JS execution process, get the results; Once you know the JS content and order of execution, use a module like Js2py to execute the JS code and get the results.Copy the code

🔆 In The End!

Start now, stick to it, a little progress a day, in the near future, you will thank you for your efforts!

This blogger will continue to update the basic column of crawler and crawler combat column, carefully read this article friends, you can like the collection and comment on your feelings after reading. And can follow this blogger, read more crawler in the days ahead!

If there are mistakes or inappropriate words can be pointed out in the comment area, thank you! If reprint this article please contact me for my consent, and mark the source and the name of the blogger, thank you!