preface
Generally, when developing a crawler, the most upset is not how complex its reverse crawl is, but its parameters are many, to copy into one by one.
Therefore, I wrote a small tool to convert the parameters of header, data and cookie into Python code.
Greatly improve the development efficiency, in fact, the code has been open source before, but it is not very convenient to use, so recently put it online, much easier to use.
The tools or techniques involved
- Python 3.7
- Fiddler v5.0
- Charles v4.2.8
Small tools
Online conversion tools: alisen39.com/
Convert webForms data to dict format
Tool address: alisen39.com/FiddlerData
In Fiddler, webForms data is the argument to the POST request
You can just copy it and convert it to dict format as follows:
Then convert the results directly copy to your crawler code can be used!
Convert Header data to dict format
Tool address: alisen39.com/FiddlerHead…
Header in Fiddler, in the Raw option:
Copy it and paste it to the left:
Convert the data for Cookies to dict format
Tool address: alisen39.com/FiddlerCook…
Cookies found in Fiddler:
Copy it and paste it to the left:
Big tools! Convert raw data into Reqeusts code
One of the most powerful features! Converting the contents of the package directly into Python code really saves 90% of your time.
Click the head recommendation!!
Tool address: alisen39.com/HttpRaw
Copy captured Raw information from Fiddler or Charles.
Copy the code on the right into your IDE
That’s it.