Address: github.com/derekhe/Pro…

introduce

In Crawler In Action: From Data to Product, I talked about a proxy pool based on ProxyBroker. After a long time of practice, this proxy pool is very convenient and stable to use.

Based on ProxyBroker, added proxy resources in China region. Docker-compose is also introduced, which can quickly and conveniently start the proxy fetching.

usage

docker-compose up
Copy the code

Then open http://localhost:8080/proxy.json brings the proxy list. Each agent is type-validated, and the agent resource grows over time. Each agent is valid for one day. There are about 10,000 effective agents a day.

Many proxy resources cannot be accessed in China. Therefore, it is recommended to deploy the proxy resources on a foreign server. I recommend using DigitalOcean for the server. Many of my servers are in the SFO2 zone and are very stable.