Go Distributed Reptiles
Trust that this project will be of great help to anyone learning about distributed systems or crawlers. Give this distributed crawler a try! The idea of the project comes from the github Apple-Han video demo program
Technology stack
Go, Protobuf, Consul, Docker, Elasticsearch, BloomFilter
It has to be done
- Git clone github.com/apple-han/i…
- cd island
- Global search 192.168.31.231 replace the IP address of your host (because there are JSON files, it is difficult to do global configuration)
Docker mode deployment
-
cd crawler_distributed/persist
- make build
- make docker
-
cd crawler_distributed/worker
- make build
- make docker
-
cd crawler_distributed
- make build
- make docker
-
cd crawler/frontend
- make build
- make docker
-
cd island
-
docker-compose up -d
-
http://192.168.31.231:8888/search?q= public IP (own)
Tips:
- Because the system is a distributed, so the overall down is a little difficult
- I hope you guys, take a good look at it, it’ll help you a lot