background

I haven’t written GO for a long time. Recently, I reviewed my knowledge of GO and found that I wrote a distributed program based on ZooKeeper and GO more than a year ago. I pulled it down and ran it for a while.

Overall analysis, this project is very suitable for eyes, realize the function of code quantity is less, and the code readability after refactoring a little better, all the functions are simple, no complicated abstruse principles, such as the election there is no such complex algorithm using raft, increasing to be based on the atoms of zk.

Therefore, this project is not to improve the content of one aspect, but to practice on GO from various aspects and master the basic knowledge of ZooKeeper.

knowledge

Through this project, the following knowledge can be learned:

  • Go Basics
  • Coroutines use
  • Channel using
  • Reflection using
  • Use of the Logrus package
  • Use of the Viper package
  • Basic use of ZK
  • Zk-based leader election algorithm
  • Distributed Node Discovery
  • Distributed node communication
  • Work tasks are assigned in distributed mode
  • Some related problems under distributed
  • Simple Web service implementation
  • other

Functional architecture

Images on Github may be slow to load, please be patient

The overall xConsumer project has very little code, and the process is divided into the following steps:

  1. Cluster initialization (check)
  2. Leader election
  3. Leader Cluster task assignment
  4. Distribute the tasks
  5. Start to work

Functional process

Note: Only the leader node can perform this operation. The leader node, like other follower nodes, must perform tasks.

Application scenarios

The project implements a distributed scenario, which can realize automatic task load, so basically all the scenarios with this feature can be applied, but the functions in some scenarios may not be comprehensive enough.

  • Distributed crawler, a characteristic of the crawler for each URL to do processing, using this framework can be set according to the need of multiple tasks to crawl.
  • Simple service discovery, node access realized through ZK, is a simple service discovery, which can be developed independently to form a slightly better service discovery.
  • Message queue distributed consumer side, the main problem to solve at the beginning of the project is this, according to the use of different topics, the number of different tasks to be allocated for processing.
  • Distributed task processor, if we have a large number of tasks that need to be timed, this framework can be implemented.
  • A temporary task handler allows us to dynamically add tasks so that task processing can be triggered at any time
  • other

Comments & Suggestions

Due to the time and energy of the whole project, the implementation period is short, coupled with the limited personal ability, there must be many shortcomings, some code or implementation is not optimal.

The main purpose of this project is to communicate with more people. If there is no communication and practice when learning something, it is often difficult to really master it. So here is a brick to introduce jade.

The project address

Cloud xconsumer – yard