Background description

A service that uses Prometheus for data acquisition was optimized for metrics use on this project today: Putting metrics scattered around in the same GO file and encapsulating the direct calls to Metrics into a method made it look much cleaner. After I happily packaged the code into a new version and went up on several machines first, When Grafana saw a sharp increase in the number of abnormal span, I panicked. Fortunately, I was very skilled in the normal rollback operation. I directly rolled back all the machines with one key, and everything was fine.

Thinking of the February

  1. This problem was modified when it was first encounteredquantileIt was found after the statistics, at that time it was assumed that there was a conflict between the dependent packages, because of the input beforego get -u allUpdate all packages, recompile after deleting locally downloaded packages, problem resolved
  2. After using the above method, it was found that it did not work. It was speculated that the method call caused the writing to be too slow, and the downstream service had a delay in receiving logs, leading to the loss of a large number of logs. The code of the previous versions was changed to test: the problem was still the same
  3. Search for problematic logs in downstream servicestraceID, find the sametraceID 的 spanDistributed over several machines, normal conditions are the sametraceID 的 spanIt will be sent to the same machine, presumably tokafkaA problem occurred while writing to the log: the client wrotekafkaThere was abalanceProcess, code is their own manual implementation of a, guess is that there is a problem with the packagebalance()And then the question remains
  4. Then look atkafka-goThe source code, in the relevant place to do log printing (this is really tricky), found when set upbalancer == nilBy default, the polling method is calledbalanceContinue to look at the new versionkafka-go(0.4.2)newWriter()There’s a question. I missed itbalancerVariable that causes the configuration to be empty regardless of whether or not the value is assignedWriter{}Initialization, but there is a problem if the configurationbrokersMultiple values are buggy, and it works fine when there is only one, eliminating formatting problems that might occur with assignments
  5. The ultimate solution is to go back to0.3.6Version,go modHere’s the catch: it upgrades itself to the new version, so if you want to stay with the old version, you need to use itreplace