Background: RECENTLY, I received a big job at work. A micro-service with daily page views of more than 120 million was handed to me. However, there was a problem when the service was released after one function iteration.

Investigation process:

1. After receiving the alarm, I checked the log and found that the container IP of the alarm was all the container IP dropped by the rolling upgrade. I thought that the service might have been shut down without completing the processing of the request. So we have to put the elegant closing arrangement on first. The approximate code is as follows:


server := http.Server{
    Addr:    ": 8081",
    Handler: engine,
}
// Enable the HTTP service
go func (a) {
    err := server.ListenAndServe()
    if err == nil {
        fmt.Println("error")}} ()// Wait to receive service shutdown signal
c := make(chan os.Signal)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
<-c
// Set the maximum waiting time to 5s after receiving the signal
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
// Wait until all HTTP requests are processed before closing the service. The maximum waiting time is not more than 5 seconds
err := server.Shutdown(ctx)
iferr ! =nil {
cancel()
}

Copy the code

2. The local code ran and tried a wave, and found that the elegant shutdown had taken effect. After adding the debug log and releasing it to the test environment, it found that the log did not record the received shutdown signal after each docker container restart. The CMD command in Dockerfile should be exec format, so that the service can receive the shutdown signal when the container is closed. If you look at the code, CMD is shell format, and the local graceful closure works because local runs directly from the command line, not from the Docker container.

CMD ["executable","param1","param2"] (exec form, this is the preferred form) CMD ["param1","param2"] (as default parameters to ENTRYPOINT) CMD command param1 param2 (shell form)Copy the code

The docker stop command executed when the container is closed will only send the shutdown signal to the process whose process number is 1. After entering the test environment, the container ps -ef found that the current service process number is not 1. The specific reason is that the current service in Dockerfile runs as a mirror managed by a process, and process 1 in the container is a process-managed service. The current service is only a sub-process of the process management service, so the current service cannot receive the shutdown signal. Instead of using the process management service to start the business service, start the business service directly so that the business process is process 1 in the container. Graceful shutdown only takes effect when the business process receives a shutdown signal when it is sent to the test environment for testing.

Conclusion: 1. Golang graceful closing needs to add the remaining HTTP request codes to the code after receiving the closing signal. 2. Docker microservice graceful closing first needs to ensure that the business process is the no. 1 process in the container, so as to receive the closing signal.Copy the code