The server code often needs to be upgraded. For online system upgrades, the common approach is to ensure that at least one service is available through front-end load balancing (such as Nginx), successively (gray scale) upgrade. A more convenient way to do this is to do a hot restart on the application, upgrading the application directly and never stopping service.

The principle of


The principle of hot restart is very simple, but involves more details such as system calls and passing of file handles between parent and child processes. The processing process is divided into the following steps:

  1. Monitoring signal (USR2)
  2. Fork the child process (using the same startup command) on receiving the signal and pass the socket file descriptor that the service listens to to the child process
  3. The child listens on the parent’s socket, and both the parent and child can receive requests
  4. After the child successfully starts, the parent stops accepting new connections and waits for the old connections to complete (or times out)
  5. The parent process exits, and the upgrade is complete

details


  • The parent process can pass the socket file descriptor to the child process through the command line, environment variables, and so on
  • The child starts with the same command line as the parent, and in golang’s case overrides the old program with a newer executable
  • The graceful Shutdown method of server.shutdown () is a new feature of go1.8
  • The server.serve (L) method returns immediately upon Shutdown, and the Shutdown method blocks until the context completes, so the Shutdown method is written in the main Goroutine

code


package main import ( "context" "errors" "flag" "log" "net" "net/http" "os" "os/exec" "os/signal" "syscall" "time" ) var  ( server *http.Server listener net.Listener graceful = flag.Bool("graceful", false, "listen on fd open 3 (internal use only)") ) func handler(w http.ResponseWriter, r *http.Request) { time.Sleep(20 * time.Second) w.Write([]byte("hello world233333!!!!" )) } func main() { flag.Parse() http.HandleFunc("/hello", handler) server = &http.Server{Addr: ":9999"} var err error if *graceful { log.Print("main: Listening to existing file descriptor 3.") // cmd.ExtraFiles: If non-nil, entry i becomes file descriptor 3+i. // when we put socket FD at the first entry, it will always be 3(0+3) f := os.NewFile(3, "") listener, err = net.FileListener(f) } else { log.Print("main: Listening on a new file descriptor.") listener, err = net.Listen("tcp", server.Addr) } if err ! = nil { log.Fatalf("listener error: %v", err) } go func() { // server.Shutdown() stops Serve() immediately, thus server.Serve() should not be in main goroutine err = server.Serve(listener) log.Printf("server.Serve err: %v\n", err) }() signalHandler() log.Printf("signal end") } func reload() error { tl, ok := listener.(*net.TCPListener) if ! ok { return errors.New("listener is not tcp listener") } f, err := tl.File() if err ! = nil { return err } args := []string{"-graceful"} cmd := exec.Command(os.Args[0], args...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr // put socket FD at the first entry cmd.ExtraFiles = []*os.File{f} return cmd.Start() } func signalHandler() { ch := make(chan os.Signal, 1) signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM, syscall.SIGUSR2) for { sig := <-ch log.Printf("signal: %v", sig) // timeout context for shutdown ctx, _ := context.WithTimeout(context.Background(), 20*time.Second) switch sig { case syscall.SIGINT, syscall.SIGTERM: // stop log.Printf("stop") signal.Stop(ch) server.Shutdown(ctx) log.Printf("graceful shutdown") return case syscall.SIGUSR2: // reload log.Printf("reload") err := reload() if err ! = nil { log.Fatalf("graceful restart error: %v", err) } server.Shutdown(ctx) log.Printf("graceful reload") return } } }Copy the code

systemd & supervisor


When the parent exits, the child hangs on top of process 1. Management programs such as systemd and supervisord will display the process in a failed state. There are two ways to solve this problem:

  • With pidFile, the pidfile is updated every time the process restarts and the process manager is aware of changes to mainPID through this file.
  • Each hot restart starts a new process and kills the old one. The master PID does not change, and the process is in a normal state for the process manager. A neat implementation

References


  • graceful
  • Graceful Restart in Golang
  • facebookgo/grace
  • endless
  • overseer