For microservices, graceful offline service is a necessity.

For online, services should not be exposed if the component or container is not successfully started, and for offline, services should be taken offline if the machine is down, so as to prevent upstream traffic from entering the unhealthy machine.

Elegant offline

Base offline (Spring/SpringBoot/ built-in container)

First, the JVM itself supports graceful shutdownHook.

Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { close(); }});Copy the code

This mode supports graceful downtime in the following scenarios:

Exit the program normally. 2. Use System.exit(). 3. Use the Kill PID command to Kill a process

So if you just want to kill -9 the program is going to be overwhelmed.

In Springboot, shutdownHook is already implemented for you, which can respond to Ctrl+ C or kill -15 TERM signals.

Just started an application, and then Ctrl + c, observe the log, it in AnnotationConfigEmbeddedWebApplicationContext this class to print out the suspected Closing… Log, the real implementation logic in his father’s class AbstractApplicationContext (this is in the spring class, what does it mean, in the spring will support the extension of elegant downtime).

public void registerShutdownHook() { if (this.shutdownHook == null) { this.shutdownHook = new Thread() { public void run() { synchronized(AbstractApplicationContext.this.startupShutdownMonitor) { AbstractApplicationContext.this.doClose(); }}}; Runtime.getRuntime().addShutdownHook(this.shutdownHook); } } public void destroy() { this.close(); } public void close() { Object var1 = this.startupShutdownMonitor; synchronized(this.startupShutdownMonitor) { this.doClose(); if (this.shutdownHook ! = null) { try { Runtime.getRuntime().removeShutdownHook(this.shutdownHook); } catch (IllegalStateException var4) { ; } } } } protected void doClose() { if (this.active.get() && this.closed.compareAndSet(false, true)) { if (this.logger.isInfoEnabled()) { this.logger.info("Closing " + this); } LiveBeansView.unregisterApplicationContext(this); try { this.publishEvent((ApplicationEvent)(new ContextClosedEvent(this))); } catch (Throwable var3) { this.logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", var3); } if (this.lifecycleProcessor ! = null) { try { this.lifecycleProcessor.onClose(); } catch (Throwable var2) { this.logger.warn("Exception thrown from LifecycleProcessor on context close", var2); } } this.destroyBeans(); this.closeBeanFactory(); this.onClose(); this.active.set(false); }}Copy the code

What can we do with it? Well, obviously, in the doClose method it publishes a ContextClosedEvent for our extension.

So we can write a listener to listen for ContextClosedEvent and do logoff logic when an event occurs, which in the case of microservices is to unregister the service from the registry.

@Component public class GracefulShutdownListener implements ApplicationListener<ContextClosedEvent> { @Override public Void onApplicationEvent (ContextClosedEvent ContextClosedEvent) {/ / cancellation logic zookeeperRegistry unregister (mCurrentServiceURL);  . }}Copy the code

One question might be, in microservices, logout is usually the first step to gracefully log off, followed by the shutdown operation, so what happens when the traffic comes in?

I would suggest that after logging out of the service, the request baffle can be turned on to reject the traffic, and the rejected traffic can be processed through the failover function of the microservice framework itself.

Logoff in Docker

I use Docker to deploy the service, whether to support elegant offline.

Here’s what docker’s stop commands do:

Generally speaking, normal people may use docker stop or Docker kill commands to close containers (of course, if USR2 custom information was registered in the previous step, it may be closed by docker exec kill -12).

For docker Stop, it sends a SIGTERM(kill -15 term message) to the container’s PID1 process, and by default, it waits 10 seconds before sending a SIGKILL(kill -9 message) to PID1.

Obviously, Docker Stop allows an application to have a default reaction time of 10 seconds to perform a graceful shutdown, as long as the application responds to a kill -15 signal, as described in the previous step. So that’s a good way to do it.

Of course, if the shutdownHook method executes for more than 50 years, it’s definitely not elegant. You can add the wait time with docker stop -t.

Shutdown script for external containers (Jetty)

If you have to deploy in an external container (I think it is a waste of resources and increases complexity). So can we gracefully shut down.

Yes and yes, there are two ways:

First, the RPC framework itself provides elegant up-down interfaces that can be called to end the entire application lifecycle, and extension points that allow developers to customize the downtime logic for services to go offline. At the same time, the call to this interface is encapsulated as a preStop operation and solidified in the Shutdown script of Jetty and other containers, ensuring that the offline interface is called before the container is stopped to end the application life cycle. In the shutdown script, the execution class initiates the offline service -> closes the port -> checks the offline service until it is complete -> closes the container.

An easier alternative is to simply add the kill -15 command to the script. Elegant online

Gracefully launching may be more difficult because there is no default implementation, but in general, a rule of thumb is to ensure that the port exists before launching the service.

Springboot built-in container comes online gracefully

This is very simple, and the industry in the application layer of elegant on-line are implemented under the premise of the built-in container, and can also be used with a series of health checks.

See sofA-Boot health check source code, it will start the program on springboot components to do some health check, and then on some of its own sofa middleware health check, the whole health check process after (Sofaboot is currently unable to do their own application level health check, It has a write interface, but the port is ready… Will expose the service or gracefully online, then its health check when it:

@Override
public void onApplicationEvent(ContextRefreshedEvent event) {
    healthCheckerProcessor.init();
    healthIndicatorProcessor.init();
    afterHealthCheckCallbackProcessor.init();
    publishBeforeHealthCheckEvent();
    readinessHealthCheck();
}
Copy the code

You can see that it’s listening for the ContextRefreshedEvent event. In the built-in container mode, the start method of the built-in container mode is in the refreshContext method, and a ContextRefreshedEvent event is issued after the method is executed. That is to say, the built-in container must be started successfully when the event is heard.

However, the event of ContextRefreshedEvent will be monitored for many times in some specific scenes due to various reasons, so there is no way to ensure that the current event is the last one, so as to correctly execute the elegant online logic.

An even more recent event in SpringBoot is called ApplicationReadyEvent, which is published on the listeners. Finished (context, null) after afterRefresh. It is absolutely guaranteed that the built-in container port exists, so we can listen for this event to do elegant on-line logic and even integrate middleware related health checks here.

@Component public class GracefulStartupListener implements ApplicationListener<ApplicationReadyEvent> { @Override public Void onApplicationEvent(ApplicationReadyEvent ApplicationReadyEvent){// API register.register (urls); . }}Copy the code

The external container (Jetty) comes online gracefully

The deployment mode of most current applications, whether jetty or Docker (also using Jetty images), is essentially an external container. This situation is more difficult, at least at the application level, because the external container is not visible, and the container itself provides no hooks for you to implement.

Then, as with graceful going-on, RPC frameworks need to provide graceful going-on interfaces to initiate the entire application lifecycle and extension points for developers to perform custom going-on logic (reporting version detection information, etc.). This interface is also called as a postStart operation, which is solidified in the Startup script of external containers such as Jetty to ensure that the application is online after the container is started. The container performs a process similar to starting the container -> Health Check -> Online service logic -> Health online service until complete.Author: FredalxinFredal. Xin/graceful – so…