One, foreword
In ShutdownHook- Elegant Shutdown Solutions for Java, we talked about how to implement elegant shutdown in Java. Let’s take a closer look at how Dubbo achieves elegant downtime.
Two, Dubbo elegant downtime to be solved
In order to achieve elegant downtime, Dubbo needs to solve some problems:
- New requests cannot be sent to the Dubbo service provider that is down.
- If the service provider is shut down, the service request has been received and the service can be offline only after being processed.
- If a service consumer is shut down, the service requests that have been sent need to wait for the response to return.
Solve the above three problems, in order to minimize the impact of downtime on business, to achieve elegant downtime.
3, 2.5 X
Dubbo elegant downtime is more complete in version 2.5.X, this version of the implementation is relatively simple and easy to understand. So let’s take a look at how Dubbo implements elegant downtime based on Dubbo’s 2.5.X source code.
3.1 Overall implementation scheme of elegant shutdown
The elegant shutdown entry class is in AbstractConfig static code with the following source code:
static {
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
public void run(a) {
if (logger.isInfoEnabled()) {
logger.info("Run shutdown hook now."); } ProtocolConfig.destroyAll(); }},"DubboShutdownHook"));
}
Copy the codeHere will register a ShutdownHook, once an outage would trigger calls ProtocolConfig. DestroyAll ().
ProtocolConfig. DestroyAll () the source code is as follows:
public static void destroyAll(a) {
// Prevent concurrent calls
if(! destroyed.compareAndSet(false.true)) {
return;
}
// Unregister the registry
AbstractRegistryFactory.destroyAll();
// Wait for registry notification
try {
Thread.sleep(ConfigUtils.getServerShutdownTimeout());
} catch (InterruptedException e) {
logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
}
ExtensionLoader<Protocol> loader = ExtensionLoader.getExtensionLoader(Protocol.class);
// Unregister the Protocol
for (String protocolName : loader.getLoadedExtensions()) {
try {
Protocol protocol = loader.getLoadedExtension(protocolName);
if(protocol ! =null) { protocol.destroy(); }}catch(Throwable t) { logger.warn(t.getMessage(), t); }}}Copy the codeAs can be seen from the above, Dubbo elegant downtime is mainly divided into two steps:
- Deregistration
- The cancellation of all
Protocol
3.2 deregister the registry
Unregister registry source code as follows:
public static void destroyAll(a) {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("Close all registries " + getRegistries());
}
// Lock up the registry shutdown process
LOCK.lock();
try {
for (Registry registry : getRegistries()) {
try {
registry.destroy();
} catch (Throwable e) {
LOGGER.error(e.getMessage(), e);
}
}
REGISTRIES.clear();
} finally {
// Release the lockLOCK.unlock(); }}Copy the codeThis method will unregister the internal build registry service. Logout registry internal logic is relatively simple, here is no longer in-depth source code, direct display with pictures.
Source code: AbstractRegistry
In the case of ZK, Dubbo will delete its corresponding service node and then unsubscribe. Due to a change in ZK node information, the ZK server will notify the Dubbo consumer to go offline from the service node and finally close the connection between the service and ZK.
Through the registry, Dubbo notifies consumers that the service is offline, and new requests are no longer sent to the offline node, which solves the first problem mentioned above: new requests can no longer be sent to the offline Dubbo service provider.
However, there are still some disadvantages. Due to the isolation of the network, the connection between the ZK server and Dubbo may be delayed, and the ZK notification may not be notified to the consumer in the first place. In this case, after registering the registry, add the wait base as follows:
// Wait for registry notification
try {
Thread.sleep(ConfigUtils.getServerShutdownTimeout());
} catch (InterruptedException e) {
logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
}
Copy the codeThe default wait time is 10000 ms, you can set dubbo. Service. Shutdown. Wait override the default parameters. 10s is only an empirical value and can be set as required. However, this wait time setting is more careful, can not be set too short, too short will cause the consumer end before receiving the ZK notification, the provider will stop. Do not set the value too long. If the value is too long, the application will be shut down for a long time, which will affect the release experience.
3.3 deregister Protocol
ExtensionLoader<Protocol> loader = ExtensionLoader.getExtensionLoader(Protocol.class);
for (String protocolName : loader.getLoadedExtensions()) {
try {
Protocol protocol = loader.getLoadedExtension(protocolName);
if(protocol ! =null) { protocol.destroy(); }}catch(Throwable t) { logger.warn(t.getMessage(), t); }}Copy the codeLoader# getLoadedExtensions returns two Protocol subclasses, DubboProtocol and InjvmProtocol.
DubboProtocol is used to interact with server requests, while InjvmProtocol is used to interact with internal requests. If the application calls its own Dubbo service, the network call is no longer executed and the internal method is executed directly.
Here we mainly analyze the internal logic of DubboProtocol.
DubboProtocol# destroy source code:
public void destroy(a) {
/ / close the Server
for (String key : new ArrayList<String>(serverMap.keySet())) {
ExchangeServer server = serverMap.remove(key);
if(server ! =null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo server: " + server.getLocalAddress());
}
server.close(ConfigUtils.getServerShutdownTimeout());
} catch(Throwable t) { logger.warn(t.getMessage(), t); }}}/ / close the Client
for (String key : new ArrayList<String>(referenceClientMap.keySet())) {
ExchangeClient client = referenceClientMap.remove(key);
if(client ! =null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo connect: " + client.getLocalAddress() + "-- >" + client.getRemoteAddress());
}
client.close(ConfigUtils.getServerShutdownTimeout());
} catch(Throwable t) { logger.warn(t.getMessage(), t); }}}for (String key : new ArrayList<String>(ghostClientMap.keySet())) {
ExchangeClient client = ghostClientMap.remove(key);
if(client ! =null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo connect: " + client.getLocalAddress() + "-- >" + client.getRemoteAddress());
}
client.close(ConfigUtils.getServerShutdownTimeout());
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
stubServiceMethodsMap.clear();
super.destroy();
}
Copy the codeBy default, Dubbo uses Netty as its underlying communication framework, which is divided into Server and Client. Server is used to receive requests from other consumer clients.
The above source code first shut down the Server, stop receiving new requests, and then shut down the Client. Doing so reduces the likelihood that the service will be invoked by the consumer.
3.4. Close the Server
The first call will be HeaderExchangeServer#close.
public void close(final int timeout) {
startClose();
if (timeout > 0) {
final long max = (long) timeout;
final long start = System.currentTimeMillis();
if (getUrl().getParameter(Constants.CHANNEL_SEND_READONLYEVENT_KEY, true)) {
// Send the READ_ONLY event
sendChannelReadOnlyEvent();
}
while (HeaderExchangeServer.this.isRunning()
&& System.currentTimeMillis() - start < max) {
try {
Thread.sleep(10);
} catch(InterruptedException e) { logger.warn(e.getMessage(), e); }}}// Disable timed heartbeat detection
doClose();
server.close(timeout);
}
private void doClose(a) {
if(! closed.compareAndSet(false.true)) {
return;
}
stopHeartbeatTimer();
try {
scheduled.shutdown();
} catch(Throwable t) { logger.warn(t.getMessage(), t); }}Copy the codeThe READ_ONLY event will be sent to the service consumer. After acceptance, the consumer actively excludes the node and sends the request to other normal nodes. This further reduces the impact of registry notification delays.
The heartbeat detection and the underlying communication framework, NettyServer, will be disabled. This will call the NettyServer#close method, which is actually implemented in AbstractServer.
AbstractServer#close
public void close(int timeout) {
ExecutorUtil.gracefulShutdown(executor, timeout);
close();
}
Copy the codeThis process will complete as many tasks as possible in the thread pool, then shut down the thread pool, and finally shut down the Netty communication underlying Server.
By default, Dubbo dispatches requests/heartbeats to the business thread pool for processing.
Shutting down the Server and gracefully waiting for the thread pool to shut down solves the second problem mentioned above: shutting down the service provider, the service request has been received and needs to be processed before the service can be taken offline.
The closing process of Dubbo service provider is shown as follows:
Ps: In order to facilitate the debugging of the source code, attach the Server close call link.
DubboProtocol#destroy
->HeaderExchangeServer#close
->AbstractServer#close
->NettyServer#doClose
Copy the code3.5 close the Client
The Client is closed in much the same way as the Server. The code is located in HeaderExchange Hannel #close.
// graceful close
public void close(int timeout) {
if (closed) {
return;
}
closed = true;
if (timeout > 0) {
long start = System.currentTimeMillis();
// Wait for the sent request response information
while (DefaultFuture.hasFuture(channel)
&& System.currentTimeMillis() - start < timeout) {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
logger.warn(e.getMessage(), e);
}
}
}
close();
}
Copy the codeIf no response is received when the Client is shut down, the system waits for a certain period of time until all requests receive responses or the waiting time exceeds the timeout period.
Ps :Dubbo requests are temporarily stored in DefaultFuture Map, so a simple look at the Map will tell you if all requests received a response.
This solves the third problem: if the service consumer is closed, service requests that have already been made need to wait for the response to return.
The overall process of Dubbo elegant shutdown is shown in the figure.
Ps: Client closes the call chain as follows:
DubboProtocol#close
->ReferenceCountExchangeClient#close
->HeaderExchangeChannel#close
->AbstractClient#close
Copy the codeFour, 2.7 X
Dubbo is commonly used with the Spring framework, and the 2.5.X version of the downtime process can cause elegant downtime failures. This is because when the Spring framework is shut down, the corresponding ShutdownHook event is also triggered to unregister the relevant Bean. If Spring performs the first outage, the associated Bean is unregistered. The Dubbo shutdown event that references the Spring Bean will cause an exception to occur during the shutdown, causing the graceful shutdown to fail.
To solve this problem, Dubbo began refactoring this part of the logic in version 2.6.X and iterated until version 2.7.X.
ShutdownHookListener, which inherits the Spring ApplicationListener interface, is added to listen for Spring-related events. Here ShutdownHookListener only listens for Spring shutdown events. When Spring starts to shut down, the internal ShutdownHookListener logic will be triggered.
public class SpringExtensionFactory implements ExtensionFactory {
private static final Logger logger = LoggerFactory.getLogger(SpringExtensionFactory.class);
private static final Set<ApplicationContext> CONTEXTS = new ConcurrentHashSet<ApplicationContext>();
private static final ApplicationListener SHUTDOWN_HOOK_LISTENER = new ShutdownHookListener();
public static void addApplicationContext(ApplicationContext context) {
CONTEXTS.add(context);
if (context instanceof ConfigurableApplicationContext) {
/ / register ShutdownHook
((ConfigurableApplicationContext) context).registerShutdownHook();
// Cancel AbstractConfig registered ShutdownHook events
DubboShutdownHook.getDubboShutdownHook().unregister();
}
BeanFactoryUtils.addApplicationListener(context, SHUTDOWN_HOOK_LISTENER);
}
// Inherits ApplicationListener, which will listen for container closure events
private static class ShutdownHookListener implements ApplicationListener {
@Override
public void onApplicationEvent(ApplicationEvent event) {
if (event instanceofContextClosedEvent) { DubboShutdownHook shutdownHook = DubboShutdownHook.getDubboShutdownHook(); shutdownHook.doDestroy(); }}}}Copy the codeWhen the Spring framework starts initialization, the SpringExtensionFactory logic will be triggered, AbstractConfig will be unregistered with ShutdownHook, and ShutdownHookListener will be added. This perfectly solves the above “double hook” problem.
Five, the last
Elegant downtime looks easy to achieve, but there are very many design details, a point to achieve a problem, will lead to elegant downtime failure. If you are also implementing elegant downtime, consider Dubbo’s implementation logic.
Dubbo series articles recommended
1. If someone asks you how the registry works in Dubbo, give them this article 2. Don’t know how to implement dynamic discovery of services? Dubbo Zk data structure 4. Dubbo: Spring XML Schema extension mechanism
Help articles
1. I highly recommend reading kirito’s article on Dubbo’s elegant downtime
Welcome to pay attention to my public account: procedures to get daily dry goods push. If you are interested in my topics, you can also follow my blog: studyidea.cn
