Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

This series is the fifth installment of my TM stupid series [Covering your face].

  • After upgrading to Spring 5.3.x, GC times increased dramatically and I was f * * king stupid
  • Mysql > alter TABLE select * from SQL where table select * from SQL
  • Get abnormal information in the abnormal can not find the log, I TM stupid
  • Spring-data-redis connection is leaking, I’m fucking stupid

This article deals with the underlying design and principles, as well as the problem positioning and possible problem points. It is very in-depth and long, so it is divided into three parts:

  • Above: A brief description of the problem and the basic structure and flow of the Spring Cloud Gateway as well as the underlying principles
  • CD: How did Spring Cloud Sleuth add link tracing to Spring Cloud Gateway and why did this problem occur
  • Next: Performance issues caused by the non-invasive design of the existing Spring Cloud Sleuth, other possible problem points, and how to solve them

Our Gateway uses the Spring Cloud Gateway and adds the spring-Cloud-sleUTH dependency for link tracing. After log4j2 is configured, the link information is output to the log. The relevant placeholders are as follows:

%X{traceId},%X{spanId}
Copy the code

But recently, it is found that the link information in the log is lost. What is going on?

The basic flow and implementation of the Spring Cloud Gateway

First, take a brief look at the basic structure of the Spring Cloud Gateway and how Spring Cloud Sleuth embedded link-tracing code in it. After Spring Cloud Sleuth and Prometheus dependencies are added, the processing flow of the Spring Cloud Gateway is as follows:

Spring Cloud Gateway is an asynchronous responsive Gateway developed based on Spring WebFlux. The asynchronous responsive code is difficult to understand and read. Here I share a method to understand it. Follow this process to understand the Spring Cloud Gateway workflow and underlying principles. The process in the figure above is to spell out a complete Mono (or Flux) flow and subscribe to execute it.

When receiving a request passes through org. Springframework.. Web server handler. DefaultWebFilterChain, this is the invocation chain WebFilter, the link includes three WebFilter:

  • org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter: The MetricsWebFilter is used to record request processing time and collect metrics after promethe-related dependencies are added.
  • org.springframework.cloud.sleuth.instrument.web.TraceWebFilterTraceWebFilter: After you add the Spring Cloud Sleuth dependencies, you get the TraceWebFilter.
  • org.springframework.cloud.gateway.handler.predicate.WeightCalculatorWebFilter: Spring Cloud Gateway route weight related configuration function related implementation class, this we do not care about here.

So in this DefaultWebFilterChain we’re going to have monOS that look like this, and we’re going to mark them up, First is the entry code org. Springframework.. Web server handler. DefaultWebFilterChain# filter:

public Mono<Void> filter(ServerWebExchange exchange) { return Mono.defer(() -> // this.currentFilter ! = null indicates that the WebFilter chain has not ended. = null indicates that the WebFilter chain is not empty this.currentFilter! = null && this.chain ! = null ? // Call WebFilter invokeFilter(this.currentFilter, this.chain, exchange) if the WebFilter chain is not terminated: Handler this.handler.handle(exchange)); }Copy the code

For the first MetricsWebFilter of the WebFilter chain here, assuming that the corresponding collection statistics are enabled, the Mono generated at this time is:

return Mono.defer(() -> chain.filter(exchange).transformDeferred((call) -> { long start = System.nanoTime(); Return the call / / success, record the response time. The doOnSuccess ((done) - > MetricsWebFilter. Enclosing onSuccess (exchange, start)) / / failure, Record the response time and exception. DoOnError ((cause) - > MetricsWebFilter. Enclosing onError (exchange, start, cause)); }); ;Copy the code

For convenience, we simplify the code here. Since we want to splice all Mono and Flux of the whole link together to complete the link, the onSuccess(Exchange, start) method in MetricsWebFilter was originally used. Has been changed to MetricsWebFilter. This. OnSuccess (exchange, start) this pseudo code.

Then, according to DefaultWebFilterChain source analysis, Chain. Filter (Exchange) will continue the WebFilter link to the next WebFilter, namely TraceWebFilter. After TraceWebFilter, Mono becomes:

return Mono.defer(() -> new MonoWebFilterTrace(source, chain.filter(exchange), TraceWebFilter.this.isTracePresent(), TraceWebFilter.this, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

As you can see, in TraceWebFilter, the entire internal Mono (chain.filter(exchange) subsequent result) is encapsulated into a single MonoWebFilterTrace, which is also the key implementation for maintaining link tracing information.

Continue the WebFilter link, after the last WebFilter WeightCalculatorWebFilter; We don’t care about the WebFilter, it does some calculation of the routing weights, so we can just ignore them. So we walked all the WebFilter link, came to the last call DefaultWebFilterChain. Enclosing handler, The handler is org. Springframework. Web. Reactive. DispatcherHandler. In the DispatcherHandler, we calculate the route and send the request to the qualified GatewayFilter. After the DispatcherHandler, Mono will become:

return Mono.defer(() -> new MonoWebFilterTrace(source, Flux. FromIterable (DispatcherHandler. This. HandlerMappings). / / read all handlerMappings concatMap (mapping - > Mapping.gethandler (exchange)) // Call all handlerMappings' getHandler methods in order, returning mono.empty () if there is a corresponding Handler; .next () / / find the first returns not Mono. Empty () Handler. SwitchIfEmpty (DispatcherHandler. This. CreateNotFoundError ()) / / if there is no return HandlerMapping Mono. Empty (), Directly back to 404. The flatMap (handler - > DispatcherHandler. Enclosing invokeHandler (exchange, Handler)) / / call the corresponding handler flatMap (result - > DispatcherHandler. Enclosing handleResult (exchange, result)), / / processing results TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

HandlerMappings include:

  • org.springframework.boot.actuate.endpoint.web.reactive.WebFluxEndPointHandlerMapping: Because of the addition of Actuator dependencies in our project, there is this HandlerMapping. The associated path mapping of the Actuator is not our concern here.However, it can be seen that the relative paths of the Actuator take precedence over the configured routes of the Spring Cloud Gateway
  • org.springframework.boot.actuate.endpoint.web.reactive.ControllerEndpointHandlerMapping: Because of the addition of Actuator dependencies in our project, there is this HandlerMapping. The path mapping of the Actuator using @ControllerEndpoint or @RestControllerendpoint annotations is not a concern here.
  • org.springframework.web.reactive.function.server.support.RouterFunctionMappingIn Spring-WebFlux, you can define many different RouterFunctions to control path routing, but that’s not our concern here either.However, you can see that the custom RouterFunction takes precedence over the Spring Cloud Gateway in configuring routes
  • org.springframework.web.reactive.result.method.annotation.RequestMappingHandlerMappingThe HandlerMapping for the @requestMapping annotation path is not our concern here.However, you can see that if you specify the RequestMapping path in the Spring Cloud Gateway, it will take precedence over the Spring Cloud Gateway to configure routing.
  • org.springframework.cloud.gateway.handler.RoutePredicateHandlerMappingThis is the Spring Cloud Gateway HandlerMapping that reads the Spring Cloud Gateway configuration and generates routes. That’s what we’re going to analyze in detail here.

Actually these handlerMappings, we must go here is RoutePredicateHandlerMapping related logic, so our Mono can be reduced to:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping.this.getHandler(exchange) .switchIfEmpty(DispatcherHandler.this.createNotFoundError()) // If handlerMapping is not returned without mono.empty (), Directly back to 404. The flatMap (handler - > DispatcherHandler. Enclosing invokeHandler (exchange, Handler)) / / call the corresponding handler flatMap (result - > DispatcherHandler. Enclosing handleResult (exchange, result)), / / processing results TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

Let’s look at RoutePredicateHandlerMapping, First the handlerMapping are inherited abstract class org. Springframework. Web. Reactive. Handler. AbstractHandlerMapping, The implementation of getHandler in Mono is in this abstract class:

Public Mono<Object> getHandler(ServerWebExchange Exchange) {// Call the abstract getHandlerInternal method to get the real Handler return GetHandlerInternal (exchange).map(handler -> {if (logger.isDebugenabled ()) { logger.debug(exchange.getLogPrefix() + "Mapped to " + handler); } // ServerHttpRequest request = exchange.getrequest (); if (hasCorsConfigurationSource(handler) || CorsUtils.isPreFlightRequest(request)) { CorsConfiguration config = (this.corsConfigurationSource ! = null ? this.corsConfigurationSource.getCorsConfiguration(exchange) : null); CorsConfiguration handlerConfig = getCorsConfiguration(handler, exchange); config = (config ! = null ? config.combine(handlerConfig) : handlerConfig); if (config ! = null) { config.validateAllowCredentials(); } if (! this.corsProcessor.process(config, exchange) || CorsUtils.isPreFlightRequest(request)) { return NO_OP_HANDLER; } } return handler; }); }Copy the code

As you can see, the core is the getHandlerInternal(Exchange) method of each implementation class, so in our concatenated Mono, we will ignore the map processing for the handler in the abstract class.

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping.this.getHandlerInternal(exchange) SwitchIfEmpty (DispatcherHandler. This. CreateNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Directly back to 404. The flatMap (handler - > DispatcherHandler. Enclosing invokeHandler (exchange, Handler)) / / call the corresponding handler flatMap (result - > DispatcherHandler. Enclosing handleResult (exchange, result)), / / processing results TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

Then after RoutePredicateHandlerMapping getHandlerInternal (exchange) method, our Mono becomes:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping. This. LookupRoute (exchange) / / according to the request for routing. FlatMap ((Function < the Route, Mono<?>>) r -> { exchange.getAttributes().put(GATEWAY_ROUTE_ATTR, r); / / routing in the Attributes, we will use the return back Mono. Just (RoutePredicateHandlerMapping. This. WebHandler); / / return RoutePredicateHandlerMapping FilteringWebHandler}). SwitchIfEmpty (. / / if for Mono empty (), Empty () // Return mono.empty ().then(mono.fromrunnable (() -> {// Return mono.empty (), If (logger.istraceEnabled ()) {logger.trace("No RouteDefinition found for [" + getExchangeDesc(exchange) + "]"); }}))). SwitchIfEmpty (DispatcherHandler. Enclosing createNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Directly back to 404. The flatMap (handler - > DispatcherHandler. Enclosing invokeHandler (exchange, Handler)) / / call the corresponding handler flatMap (result - > DispatcherHandler. Enclosing handleResult (exchange, result)), / / processing results TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

RoutePredicateHandlerMapping. This. LookupRoute (exchange) according to the request for the routing, that we won’t open, is according to your Spring Cloud Gateway configuration, to find the right routing. Let’s look at calling the corresponding Handler, FilteringWebHandler. DispatcherHandler. This. InvokeHandler exchange, handler) we here also is not in detail, we know that in fact is the handle method invocation handler, The Handle method of FilteringWebHandler, so our Mono becomes:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping. This. LookupRoute (exchange) / / according to the request for routing. FlatMap ((Function < the Route, Mono<?>>) r -> { exchange.getAttributes().put(GATEWAY_ROUTE_ATTR, r); / / routing in the Attributes, we will use the return back Mono. Just (RoutePredicateHandlerMapping. This. WebHandler); / / return RoutePredicateHandlerMapping FilteringWebHandler}). SwitchIfEmpty (. / / if for Mono empty (), Empty (). Then (mono.fromrunnable (() -> {// return mono.empty (), If (logger.istraceEnabled ()) {logger.trace("No RouteDefinition found for [" + getExchangeDesc(exchange) + "]"); }}))). SwitchIfEmpty (DispatcherHandler. Enclosing createNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Directly back to 404. Then (FilteringWebHandler. This. Handle (exchange). Then (Mono) empty ())) / / call the corresponding Handler. FlatMap (result - > DispatcherHandler. This. HandleResult (exchange, result)), / / processing results TraceWebFilter. Enclosing isTracePresent (), TraceWebFilter.this, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

Since calling the corresponding Handler returns mono.empty (), the flatMap will not actually be executed. So we can get rid of the final result. So our Mono becomes:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping. This. LookupRoute (exchange) / / according to the request for routing. FlatMap ((Function < the Route, Mono<?>>) r -> { exchange.getAttributes().put(GATEWAY_ROUTE_ATTR, r); / / routing in the Attributes, we will use the return back Mono. Just (RoutePredicateHandlerMapping. This. WebHandler); / / return RoutePredicateHandlerMapping FilteringWebHandler}). SwitchIfEmpty (. / / if for Mono empty (), Empty (). Then (mono.fromrunnable (() -> {// return mono.empty (), If (logger.istraceEnabled ()) {logger.trace("No RouteDefinition found for [" + getExchangeDesc(exchange) + "]"); }}))). SwitchIfEmpty (DispatcherHandler. Enclosing createNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Directly back to 404. Then (FilteringWebHandler. This. Handle (exchange). Then (Mono) empty ()))), / / call the corresponding Handler TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

FilteringWebHandler. This. Handle (exchange) is removed from the Attributes routing, removed from the routing corresponding GatewayFilters, And global GatewayFilters on the same List, and according to the order of these GatewayFilter (can be implemented by org. Springframework. Core. Ordered interface to set order), Then generate DefaultGatewayFilterChain GatewayFilter link. The corresponding source code is:

Public Mono<Void> Handle (ServerWebExchange Exchange) { Removed from the routing corresponding GatewayFilters Route the Route = exchange. GetRequiredAttribute (GATEWAY_ROUTE_ATTR); List<GatewayFilter> gatewayFilters = route.getFilters(); List<GatewayFilter> combined = new ArrayList<>(this.globalfilters); combined.addAll(gatewayFilters); / / in accordance with the order of these GatewayFilter (can be realized through ` org. Springframework. Core. The Ordered ` interface to make order) AnnotationAwareOrderComparator.sort(combined); if (logger.isDebugEnabled()) { logger.debug("Sorted gatewayFilterFactories: " + combined); } / / generated invocation chain return new DefaultGatewayFilterChain (combined) filter (exchange); }Copy the code

The GatewayFilter invocation chain and WebFilter invocation chain, reference DefaultGatewayFilterChain source code:

Public Mono<Void> filter(ServerWebExchange exchange) {return mono.defer (() -> {// If the link does not end, If (this.index < filter.size ()) {GatewayFilter filter = filter.get (this.index); // index + 1, Who is the next GatewayFilter of call link DefaultGatewayFilterChain chain = new DefaultGatewayFilterChain (this, this. The index + 1); Return filter.filter(exchange, chain); return filter.filter(exchange, chain); } else {// reach the end of the link return mono.empty (); // complete } }); }Copy the code

So, after DefaultGatewayFilterChain our Mono becomes:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping. This. LookupRoute (exchange) / / according to the request for routing. FlatMap ((Function < the Route, Mono<?>>) r -> { exchange.getAttributes().put(GATEWAY_ROUTE_ATTR, r); / / routing in the Attributes, we will use the return back Mono. Just (RoutePredicateHandlerMapping. This. WebHandler); / / return RoutePredicateHandlerMapping FilteringWebHandler}). SwitchIfEmpty (. / / if for Mono empty (), Empty (). Then (mono.fromrunnable (() -> {// return mono.empty (), If (logger.istraceEnabled ()) {logger.trace("No RouteDefinition found for [" + getExchangeDesc(exchange) + "]"); }}))). SwitchIfEmpty (DispatcherHandler. Enclosing createNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Directly back to 404. Then (new DefaultGatewayFilterChain (combined). The filter (exchange). Then (Mono) empty ()))), / / call the corresponding Handler TraceWebFilter. This. IsTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

Call, continue to expand DefaultGatewayFilterChain link can get:

return Mono.defer(() -> new MonoWebFilterTrace(source, RoutePredicateHandlerMapping. This. LookupRoute (exchange) / / according to the request for routing. FlatMap ((Function < the Route, Mono<?>>) r -> { exchange.getAttributes().put(GATEWAY_ROUTE_ATTR, r); / / routing in the Attributes, we will use the return back Mono. Just (RoutePredicateHandlerMapping. This. WebHandler); / / return RoutePredicateHandlerMapping FilteringWebHandler}). SwitchIfEmpty (. / / if for Mono empty (), Empty (). Then (mono.fromrunnable (() -> {// return mono.empty (), If (logger.istraceEnabled ()) {logger.trace("No RouteDefinition found for [" + getExchangeDesc(exchange) + "]"); }}))). SwitchIfEmpty (DispatcherHandler. Enclosing createNotFoundError ()) / / if there is no return to Mono. Empty handlerMapping (), Then (mono.defer (() -> {// If the link does not end, Continues to link the if (DefaultGatewayFilterChain. This index < DefaultGatewayFilterChain. This. Filters. The size ()) {GatewayFilter filter =  DefaultGatewayFilterChain.this.filters.get(DefaultGatewayFilterChain.this.index); // index + 1, Who is the next GatewayFilter of call link DefaultGatewayFilterChain chain = new DefaultGatewayFilterChain(DefaultGatewayFilterChain.this, DefaultGatewayFilterChain.this.index + 1); Return filter.filter(exchange, chain); return filter.filter(exchange, chain); } else { return Mono.empty(); / / link complete}}). Then (Mono) empty ()))), / / call the corresponding Handler TraceWebFilter. Enclosing isTracePresent (), TraceWebFilter. This, TraceWebFilter. This. SpanFromContextRetriever ()). TransformDeferred (- > (call) {/ / MetricsWebFilter related processing, in front of the code is given, });) ;Copy the code

This creates a complete chain of Mono calls to the Spring Cloud Gateway for routing requests.

Wechat search “my programming meow” public account, a daily brush, easy to improve skills, won a variety of offers