background

There are many microservices in the project. When an HTTP request is made to service A with some header information, service A needs to call service B and C, and service B and C also need to obtain these header information, this scenario is very common in the project. In order to call other services, these headers are automatically passed to the next service. We write a global Feign RequestInterceptor that uses the RequestContextHolder to get the headers of the current request and set them into the header of the interface that calls the downstream service.

@Bean
RequestInterceptor globalRequestInterceptor() {
    return (template) -> {
        WHITE_LIST_HEADER_FIELDS.forEach((headerField) -> {
            String headerValue = this.getHeader(headerField);
            if (!template.headers().containsKey(headerField) && Objects.nonNull(headerValue)) {
              template.header(headerField, new String[]{headerValue});
            }
        });
    };
}
      
private String getHeader(String header) {
    HttpServletRequest request = this.getHttpServletRequest();
    return request == null ? null : request.getHeader(header);
}

private HttpServletRequest getHttpServletRequest() {
    ServletRequestAttributes attributes = (ServletRequestAttributes)RequestContextHolder.getRequestAttributes();
    return attributes == null ? null : attributes.getRequest();
}
Copy the code

The problem

An HTTP request comes to interface A1 of service A. Interface A1 has A part of the code logic A2 is asynchronous. In the asynchronous a2 method, service B needs to be called. When calling the B service through feign and getting to the RequestInterceptor, the online environment occasionally doesn’t get the header.

@Service
public class Service1 {

  @Autowired
  private Service2 service2;

  public void a1() {
    // some sync code
    service2.a2();
    //some sync code
  }
}
Copy the code
@Service
public class Service2 {

  @Autowired
  private BClient bClient;

  @Async
  public void a2() {/ /dosome thing bClient.callB1(); }}Copy the code
public interface BClient {

  @PostMapping("/b1")
  void callB1();
}
Copy the code

Analysis of the

See first obtain RequestAttributes code RequestContextHolder. GetRequestAttributes (), through the debug is found here get RequestAttributes is null.

Into the RequestContextHolder source can see the value of is actually the requestAttributesHolder or inheritableRequestAttributesHolder returned, These are threadLocal objects, and the source code comment tells us that this method will actually put back the requestAttributes bound to the current thread. The requestAttribute is not available in the child thread.

The most direct way to do this is to get the header in the main thread code and pass it to the Feign Client layer by layer. But it feels a bit cumbersome to write similar code for each asynchronous method. Again wanted to think we let each child threads to obtain the main thread RequestAtribute and then set the child thread is not good, direct use of ThreadPoolTaskExecutorsetTaskDecorator setTaskDecorator for decoration.

    threadPoolTaskExecutor.setTaskDecorator((runnable)->{
      final RequestAttributes requestAttributes = RequestContextHolder.getRequestAttributes();
      return() -> { try { RequestContextHolder.setRequestAttributes(requestAttributes); runnable.run(); } finally { RequestContextHolder.resetRequestAttributes(); }}; });Copy the code

The commit code seemed to be done, and the local tests were fine, but then something magical happened, and the bug went from stable to occasional.

It turns out that things aren’t that simple, but following the source code, Spring actually initializes and destroys RequestAttributes in the RequestContextFilter class’s doFilterInternal method.

    @Override
	protected void doFilterInternal( HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException { ServletRequestAttributes attributes = new ServletRequestAttributes(request, response); // Initialize ServletRequestAttributes to ThreadLocal initContextHolders(Request, Attributes); // Initialize ServletRequestAttributes to ThreadLocal initContextHolders(Request, Attributes); try { filterChain.doFilter(request, response); } finally {//remove ThreadLocal<RequestAttributes> resetContextHolders();if (logger.isTraceEnabled()) {
				logger.trace("Cleared thread-bound request context: "+ request); } attributes.requestCompleted(); }}Copy the code

When the child thread obtains the RequestAttribute, the main thread has already completed its execution and cannot obtain the RequestAttribute.

In setTaskDecorator, copy the RequestAttribute obtained by the main thread and set it to the child thread.

Fortunately, the bug happened again, but this time it was not the RequestAttribute that failed to get, but the header became null. After some operation, I finally found the bug.

Tomcat’s Http11InputBuffer has a “Recycle” method that is called when an HTTP connection is closed. The “Recycle” method also calls the “Request” method. This method clears the contentLength, headers and other information

    /**
     * Recycle the input buffer. This should be called when closing the
     * connection.
     */
    void recycle() {
        wrapper = null;
        request.recycle();

        for (int i = 0; i <= lastActiveFilter; i++) {
            activeFilters[i].recycle();
        }

        byteBuffer.limit(0).position(0);
        lastActiveFilter = -1;
        parsingHeader = true;
        swallowInput = true;

        headerParsePos = HeaderParsePosition.HEADER_START;
        parsingRequestLine = true;
        parsingRequestLinePhase = 0;
        parsingRequestLineEol = false;
        parsingRequestLineStart = 0;
        parsingRequestLineQPos = -1;
        headerData.recycle();
    }

Copy the code
    public void recycle() {
        bytesRead=0;

        contentLength = -1;
        contentTypeMB = null;
        charset = null;
        characterEncoding = null;
        expectation = false;
        headers.recycle();
        trailerFields.clear();
        serverNameMB.recycle();
        serverPort=-1;
        localAddrMB.recycle();
        localNameMB.recycle();
        localPort = -1;
        remoteAddrMB.recycle();
        remoteHostMB.recycle();
        remotePort = -1;
        available = 0;
        sendfile = true;

        serverCookies.recycle();
        parameters.recycle();
        pathParameters.clear();

        uriMB.recycle();
        decodedUriMB.recycle();
        queryMB.recycle();
        methodMB.recycle();
        protoMB.recycle();

        schemeMB.recycle();

        remoteUser.recycle();
        remoteUserNeedsAuthorization = false;
        authType.recycle();
        attributes.clear();

        listener = null;
        allDataReadEventSent.set(false);

        startTime = -1;
    }
Copy the code

There are two solutions to this problem. One is to make a deep copy of the RequestAttribute object and store it in the child thread. Another idea: Since we only need the header information, we can simply retrieve the header information from the main thread, define a HeaderContextHolder, and set it to the threadLocal of the child thread. When we retrieve the header from the interceptor, Go to HeaderContextHolder to retrieve the header information (if it is available, the current thread is the main thread). If HeaderContextHolder is not available, go to Spring’s RequestContextHolder to retrieve the heder. Let’s do the second thought here.

@Slf4j public class ContextTaskDecorator implements TaskDecorator { /** * Bind the given RequestAttributes to the Thread */ @override public Runnable (Runnable Runnable) { String> map = wrapHeaderMap();return() -> { try { HeaderContextHolder.setContext(map); runnable.run(); } finally { HeaderContextHolder.removeContext(); }}; }}Copy the code
public class HeaderContextHolder {

  private HeaderContextHolder() {
  }

  private static final ThreadLocal<Map<String, String>> CTX = new ThreadLocal<>();

  public static void setContext(Map<String, String> map) {
    CTX.set(map);
  }

  public static void removeContext() {
    CTX.remove();
  }

  public static String getHeader(String key) {
    if (isEmpty()) {
      return null;
    }
    return CTX.get().get(key);
  }

  public static boolean isEmpty() {
    returnMapUtils.isEmpty(CTX.get()); }}Copy the code

That settles the matter at last.