HttpClient hidden BUG parsing

This is the third day of my participation in the First Challenge 2022

preface

• During the development of a project, there will always be a scenario for interacting with external three-party interfaces. In general, such interfaces are usually accessed using HTTP requests.
• HTTP requests are typically sent using HttpClient and OkHttp.
• This article describes some considerations when using HttpClient.

Example HttpClient request

• Start by creating a simple SpringBoot project with IDEA and introducing maven configuration for HttpClient

< the dependency > < groupId > org, apache httpcomponents < / groupId > < artifactId > httpclient < / artifactId > < version > 4.5.6 < / version > </dependency>Copy the code

• Build a simple HttpClient utility class with a configured timeout of 3000 ms, a connection pool of up to 200, and a method to send get requests.

Slf4j Public Final Class HttpClientUtil {private HttpClientUtil () {} /** * Timeout (ms) */ private static final int SOCKET_TIMEOUT = 3000; private static RequestConfig requestConfig; private static CloseableHttpClient httpClient; static { PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(); cm.setMaxTotal(200); cm.setDefaultMaxPerRoute(100); requestConfig = RequestConfig.custom() .setConnectionRequestTimeout(SOCKET_TIMEOUT) .setConnectTimeout(SOCKET_TIMEOUT) .setSocketTimeout(SOCKET_TIMEOUT) .build(); httpClient = HttpClients.custom() .setConnectionManager(cm) .setConnectionManagerShared(true) .build(); } public static String httpGetRequest (String url) { long start = System.currentTimeMillis(); HttpGet httpGet = new HttpGet(url); Try {// Set timeout configuration httpget.setConfig (requestConfig); // Execute the request CloseableHttpResponse Response = httpClient.execute(httpGet); Log.info ("HttpRequest request time: {}", (system.currentTimemillis () -start)); // Get the result of the request HttpEntity entity = response.getentity (); if (entity ! = null) { String result = EntityUtils.toString(entity); response.close(); return result; }} catch (IOException e) {log.info("HttpRequest time: {}, exception: {}", (system.currentTimemillis () -start), LLDB message ()); e.printStackTrace(); } return null; }}Copy the code

• Set port 8801 in the project configuration file, then open an external interface and write a main method to access the HTTP interface.

/** * @Author: ZRH * @Date: 2022/1/24 15:00 */ @Slf4j @RestController public class HttpClientController { @GetMapping("/httpClient/test") public String test () throws InterruptedException {log.info(" The interface request came in successfully "); return "ok"; } public static void main (String[] args) { String url = "http://localhost:8801/httpClient/test"; String result = HttpClientUtil.httpGetRequest(url); System.out.println(" request result: "+ result); }}Copy the code

• After the Springboot project is started, the current main method is executed and the result is as follows:

If the request is abnormal, try again

• The HttpClient configuration is enabled by default, and the source of the configuration is found in the httpClientBuilder.build () method:

public CloseableHttpClient build() { // Create main request executor // We copy the instance fields to avoid changing them, and rename to avoid accidental use of the wrong version ...... // Add request retry executor, if not disabled if (! automaticRetriesDisabled) { HttpRequestRetryHandler retryHandlerCopy = this.retryHandler; if (retryHandlerCopy == null) { retryHandlerCopy = DefaultHttpRequestRetryHandler.INSTANCE; } execChain = new RetryExec(execChain, retryHandlerCopy); }... }Copy the code

• Through DefaultHttpRequestRetryHandler. The default configuration INSTANCE can enter DefaultHttpRequestRetryHandler class.
• DefaultHttpRequestRetryHandler HttpRequestRetryHandler interface is achieved, There is a retryRequest(IOException Exception, int executionCount, HttpContext Context) method. The retryRequest method mainly performs retry logic. The three parameter sub-tables are the exceptions that occur, the number of retries, and the context of the current request link.
• If the retry mechanism is enabled by default, block the server interface for a few seconds to see if the client will retry.

@getMapping ("/httpClient/test") public String test () throws InterruptedException {log.info(" Interface request came in successfully "); Thread.sleep(4000); return "ok"; } -- -- -- -- -- -- -- -- -- -- -- -- run the main method, the results are as follows: 17:07:59. 229 logback [main] INFO com. Mysql. Web. Utils. HttpClientUtil - time-consuming HttpRequest request: 3040, exceptions: the Read timed out request results: null java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.mysql.web.utils.HttpClientUtil.httpGetRequest(HttpClientUtil.java:75) at com.mysql.web.controller.HttpClientController.main(HttpClientController.java:41)Copy the code

• Here you can see directly throws a timeout exception: Read the link java.net.SocketTimeoutException: Read timed out. There is no retry process.
• There are no secrets under the source code, and since this is not what we expected, let’s dig into the source code to see how it works…
  • The first step is to check whether the number of retries exceeds 3, if so, no more retry, otherwise continue,
  • This. NonRetriableClasses is a set of classes whose name indicates that this is a class that cannot be retried.
  • Then determine if the exception has a subclass instance in the collection this.nonRetriableClasses,
  • After the process has nothing to do with the node, press not table.

/** * Used {@code retryCount} and {@code requestSentRetryEnabled} to determine * if the given method should be retried. */ @Override public boolean retryRequest( final IOException exception, final int executionCount, final HttpContext context) { Args.notNull(exception, "Exception parameter"); Args.notNull(context, "HTTP context"); if (executionCount > this.retryCount) { // Do not retry if over max retry count return false; } if (this.nonRetriableClasses.contains(exception.getClass())) { return false; } else { for (final Class<? extends IOException> rejectException : this.nonRetriableClasses) { if (rejectException.isInstance(exception)) { return false; } } } final HttpClientContext clientContext = HttpClientContext.adapt(context); final HttpRequest request = clientContext.getRequest(); if(requestIsAborted(request)){ return false; } if (handleAsIdempotent(request)) { // Retry if the request is considered idempotent return true; } if (! clientContext.isRequestSent() || this.requestSentRetryEnabled) { // Retry if the request has not been sent fully or // if it's OK to retry methods that have been sent return true; } // otherwise do not retry return false; }Copy the code

• After understanding the retry process, through the debug is go to rejectException. The isInstance (exception) finally returned to the false judgment, finally, there is no retry

• Enclosing nonRetriableClasses collection has four different exception class, through DefaultHttpRequestRetryHandler constructor.
• So if an HTTP request throws an exception that falls into one of the following four categories, it will not retry. The SocketTimeoutException in the previous test example is InterruptedIOException, so there is no retry mechanism.
• Note: The retryCount parameter in the current constructor is the default three retries

/** * Create the request retry handler using the following list of * non-retriable IOException classes: <br> * <ul> * <li>InterruptedIOException</li> * <li>UnknownHostException</li> * <li>ConnectException</li> * <li>SSLException</li> * </ul> * @param retryCount how many times to retry; 0 means no retries * @param requestSentRetryEnabled true if it's OK to retry non-idempotent requests that have been sent  */ @SuppressWarnings("unchecked") public DefaultHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled) { this(retryCount, requestSentRetryEnabled, Arrays.asList( InterruptedIOException.class, UnknownHostException.class, ConnectException.class, SSLException.class)); }Copy the code

• If you want to use the retry mechanism, you can implement the HttpRequestRetryHandler interface and override the retryRequest logic. Simply add the following configuration to the HttpClient utility class:

httpClient = HttpClients.custom() .setConnectionManager(cm) .setConnectionManagerShared(true) .setRetryHandler(new HttpRequestRetryHandler() { @Override public boolean retryRequest (IOException exception, int executionCount, HttpContext context) { if (executionCount > 3) { // Do not retry if over max retry count return false; } // Allow retry exception if (exception instanceof SocketTimeoutException) {return true; } // otherwise do not retry return false; } }) .build();Copy the code

• Finally, using the main method again, the result is as follows:

----------------------- Client printing: 17:31:39. 097 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - I/O exception (java.net.SocketTimeoutException) caught when processing request to {}->http://localhost:8801: Read timed out 17:31:39. 100 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - Retrying the request to {} - > http://localhost:8801 17:31:42. 103 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - I/O exception (java.net.SocketTimeoutException) caught when processing request to {}->http://localhost:8801: Read timed out 17:31:42. 103 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - Retrying the request to {} - > http://localhost:8801 17:31:45. 106 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - I/O exception (java.net.SocketTimeoutException) caught when processing request to {}->http://localhost:8801: Read timed out 17:31:45. 106 logback [main] INFO O.A.H TTP. The impl. Execchain. RetryExec - Retrying the request to {} - > http://localhost:8801 17:31:48. 108 logback [main] INFO com. Mysql. Web. Utils. HttpClientUtil - time-consuming HttpRequest request: 12063, exceptions: the Read timed out request results: null java.net.SocketTimeoutException: Read timed out... ---------------------- Server printing: 17:31:36. 143 logback task [XNIO - 1-1] INFO C.M.W.C.H ttpClientController - successful 17:31:39 interface request came in. 102 logback task [XNIO - 1-2] The INFO C.M.W.C.H ttpClientController - successful 17:31:42 interface request came in. 107 logback task [XNIO - 1-1] INFO C.M.W.C.H ttpClientController - Interface request in successful 17:31:45. 107 logback task [XNIO - 1-2] INFO C.M.W.C.H ttpClientController success - interface request came inCopy the code

• So the retry mechanism was implemented, and you can see from the result that the HTTP request was retried three times, and then the original request was added, so there were actually four HTTP requests sent.

SocketTimeout scope

• The following link timeout configuration was added to the httpClient utility class in the above example:
  • ConnectionRequestTimeout: Gets the connection timeout configuration from the local connection pool.
  • ConnectTimeout: indicates the socket timeout configuration for connecting to the server.
  • SocketTimeout: Indicates the timeout configuration for reading socket packets from the server.
• SocketTimeout indicates that the client has connected to the server socket connection, but the data read times out. For example, the server is blocked due to timeout.
• However, socketTimeout only refers to the packet transmission and reading time. If an HTTP request is returned with multiple packets, and each packet does not reach the socketTimeout value, but the total packet transmission time exceeds the socketTimeout value, no timeout exception is reported.
• For example, httpClient uses the same 3000 ms configuration as before, and then adds the following methods:

@GetMapping("/httpClient/test/string") public void string (HttpServletResponse response) throws InterruptedException, IOException { final long start = System.currentTimeMillis(); Log.info (" interface request came in successfully "); String[] arr = new String[]{"a", "b", "c", "d"}; for (int i = 0; i < 4; i++) { response.setStatus(HttpServletResponse.SC_OK); response.getWriter().println(arr[i]); response.flushBuffer(); Thread.sleep(2000); } log.info(" Total interface time: {}", (system.currentTimemillis () -start)); } public static void main (String[] args) { final long start = System.currentTimeMillis(); String regUrl = "http://localhost:8801/httpClient/test/string"; String respStr = HttpClientUtil.httpGetRequest(regUrl); System.out.println(" Request time: "+ (system.currentTimemillis () -start)); System.out.println(respStr); } -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - to run the main method, the service side print: 17:49:24. 928 logback task [XNIO - 1-2] INFO C.M.W.C.H ttpClientController - interface total time: 8007 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - the client printing: 17:57:24. 022 logback [main] INFO com. Mysql. Web. Utils. HttpClientUtil - time-consuming HttpRequest request: 48 Request duration: 8436 A B C DCopy the code

• The server always time-consuming to perform for eight seconds, but the client is no exception: java.net.SocketTimeoutException: Read timed out

The last

• The above problem is obviously not a bug in HttpClient, and the design concept is consistent with most scenarios.
• It is only when the user does not understand the implementation mechanism that it is possible to write buggy code when it is not used properly in a project.
• Learn with an open mind and make progress together