background
Recently, a new project was launched. This project is a compression system, which can be seen as simply sending requests to the service by playing back the word table (HTTP request data) to achieve the purpose of compression service. In the test process, everything is still smooth, fixed a few small bugs, on the line. A serious problem was found when the application was used by the first business party after it was launched. After running for about 10 minutes, a large number of Full GC alarms were received.
In view of this problem, we first confirmed the content of the scene of pressure test with the business side. The number of word lists played back is about 100,000, and the playback rate is about 100qps for a single machine, which is far lower than the limit that a single machine can bear according to our previous estimate. There should be no memory problems.
Online screening
First, we need to run a sweep on the server. Use the JDK’s jmap tool to see what objects exist in a JAVA application, the number of actual objects, and their size. Specific commands are as follows:
Jmap-histo :live 'pid of Java' > / TMP /jmap00Copy the codeAccording to the business logic, the most instantiated object is the word list, that is more than 100,000. How can there be more than 200,000? We did not find any place in the code that shows the declaration of instantiation. Dump = dump = dump = dump = dump
jmap -dump:format=b,file=heap.dump `pid of java
Copy the codeOffline analysis
After downloading the dump heap. Dump from the server, we need a tool for further analysis. The recommended tools are Mat and visualVM.
I personally prefer to use visualVM for analysis, which can not only analyze offline dump files, but also integrate with IDEA, start applications through IDEA, and conduct real-time analysis of application CPU, memory and GC status (GC status, The Visual GC plug-in needs to be installed in visualVM). The tool is shown below (this is just for show, the data is not real) :
Of course, MAT is also a very useful tool, which can help us quickly locate memory leaks and facilitate our investigation. The display is as follows:
scenes
After analysis, we finally locate the memory leak problem caused by using Httpasyncclient.
Httpasyncclient is an HTTP toolkit provided by Apache. It mainly provides THE I/O non-blocking model of REACTOR and implements the function of sending HTTP requests asynchronously.
The following is a Demo to briefly explain the causes of specific memory leaks.
Httpasyncclient
1. The maven dependencies
<dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpasyncclient</artifactId> The < version > 4.1.3 < / version > < / dependency >Copy the code2. HttpAsyncClient client
public class HttpAsyncClient { private CloseableHttpAsyncClient httpclient; public HttpAsyncClient() { httpclient = HttpAsyncClients.createDefault(); httpclient.start(); } public void execute(HttpUriRequest request, FutureCallback<HttpResponse> callback){ httpclient.execute(request, callback); } public void close() throws IOException { httpclient.close(); }}Copy the codeMain logic:
The main logic of the Demo is to first create a cache list to hold the request data that needs to be sent.
The requests to be sent are then looped from the cache list to the Httpasyncclient client for sending.
The specific code is as follows:
Public class ReplayApplication {public static void main(String[] args) throws InterruptedException {// Create a playback client that has memory leaks ReplayWithProblem replay1 = new ReplayWithProblem(); List<HttpUriRequest> cache1 = replay1.loadMockRequest(10000); Replay1. start(cache1); }}Copy the codePlayback client implementation (memory leak) :
Take baidu playback as an example to create 10000 mock data into the cache list. When played back, the request is sent out every 100ms in a while loop. The specific code is as follows:
public class ReplayWithProblem { public List<HttpUriRequest> loadMockRequest(int n){ List<HttpUriRequest> cache = new ArrayList<HttpUriRequest>(n); for (int i = 0; i < n; i++) { HttpGet request = new HttpGet("http://www.baidu.com?a="+i); cache.add(request); } return cache; } public void start(List<HttpUriRequest> cache) throws InterruptedException { HttpAsyncClient httpClient = new HttpAsyncClient(); int i = 0; while (true){ final HttpUriRequest request = cache.get(i%cache.size()); httpClient.execute(request, new FutureCallback<HttpResponse>() { public void completed(final HttpResponse response) { System.out.println(request.getRequestLine() + "->" + response.getStatusLine()); } public void failed(final Exception ex) { System.out.println(request.getRequestLine() + "->" + ex); } public void cancelled() { System.out.println(request.getRequestLine() + " cancelled"); }}); i++; Thread.sleep(100); }}}Copy the codeMemory analysis:
Start the ReplayApplication application (VisualVM Launcher can be directly launched after the VisualVM Launcher is installed in IDEA) and observe it through VisualVM.
1. Startup status:
2. VisualVM memory object ratio before and after 3 minutes:
Description:
1 represents the first inner class in the object. So ReplayWithProblem$1: represents the callback class of FutureCallback in ReplayWithProblem.
From this, we can see that the FutureCallback class is constantly being created. Because every time you send an HTTP request asynchronously, you create a callback class to receive the result, it looks logical. No hurry. Let’s move on.
3. GC in visualVM:
As can be seen from the figure, the old of memory is increasing, which is not correct. In memory, only the HTTP request body of the cache list should be maintained. Now it is increasing continuously, indicating that objects are continuously entering the old area. Combined with the situation of the memory object above, indicating that the FutureCallback object is not timely collected.
However, after the HTTP callback, the reference relationship is gone and should be reclaimed in the next GC. Httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient: httpasyncclient The result is Full GC.
Core code analysis:
Code optimization
Finding the cause of the problem, let’s now optimize the code and verify our conclusions. Since callback objects are held in List Cache1, we can’t cache the request class. Instead, we can cache the basic data and generate it dynamically at use to ensure that callback objects are recycled in a timely manner.
The code is as follows:
public class ReplayApplication { public static void main(String[] args) throws InterruptedException { ReplayWithoutProblem replay2 = new ReplayWithoutProblem(); List<String> cache2 = replay2.loadMockRequest(10000); replay2.start(cache2); }}Copy the codepublic class ReplayWithoutProblem { public List<String> loadMockRequest(int n){ List<String> cache = new ArrayList<String>(n); for (int i = 0; i < n; i++) { cache.add("http://www.baidu.com?a="+i); } return cache; } public void start(List<String> cache) throws InterruptedException { HttpAsyncClient httpClient = new HttpAsyncClient(); int i = 0; while (true){ String url = cache.get(i%cache.size()); final HttpGet request = new HttpGet(url); httpClient.execute(request, new FutureCallback<HttpResponse>() { public void completed(final HttpResponse response) { System.out.println(request.getRequestLine() + "->" + response.getStatusLine()); } public void failed(final Exception ex) { System.out.println(request.getRequestLine() + "->" + ex); } public void cancelled() { System.out.println(request.getRequestLine() + " cancelled"); }}); i++; Thread.sleep(100); }}}Copy the coderesults
1. Startup status:
2. VisualVM memory object ratio before and after 3 minutes:
3. GC in visualVM:
From the graph, it can be proved that the conclusion we reached is correct. The callback class in Eden will be collected in time. There is no continuous growth in the old area. This time the memory leak was solved.
conclusion
The first time you troubleshoot a memory leak, it’s often a bit overwhelming. We need to have the right methods, the right tools, so that when we solve the problem, we can do it with ease. Of course, basic knowledge of JAVA memory is also essential, so you can locate the key to the problem, otherwise even if the tool tells you that there is a problem, you cannot locate the cause.
Finally, there is nothing wrong with the tool itself regarding the use of Httpasyncclient. But we have to understand its use scenarios, often produce problems, are caused by improper use. So, when using a tool, how much you know about it often determines the probability of bugs.
Author: jasonGeng88
Source:Github.com/jasonGeng88…
Welcome to follow my wechat public account “Code farming breakthrough”, share Python, Java, big data, machine learning, artificial intelligence and other technologies, pay attention to code farming technology improvement, career breakthrough, thinking transition, 200,000 + code farming growth charge first stop, accompany you have a dream to grow together