A fault review meeting was just added today. The fault is very typical, and similar cases can also be found under Google. In the process of troubleshooting, a large number of tools were used, and more than one was found to be problematic. To sum up, (this article will not focus on the case itself, but will focus on personal troubleshooting ideas and the use of various tools for Java memory leakage).

Typical features of Java memory leaks

Symptom 1: The heap /Perm area keeps growing without any downward trend (the recycling rate cannot catch up with the growth rate), and finally FullGC is constantly triggered, or even crash(the following two figures are GC and Perm data of the same application, and the cause of GC triggering is confirmed to be insufficient Perm)
. It’s usually a late manifestation of phenomenon two.







Symptom 2: After each FullGC, the heap /Perm area grows slowly, and finally FullGC is constantly triggered, even crash(as shown in the diagram below)




Java memory leak scenario -PermGen Space

Cause: Perm is insufficient. Perm holds class, method-related objects, and runtime constant objects. If an application loads a large number of classes, the Perm section stores large amounts of information. Also, a large number of Intern String objects will cause the Perm area to grow. The size of this area is set by the -xx :MaxPermSize parameter.

Example :Groovy dynamically compiled class, xStream string.intern

Essential cause: Classloader.defineclass and java.lang.string.intern are called in a large number of inappropriate scenarios.

Solution:

Option 1(directly valid): Use btrace to output the stack information of calling classLoader.defineclass and trace the problem from the stack information (code as shown below). However, Btrace cannot trace JVM native methods (release 1.3.1 states that it can trace native methods, but attempts to trace native methods failed. If you know how to use it, please let me know, thanks).

! [_2016_09_29_12_59_38](http://ata2-img.cn-hangzhou.img-pub.aliyun-inc.com/c5bbdd783fb0606a9817c8296babce42)* ** Use JProfiler to trace the String. Intern stackCopy the code

Option 2: Dump heap, see which classes have exceptions (number),String information about objects referenced by Perm, etc. However, this method is not very intuitive, you can look at the String data to find suspicious problems, which is not as intuitive as scheme 1. JProfiler** is recommended for daily debugging.

! [_2016_09_29_1_46_59](http://ata2-img.cn-hangzhou.img-pub.aliyun-inc.com/8e5d32f29e6166695ea976ae8f9ba89d)Copy the code

1, Class loading: -xx :+TraceClassLoading and -xx :+TraceClassLoading to see which classes are loaded and which classes are unloaded. There is also a problem if a particular class keeps being loaded without being unloaded. (As shown below)

College! [_2016_09_29_1_06_46](http://ata2-img.cn-hangzhou.img-pub.aliyun-inc.com/b826a15b81570c02590fda7e598100cd)Copy the code

Option 4: Run jmap-permgen(jstat -gcutil can check the memory growth rate and area) to see the contents of the Perm area and determine if there is a problem (see figure below).







Java memory leak scenario -Java Heap Space

Cause: An object with a long life refers to an object with a short life that should be collected by GC as soon as possible, resulting in an object that can no longer be allocated enough space in the heap. Note: This is not necessarily a memory leak, as the heap itself is set too small.

Example: I have not personally encountered this kind of case, but have simulated this kind of case: refer to my article “JProfiler in Simple Terms”, and have studied other students’ cases: for example, too many parameters and frequent timeouts lead to memory leaks

Solution:

FullGC is triggered to dump live heap. Marks the number of objects in the heap, focusing on suspicious objects

FullGC is triggered to dump live heap. Marks the number of objects in the heap, focusing on suspicious objects

Compare the number and size of the same objects in Step 1 and Step 2, and identify suspicious objects one by one.

If step 3 still fails to identify the object in question, perform Steps 1 and 2 several more times. During this process, GC trigger time can be adjusted to simulate a real fault scenario 🙂

See if the heap size grows after GC, and if it doesn’t, and continues for a long period of time, it’s almost normal (see the practical section in the JProfiler article for details).

Note: There are many more scenarios for Java memory leaks (see the official Java memLeaks document), which will be added later as opportunities arise.