preface

From the previous article on JVM garbage collection, we learned about the JVM’s specific garbage collection algorithm and several garbage collectors. Theory is the tool to guide practice. With the guidance of theory, knowledge and experience are the key basis for problem positioning, and data can provide us with the basis.

Here are a few questions we often encounter online:

  • Memory leak;
  • A process suddenlyCPUSoaring;
  • Thread deadlock;
  • The response is slow.

If you encounter this problem, there are various visual native tools available to support viewing in the offline environment. But once in the online environment, there is not as much support for native debugging tools, so how do we locate problems based on monitoring tools?

We typically locate problems based on data collection, which requires monitoring tools such as run logs, exception stacks, GC logs, thread snapshots, heap memory snapshots, etc. In order to solve the above problems, we commonly use JVM performance tuning monitoring tools: JPS, jstat, jStack, JMAP, JHAT, hprof, jinfo.

The body of the

If you want to see information about the thread stack in a Java process, choose the jstack command. If you want to see heap memory, you can use jMap exports and jHAT for analysis, including viewing class loading information, GC algorithms, object usage, and so on. You can use Jstat for statistical monitoring of the JVM, including looking at memory and GC by region, as well as hprof for CPU usage and heap memory usage. The following sections describe the usage of these tools in detail.

JVM common monitoring tools & directives

1. JPS process monitoring tool

JPS is the process used to view hotspot virtual machines that have access to it. If you do not specify hostid, the host JVM process is displayed by default. Otherwise, the JStatd service must be enabled on the host specified by hostid.

JPS can list LVMID, main class name, main function parameters, JVM parameters, jar names, etc.

The command format is as follows:

usage: jps [-help]
       jps [-q] [-mlvV] [<hostid>]

Definitions:
    <hostid>:      <hostname>[:<port>]
Copy the code

The meanings of the parameters are as follows:

  • – q: no outputThe class name,Jar The name of theAnd the incomingmainmethodsparameter;
  • – l: outputmainClass orJar 的 Fully qualified name;
  • -m: indicates the incoming outputmainmethodsparameter;
  • -v: indicates the incoming outputJVMThe parameters.

2. Jinfo configuration information viewing tool

The jinfo (JVM Configuration info) command is used to view and adjust vm running parameters in real time. The previous JPS -v command can only view the specified parameters. If you want to view the values of the parameters that are not displayed, you need to use the jinfo command.

Usage: jinfo [option] <pid> (to connect to running process) jinfo [option] <executable <core> (to connect to a core file) jinfo  [option] [server_id@]<remote server IP or hostname> (to connect to remote debug server)Copy the code

The meanings of the parameters are as follows:

  • Pid: localjvmService processID;
  • Executable core: the core file that prints the stack trace
  • Remote server IP/hostname: remotedebugThe service ofThe host name 或 IPAddress;
  • Server ID: RemotedebugThe service ofprocess ID.

The options are described as follows:

parameter Parameter meaning
flag Output the specifiedargsThe value of the parameter
flags Don’t needargsParameter, output allJVMThe value of the parameter
sysprops Output system properties, equivalent toSystem.getProperties()
  • View the runningjvmThe process ofExtension parameters.
$ jinfo -flags 31983 
Attaching to process ID 31983, please wait... Debugger attached successfully. Server Compiler detected. JVM version is 25.91-B14 non-default VM Flags: -XX:CICompilerCount=3 -XX:InitialHeapSize=20971520 -XX:MaxHeapFreeRatio=90 -XX:MaxHeapSize=20971520 -XX:MaxNewSize=2097152 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=2097152 -XX:OldSize=18874368 -XX:+PrintGC -XX:+PrintGCDetails -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseParallelGC Command line: - Xmx20m - Xms20m - Xmn2m - javaagent: / opt/idea - IU - 181.4668.68 / lib/idea_rt jar = 34989: / opt/idea - IU - 181.4668.68 / bin -Dfile.encoding=UTF-8Copy the code
  • View the runningjvmOwnership of a processParameter information.
$ jinfo 31983 Attaching to process ID 31983, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 25.91-B14 Java System Properties: Java.runtime. name = Java(TM) SE Runtime Environment java.vm.version = 25.91-b14 sun.boot.library.path = /opt/jdk1.8.0_91/jre/lib/amd64 java.vendor.url = http://java.oracle.com/ java.vm.vendor = Oracle Corporation path.separator = : file.encoding.pkg = sun.io java.vm.name = Java HotSpot(TM) 64-Bit Server VM sun.os.patch.level = unknown sun.java.launcher = SUN_STANDARD user.country = CN user.dir = /home/linchen/projects java.vm.specification.name = Java Java Virtual Machine Specification, the runtime version = 1.8.0 comes with _91 - b14. Java awt. Graphicsenv = sun. The awt. X11GraphicsEnvironment Os.arch = AMd64 java.endorsed.dirs = /opt/jdk1.8.0_91/jre/lib/endorsed java.io. Tmpdir = / TMP line. Separator = java.vm.specification.vendor = Oracle Corporation os.name = Linux sun.jnu.encoding = UTF-8 java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.specification.name = Java Platform API Specification Java.class. version = 52.0 sun.management.compiler = HotSpot 64-bit Tiered Compilers OS. version = 4.15.0-24-generic user.home = /home/linchen user.timezone = java.awt.printerjob = sun.print.PSPrinterJob file.encoding = UTF-8 Java. Specification. Version = 1.8. The user name = linchen. Java class. Path = / opt/jdk1.8.0 _91 / jre/lib/charsets. The jar: / opt/jdk1.8.0 _91 / jre/lib/deploy the jar: / opt/jdk1.8.0 _91 / jre/lib/ext/cldrdata jar: / opt / jdk1.8.0 _91 / jre/lib/ext/DNSNS jar: / opt/jdk1.8.0 _91 / jre/lib/ext/jaccess jar: / opt/jdk1.8.0 _91 / jre/lib/ext/JFXRT jar: / opt / Jdk1.8.0 _91 / jre/lib/ext/localedata jar: / opt/jdk1.8.0 _91 / jre/lib/ext/nashorn jar: / opt/jdk1.8.0 _91 / jre/lib/ext/sunec jar: / Opt/jdk1.8.0 _91 / jre/lib/ext/sunjce_provider jar: / opt/jdk1.8.0 _91 / jre/lib/ext/sunpkcs11 jar: / opt/jdk1.8.0 _91 / jre/lib/ext / Zipfs. Jar: / opt/jdk1.8.0 _91 / jre/lib/javaws jar: / opt/jdk1.8.0 _91 / jre/lib/jce jar: / opt/jdk1.8.0 _91 / jre/lib/JFR jar: / opt/JDK 1.8.0 comes with _91 / jre/lib/JFXSWT. Jar: / opt/jdk1.8.0 _91 / jre/lib/jsse jar: / opt/jdk1.8.0 _91 / jre/lib/management - agent. The jar: / opt/jdk1.8. 0 _91 / jre/lib/plugin jar: / opt/jdk1.8.0 _91 / jre/lib/resources. The jar: / opt/jdk1.8.0 _91 / jre/lib/rt. The jar: / home/linchen/IdeaProject S/core_java/target/classes: / home/linchen/m2 / repository/IO/netty/netty - all / 4.1.7. The Final/netty - all - 4.1.7. Final. Jar: / home/l Inchen/m2 / repository/junit/junit / / junit 4.12-4.12. The jar: / home/linchen/m2 / repository/org/hamcrest/hamcrest - core / 1.3 / hamcre St - core - 1.3. The jar: / home/linchen/m2 / repository/com/lmax/disruptor / 3.3.0 / disruptor - 3.3.0. Jar: / home/linchen/m2 / repository/c Om/rabbitmq/closer - client / 5.3.0 / closer - the client - 5.3.0. Jar: / home/linchen/m2 / repository/org/slf4j/slf4j - API / 1.7.25 / slf4j - API - 1. 7.25. The jar: / opt/idea - IU - 181.4668.68 / lib/idea_rt jar. Java vm. Specification. Sun version = 1.8. The arch. The data. The model = 64 Mand = sun.java.com com. Own. Learn the JVM. JinfoTest Java. Home = / opt/jdk1.8.0 _91 / jre user. Language = useful java.specification.vendor = Oracle Corporation awt.toolkit = sun.awt.X11.XToolkit java.vm.info = mixed mode java.version = 1.8.0 comes with _91 Java. Ext dirs = / opt/jdk1.8.0 _91 / jre/lib/ext: / usr/Java/packages/lib/ext sun. Boot. Class. Path = / opt/jdk1.8.0 _91 / jre/lib/resources. The jar: / opt/jdk1.8.0 _91 / jre/lib/rt. The jar: / opt/jdk1.8.0 _91 / jre/lib/sunrsasign jar: / opt/jdk1 8.0 _91 / jre/lib/jsse jar: / opt/jdk1.8.0 _91 / jre/lib/jce jar: / opt/jdk1.8.0 _91 / jre/lib/charsets. The jar: / opt/jdk1.8.0 _91 / jre/lib Separator = Oracle Corporation file.separator = / java.vendor.url.bug = /opt/jdk1.8.0_91/jre/classes java.vendor = Oracle Corporation file.separator = / java.vendor.url.bug = http://bugreport.sun.com/bugreport/ sun.io.unicode.encoding = UnicodeLittle sun.cpu.endian = little sun.desktop = gnome sun.cpu.isalist = VM Flags: Non-default VM flags: -XX:CICompilerCount=3 -XX:InitialHeapSize=20971520 -XX:MaxHeapFreeRatio=90 -XX:MaxHeapSize=20971520 -XX:MaxNewSize=2097152 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=2097152 -XX:OldSize=18874368 -XX:+PrintGC -XX:+PrintGCDetails -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseParallelGC Command line: - Xmx20m - Xms20m - Xmn2m - javaagent: / opt/idea - IU - 181.4668.68 / lib/idea_rt jar = 34989: / opt/idea - IU - 181.4668.68 / bin -Dfile.encoding=UTF-8Copy the code
  • View the runningjvmThe process ofEnvironment variable information.
$ jinfo -sysprops 31983 
Attaching to process ID 31983, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 25.91-B14 java.runtime.name = Java(TM) SE Runtime Environment java.vm.version = 25.91-b14 sun.boot.library.path = /opt/jdk1.8.0_91/jre/lib/amd64 java.vendor.url =  http://java.oracle.com/ java.vm.vendor = Oracle Corporation path.separator = : file.encoding.pkg = sun.io java.vm.name = Java HotSpot(TM) 64-Bit Server VM sun.os.patch.level = unknown sun.java.launcher = SUN_STANDARD user.country = CN user.dir = /home/linchen/projects java.vm.specification.name = Java Java Virtual Machine Specification, the runtime version = 1.8.0 comes with _91 - b14. Java awt. Graphicsenv = sun. The awt. X11GraphicsEnvironment Os.arch = AMd64 java.endorsed.dirs = /opt/jdk1.8.0_91/jre/lib/endorsed java.io. Tmpdir = / TMP line. Separator =Copy the code

2. Jstat Information statistics monitoring tool

Jstat is a command line tool used to identify various running information about virtual machines. It can display classloading, memory, garbage collection, JIT compilation, and other running data in local or remote virtual machine processes, and is the preferred tool for locating JVM performance online.

The Jstat tool provides the following JVM monitoring capabilities:

  1. Loading and unloading of classes;
  2. To viewThe new generation,The old generation 及 dimension(MetaSpace)capacityAnd usage;
  3. To viewThe new generation,The old generation 及 dimension(MetaSpace)Garbage Collection, including garbage collectionThe number of, which is occupied by garbage collectiontime;
  4. To viewThe new generation 中 EdenArea andSurviorIn the areacapacity 及 distributionAnd so on.

The command format is as follows:

Usage: jstat -help|-options
       jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Copy the code

The meanings of the parameters are as follows:

  • Option: parameter options.
    • -t: can be added to the printed columntimestampColumn to show how long the system is running.
    • -h: In the case of periodic data, you can output the header once after specifying how many lines to output.
  • Vmid: indicates the Virtual Machine ID of a processpid).
  • Lines: indicates the number of rows separated from the table header.
  • Interval: indicates the interval for each execution, in milliseconds.
  • Count: Specifies the number of times a record is output. By default, the record is always printed.

The options are described as follows:

  • Class: displayClass loading ClassLoadRelevant information of;
  • Compiler: displayJIT compileRelevant information of;
  • Gc: display andgcThe relevantHeap information;
  • Gccapacity: displays the capacity and usage of each generation.
  • Gcmetacapacity: displaydimension metaspaceThe size of the;
  • Gcnew: display information of the new generation;
  • Gcnewcapacity: displays the size and usage of Cenozoic generation;
  • Gcold: displays old age and permanent generation information;
  • Gcoldcapacity: displays the size of the old age.
  • Gcutil: Displays garbage collection information;
  • Gccause: displayThe garbage collectionInformation related to-gcutil) at the same timeThe last time 或 The currentGarbage collection is taking placeincentive;
  • Printcompilation: outputJIT compileMethod information;

2.1. The class

Displays and monitors class loads, unload numbers, total space, and elapsed time.

$jstat -class 8615 Loaded Bytes Unloaded Bytes Time 7271 13325.8 1 0.9 2.98Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
Loaded Number of classes that have been loaded
Bytes The number of bytes used to load the class
Unloaded The number of classes that have been unloaded
Bytes Number of bytes for the unload class
Time The time it takes to load and unload classes

2.2. The compiler

Displays real-time compilation (JIT) times and elapsed time of the virtual machine.

$ jstat -compiler 8615
Compiled   Failed  Invalid  Time     FailedType   FailedMethod
  3886        0       0     1.29          0
Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
Compiled Number of compilation tasks executed
Failed Number of failed compilation tasks
Invalid Number of compile task execution failures
Time Compilation tasks consume time
FailedType The type of the last compile failed task
FailedMethod The class and method of the last failed compile task

2.3. The gc

Display heap information related to garbage collection (GC) to see the number and time of GC.

$jstat -GC 8615 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 20480.0 10752.0 0.0 0.0 262128.0 130750.7 165376.0 24093.7 35456.0 33931.0 4992.0 4582.0 5 0.056 2 0.075 0.131Copy the code

For example, the following output is GC information, sampling interval is 250ms, sampling number is 4:

$jstat -gc 8615 250 4 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 20480.0 10752.0 0.0 0.0 $jstat -GC 8615 250 4 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT 20480.0 10752.0 0.0 0.0 262144.0 130750.7 165376.0 24093.7 35456.0 33931.0 4992.0 4582.0 5 0.056 2 0.075 0.131 20480.0 10752.0 0.0 0.0 262872.0 130750.7 165376.0 24093.7 35456.0 33931.0 4992.0 4582.0 5 0.056 2 0.075 0.131 20480.0 10752.0 0.0 262720.0 130750.7 165376.0 24093.7 35456.0 33931.0 4992.0 4582.0 5 0.056 2 0.075 0.131 20480.0 10752.0 0.0 262446.0 130750.7 165376.0 24093.7 35456.0 33931.0 4992.0 4582.0 5 0.056 2 0.075 0.131Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
S0C The first of the younger generationsurvivorThe capacity of the
S1C Second in the younger generationsurvivorThe capacity of the
S0U The first of the younger generationsurvivorCurrently used space
S1U Second in the younger generationsurvivorCurrently used space
EC In the young generationEdenThe capacity of the
EU In the young generationEdenCurrently used space
OC The capacity of the old age
OU The old age is currently in use
MC dimensionmetaspaceThe capacity of the
MU dimensionmetaspaceCurrently used space
YGC From application startup to sampling timeThe young generation 中 gc The number of
YGCT From application startup to sampling timeThe young generation 中 gc The time used
FGC From application startup to sampling timeThe old s 中 gc The number of
FGCT From application startup to sampling timeThe old s 中 gc The time used
GCT From application startup to sampling timegcUse of theThe total time

2.4. gccapacity

Displays the usage and usage of three generations of young, old, and metaspace objects in the VM memory.

$jstat -gcCapacity 8615 NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC 87040.0 1397760.0 372736.0 20480.0 10752.0 262144.0 175104.0 2796544.0 165376.0 165376.0 0.0 1079296.0 35456.0 0.0 1048576.0 4992.0 5 2Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
NGCMN The young generationInitialize the(Minimum) capacity
NGCMX The young generationThe maximum capacity
NGC The young generationCurrent capacity
S0C In the young generationThe first one survivorThe capacity of
S1C In the young generationThe second survivorThe capacity of
EC In the young generationEden(The garden of EdenThe capacity of)
OGCMN In the old sInitialize the(Minimum) capacity
OGCMX The old sThe maximum capacity
OGC The old sCurrent new generationThe capacity of the
OC The capacity of the old age
MCMN dimension 的 Initializing capacity
MCMX dimension 的 The maximum capacity
MC dimensionThe currentThe new generationThe capacity of the
CCSMN The minimumCompress the class space size
CCSMX The biggestCompress the class space size
CCSC The currentCompress the class space size
YGC From application startup to sampling timeThe young generationIn thegcThe number of
FGC From application startup to sampling timeThe old sIn thegcThe number of

2.5. gcmetacapacity

Displays information about objects in metaspace and their usage.

$jstat -gcmetacapacity 8615 MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC FGCT GCT 0.0 1079296.0 35456.0 0.0 1048576.0 4992.0 5 2, 0.075, 0.131Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
MCMN The minimumCapacity of metadata space
MCMX The biggestCapacity of metadata space
MC The currentCapacity of metadata space
CCSMN The minimum compressionSpace like capacity
CCSMX The largest compressionSpace like capacity
CCSC The currentCompress class space capacity
YGC From application startup to sampling timeThe young generation 中 gcThe number of
FGC From application startup to sampling timeThe old s 中 gcThe number of
FGCT From application startup to sampling timeThe old s gc The time used
GCT From application startup to sampling timegcUse of theThe total time

2.6. gcnew

Displays information about young generation objects, including two survivor zones and one Eden zone.

$jstat -gcnew 8615 S0C S1C S0U S1U TTv MTT DSS EC EU YGC YGCT 20480.0 10752.0 0.0 0.0 615 20480.0 262144.0 131406.0 5 0.056Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
S0C The first of the younger generationsurvivorThe capacity of the
S1C Second in the younger generationsurvivorThe capacity of the
S0U The first of the younger generationsurvivorCurrently used space
S1U Second in the younger generationsurvivorCurrently used space
TT Limit of holding times
MTT Maximum number of holding limits
DSS The desiredSurvival areaThe size of the
EC In the young generationEdenThe capacity of the
EU In the young generationEdenCurrently used space
YGC From application startup to sampling timeThe young generation 中 gc The number of
YGCT From application startup to sampling timeThe young generation 中 gc The time used

2.7. gcnewcapacity

View information about young objects and their usage.

$jstat -gcnewCapacity 8615 NGCMN NGCMX NGC S0CMX S0C S1CMX S1C ECMX EC YGC FGC 87040.0 1397760.0 372736.0 465920.0 20480.0 465920.0 10752.0 1396736.0 262144.0 52Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
NGCMN The size initialized (minimum) in the young generation
NGCMX Maximum capacity of young generation
NGC The current capacity of the young generation
S0CMX The first of the younger generationsurvivorMaximum capacity
S0C The first of the younger generationsurvivorThe capacity of the
S1CMX Second in the younger generationsurvivorMaximum capacity
S1C Second in the younger generationsurvivorThe capacity of the
ECMX In the young generationEdenMaximum capacity
EC In the young generationEdenThe capacity of the
YGC From application startup to sampling timeThe young generation 中 gcThe number of
FGC From application startup to sampling timeThe old s 中 gcThe number of

2.8. gcold

Displays information about older objects.

$jstat -gcold 8615 MC MU CCSC CCSU OC OU YGC FGC FGCT GCT 35456.0 33931.0 4992.0 4582.0 165376.0 24093.7 5 2 0.075 0.131Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
MC dimension(metaspaceThe capacity of)
MU dimension(metaspace) Currently used space
CCSC Compress the class space size
CCSU Compressed class spaceuseThe size of the
OC The old sThe capacity of the
OU The old sCurrently used space
YGC From application startup to sampling timeThe young generation 中 gcThe number of
FGC From application startup to sampling timeThe old s 中 gcThe number of
FGCT From application startup to sampling timeThe old s gc The time used
GCT From application startup to sampling timegcUse of theThe total time

2.9. gcoldcapacity

View information about old objects and their usage.

$jstat -gcoldCapacity 8615 OGCMN OGCMX OGC OC YGC FGC FGCT GCT 175104.0 2796544.0 165376.0 165376.0 5 2 0.075 0.131Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
OGCMN The old sThe size of the initialization (minimum) in
OGCMX The old sMaximum capacity
OGC The old sThe current generated capacity
OC The old sThe capacity of the
YGC From application startup to sampling timeThe young generation 中 gcThe number of times
FGC From application startup to sampling timeThe old s 中 gcThe number of times
FGCT From application startup to sampling timeThe old s 中 gcThe time used
GCT From application startup to sampling timegcUse of theThe total time

2.10. gcutil

Displays information about the garbage collection (GC) process, including the percentage of memory used, garbage collection time, and collection times.

$jstat -gcutil 8615 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 0.00 50.13 14.57 95.70 91.79 5 0.056 2 0.075 0.131Copy the code

The parameter list and meanings are as follows:

parameter Parameter meaning
S0 In the young generationThe first one survivor 区 Has been usedPercentage of the current capacity
S1 In the young generationThe second survivor 区 Has been usedPercentage of the current capacity
E In the young generationEden 区 Has been usedPercentage of the current capacity
O The old s 中 Has been usedPercentage of the current capacity
M dimension(metaspace)Has been usedPercentage of the current capacity
YGC From application startup to sampling timeThe young generation 中 gc The number of
YGCT From application startup to sampling timeThe young generation 中 gc The time used
FGC From application startup to sampling timeThe old s gc The number of
FGCT From application startup to sampling timeThe old s gc The time used
GCT From application startup to sampling timegcUse of theThe total time

3. Jmap Heap memory statistics tool

The jmap (JVM Memory Map) command is used to view heap Memory usage. It is generally used in combination with jHAT to generate heap dump files. Jmap can not only generate dump files, but also query finalize execution queue, Java heap and metaspace details such as current usage, which collector is currently in use and so on.

If you don’t use this command, you can also use – XX: + HeapDumpOnOutOfMemoryError parameters to make the virtual machine appears OOM, automatically generate the dump file.

The command format is as follows:

Usage:
    jmap [option] <pid>
        (to connect to running process)
    jmap [option] <executable <core>
        (to connect to a core file)
    jmap [option] [server_id@]<remote server IP or hostname>
        (to connect to remote debug server)
Copy the code

The meanings of the parameters are as follows:

  • Pid: localjvmService processID;
  • Executable core: the core file that prints the stack trace
  • Remote server IP/hostname: remotedebugThe service ofThe host name 或 IPAddress;
  • Server ID: RemotedebugThe service ofprocess ID.

The options are described as follows:

parameter Parameter meaning
heap According toThe heapSummary information in
histo According toThe heapStatistics about objects in
histo[:live] Display onlyThe heap 中 Living objectsStatistics of
clstats According toClass loadingStatistics of
finalizerinfo Displayed in theF-Queue The queueWaiting for theFinalizerThreads executefinalizerMethod object
dump Example Export a memory dump snapshot

Note: Dump memory snapshot analysis basically includes histo, CLSTATS, FinalizerInfo, etc.

3.1. heap

Displays summary information in the heap. This includes heap memory usage, GC algorithms being used, heap configuration parameters, and heap memory usage in each generation. You can use this to determine current memory usage and garbage collection.

$ jmap -heap 11368 Attaching to process ID 11368, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 25.101-B13 using ththread -local object allocation. Parallel GC with 2 thread(s) Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 2684354560 (2560.0MB) NewSize = 1073741824 (1024.0MB) MaxNewSize = 1073741824 (1024.0MB) OldSize = 1610612736 (1536.0MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 Heap Usage: PS Young Generation Eden Space: Capacity = 852492288 (813.0MB) Used = 420427144 (400.95056915283203MB) Free = 432065144 (412.04943084716797MB) 49.31741317993014%, informs the From Space: Capacity = 113770496 (108.5MB) Used = 2299712 (2.19317626953125MB) Free = 111470784 (106.30682373046875MB) 2.021360617079493%, informs the To Space: Capacity = 107479040 (102.5MB) Used = 0 (0.0MB) Free = 107479040 (102.5MB) 0.0% Used PS Old Generation Capacity = 1610612736 (1536.0MB) Used = 50883368 (48.52616119383847656 MB) Free = 1559729368 (1487.4738388061523MB) 3.1592552860577903% used 27595 Interned Strings evicted 3138384 bytesCopy the code

The heap Configuration parameter list is described here:

parameter Corresponding startup parameters Parameter meaning
MinHeapFreeRatio -XX:MinHeapFreeRatio Minimum JVM heap idle ratio (Default 40)
MaxHeapFreeRatio -XX:MaxHeapFreeRatio Maximum JVM heap idle ratio (Default 70)
MaxHeapSize XX:Xmx Maximum size of JVM heap
NewSize -XX:NewSize The default (initialized) size of the JVM heap generation
MaxNewSize -XX:MaxNewSize Maximum size of the JVM heap generation
OldSize -XX:OldSize Default (initialized) size for JVM heap age
NewRatio -XX:NewRatio The ratio of the size of the JVM heap to that of the new generation
SurvivorRatio -XX:SurvivorRatio Size ratio of Eden region to Survivor region in JVM heap young generation
MetaspaceSize -XX:MetaspaceSize JVM metaspace initialization size
MaxMetaspaceSize -XX:MaxMetaspaceSize Maximum size of JVM metaspace
CompressedClass SpaceSize -XX:CompressedClass SpaceSize JVM class pointer compression space, default is 1G
G1HeapRegionSize -XX:G1HeapRegionSize Size of a Region when G1 garbage collector is used. The value ranges from 1 MB to 32 MB

3.2. histo

Print object statistics for the heap, including object instances, memory size, and so on. Because full GC is performed before Histo: Live, only live objects are counted if live is included. The heap without live is larger than the heap with live.

$ jmap -histo:live 12498
 num     #instances #bytes class name
----------------------------------------------
   1:         50358        7890344  [C
   2:         22887        2014056  java.lang.reflect.Method
   3:          3151        1485512  [B
   4:         49267        1182408  java.lang.String
   5:          7836         871384  java.lang.Class
   6:         24149         772768  java.util.concurrent.ConcurrentHashMap$Node
   7:         20785         482256  [Ljava.lang.Class;
   8:          8357         435248  [Ljava.lang.Object;
   9:         10035         401400  java.util.LinkedHashMap$Entry
  10:          4803         369488  [Ljava.util.HashMap$Node;
  11:         10763         344416  java.util.HashMap$Node
  12:          5205         291480  java.util.LinkedHashMap
  13:          3055         219960  java.lang.reflect.Field
  14:           120         193408  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  15:         11224         179584  java.lang.Object
  16:          1988         146152  [Ljava.lang.reflect.Method;
  17:          3036         145728  org.aspectj.weaver.reflect.ShadowMatchImpl
  18:          1771         141680  java.lang.reflect.Constructor
  19:          4903         117672  org.springframework.core.MethodClassKey
  20:          3263         104416  java.lang.ref.WeakReference
  21:          2507         100280  java.lang.ref.SoftReference
  22:          2523          97600  [I
  23:          3036          97152  org.aspectj.weaver.patterns.ExposedState
  24:          2072          95280  [Ljava.lang.String;
  25:           954          91584  org.springframework.beans.GenericTypeAwarePropertyDescriptor
  26:          1633          91448  java.lang.Class$ReflectionData
  27:          3142          90520  [Z
  28:          1671          80208  java.util.HashMap
  29:          3244          77856  java.util.ArrayList
  30:          3037          72880  [Lorg.aspectj.weaver.ast.Var;
  31:          1809          72360  java.util.WeakHashMap$Entry
  32:          1967          62944  java.util.LinkedList
Copy the code

Where, class name is the object type. The mapping between the object abbreviated type and the real type is described as follows:

Object abbreviation type Object real type
B byte
C char
D double
F float
I int
J long
Z boolean
[ Arrays, such as [I for int[]
[L + the name of the class Other objects

3.3. The dump

Dump is used to export memory dump snapshots. The most common way is to dump the process memory usage to a file using JMAP, and then analyze it using JHAT. The jmap dump command is in the following format:

jmap -dump:format=b,file=dumpFileName
Copy the code

The meanings of the parameters are as follows:

parameter Parameter meaning
dump Heap to file
format Specify the output format
live Indicate a living object
file Specify file name
  • throughjmapexportMemory snapshot, the file is named asdump.dat:
jmap -dump:format=b,file=dump.dat 12498
Dumping heap to /Users/XXX/dump.dat ...
Heap dump file created
Copy the code

The exported dump files can be viewed for analysis using tools such as MAT, VisualVM, and JHAT, as described below.

4. Jhat Heap Snapshot analysis tool

The jHAT (JVM Heap Analysis Tool) command is usually used in conjunction with Jmap to analyze jMAP-generated dumps. Jhat has a mini-HTTP /HTML server built in that generates dump analysis results that can be viewed in a browser.

Note: Analysis is usually not performed directly on the server because using JHAT is a time consuming and hardware resource-consuming process. Instead, the dump file generated by the server is copied locally or to another machine for analysis.

The command format is as follows:

Usage:  jhat [-stack <bool>] [-refs <bool>] [-port <port>] [-baseline <file>] [-debug <int>] [-version] [-h|-help] <file>

	-J<flag>          Pass <flag> directly to the runtime system. For
			  example, -J-mx512m to use a maximum heap size of 512MB
	-stack false:     Turn off tracking object allocation call stack.
	-refs false:      Turn off tracking of references to objects
	-port <port>:     Set the port for the HTTP server.  Defaults to 7000
	-exclude <file>:  Specify a file that lists data members that should
			  be excluded from the reachableFrom query.
	-baseline <file>: Specify a baseline object dump.  Objects in
			  both heap dumps with the same ID and same class will
			  be marked as not being "new".
	-debug <int>:     Set debug level.
			    0:  No debug output
			    1:  Debug hprof file parsing
			    2:  Debug hprof file parsing, no server
	-version          Report version number
	-h|-help          Print this help and exit
	<file>            The file to read
Copy the code

The meanings of the parameters are as follows:

parameter Parameter Value Default value Parameter meaning
stack true Shut downObject allocation call stack trace. If the allocation location information is not available in the heap dump. You must set this flag to false.
refs true Shut downObject reference tracing. By default, the pointer returned is an object that points to another specific object. Such asbacklinks 或 Input reference, will count/count all objects in the heap
port 7000 Example Set the port number of the JHAT HTTP Server
exclude — Specifies the list of data members to exclude when querying objects
baseline — To specify aBase heap dump. When two Heap dumps have the same object ID, they are marked as not new and the other objects are marked as new. Useful when comparing two different heap dumps
debug 0 Set the debug level. 0 indicates that no debugging information is output. A larger value indicates more detailed debugging information
version — After startup, only the version information is displayed and exit
J — The jhat command actually starts a JVM to execute, with -j passing in some when starting the JVMLaunch parameters. For example, -j -xmx512m specifies that the maximum heap memory used by the Java virtual machine running jhat is 512MB.
  • As mentioned earlier, throughjmap dumpThe files that come out can be usedMAT,VisualVMAnd so on. Here we usejhatTo view:
$ jhat -port 7000 dump.dat
Reading from dump.dat...
Dump file created Sun Aug 12 12:15:02 CST 2018
Snapshot read, resolving... Resolving 1788693 objects... Chasing references, expect 357 dots.................................................................................................................... . . . Eliminating duplicate references.............................................................................................................. . . . Snapshot resolved. Started HTTP server on port 7000 Server is ready.Copy the code
  • Open your browser and typehttp://localhost:7000To see thejhatAnalysis report page:

  • Examples of concrete objects in the project module can be viewed by package name:

In addition, the final page of report analysis also provides some extended queries:

  • Display allRootThe collection;
  • Show allclassThe currentNumber of object instances(including,JVMPlatform related);
  • Show allclassThe currentNumber of object instances(excludingJVMPlatform related);
  • According toHeap memoryOf the instance object inStatistical histogram(and direct usejmapThere is no difference.
  • According tofinalizerThe virtual machineSecondary recoveryAbstract of information;
  • performjhatTo provide theObject Query language(OQL) gets instance information about the specified object.

Note: JHAT supports filtering or querying objects in the heap based on certain criteria. An OQL statement can be executed in the HTML page of JHAT to query eligible objects. OQL ` specific syntax can be directly to http://localhost:7000/oqlhelp.

In specific troubleshooting, you need to check whether a large number of objects that should be reclaimed are referenced all the time, or whether objects that occupy too much memory cannot be reclaimed.

5. Jstack stack tracing tool

Jstack is used to generate a thread snapshot of the Java VIRTUAL machine at the current time. A thread snapshot is a collection of method stacks being executed by each thread in the current Java VIRTUAL machine. The main purpose of creating thread snapshots is to locate the causes of long pauses in threads, such as deadlocks between threads, dead loops, long waits for external resources, and so on.

When a thread pauses, you can look at the call stack of each thread through JStack to see what the unresponsive thread is doing in the background or waiting for resources. If a Java program crashes and generates a core file, the JStack tool can obtain information about the Java Stack and Native Stack from the core file to locate the cause of the program crash.

The command format is as follows:

Usage:
    jstack [-l] <pid>
        (to connect to running process)
    jstack -F [-m] [-l] <pid>
        (to connect to a hung process)
    jstack [-m] [-l] <executable> <core>
        (to connect to a core file)
    jstack [-m] [-l] [server_id@]<remote server IP or hostname>
        (to connect to a remote debug server)
Copy the code

The meanings of the parameters are as follows:

  • Pid: localjvmService processID;
  • Executable core: the core file that prints the stack trace
  • Remote server IP/hostname: remotedebugThe service ofThe host name 或 IPAddress;
  • Server ID: RemotedebugThe service ofprocess ID.

The options are described as follows:

parameter Parameter meaning
F When normal output requestDon’t be response, force outputThread stack
l In addition to the stack, displays aboutAdditional information about locks
m If the call toLocal methodIf so, you can displayC/C++The stack

Note: In practice, one dump is often insufficient to confirm the problem. You are advised to generate three dumps. If each dump points to the same problem, you can determine the typicality of the problem.

5.1. System thread status

In the dump file, the thread states of interest are:

  1. Deadlock: A Deadlock (focus)
  2. Running: Runnable
  3. Waiting on condition
  4. Waiting on monitor entry
  5. Suspended: Suspended
  6. Object waiting: object.wait () or TIMED_WAITING
  7. Blocked: Blocked
  8. Stop: the Parked

The specific meanings are as follows:

(a). Deadlock

Deadlocked threads generally refer to the situation that resources are occupied by each other during the invocation of multiple threads and cannot be released.

(b). Runnable

Generally, the thread is in the executing state, occupying resources, and processing a request. You might be passing SQL to the database for execution, you might be operating on a file, you might be converting data types, etc.

(c). Waiting on condition

This state is in the thread waiting for a condition to occur. The reason can be analyzed in combination with StackTrace. The thread is in this wait state and reactivates to read and process the data once it is ready to read it.

The thread is waiting for a resource or a condition to occur. The specific reason needs to be analyzed in combination with the following stack information.

  • If the stack information is explicitly application code, it indicates that the thread is waiting for resources. Generally, a thread enters the wait state when a resource is read in bulk and the resource is locked.

  • If you find that a large number of threads are in this state, and the stack message indicates that they are waiting for network access, this is because the thread cannot execute because of network congestion, which may be a sign of a network bottleneck:

    • The network is very busy, almost consumes all the bandwidth, there is still a lot of data waiting for network read and write;
    • The network may be idle, but packets cannot arrive due to routing or firewall problems.

  • Another common situation is that the thread is sleeping, waiting for the time to sleep to wake up.

(d). Locked

Thread blocking refers to a thread that waits for resources for a long time but fails to obtain them, which is identified as blocked by the thread manager of the container and can be understood as a thread waiting for resources to timeout.

(e).waiting for monitor entry and in object.wait ()

Monitor is Java’s primary means of achieving mutual exclusion and collaboration between threads. It can be thought of as a lock on an object or Class. Each object has a Monitor.

5.1. Deadlock example

Here is an example of a deadlock in which two static instances of a reentrant lock are defined in IntLock and two threads are declared to compete for resources on both locks in the main method.

public class DeadLockRunner {
    public static void main(String[] args) {
        IntLock r1 = new IntLock(1);
        IntLock r2 = new IntLock(2);
        Thread t1 = new Thread(r1);
        Thread t2 = new Thread(r2);
        t1.start();
        t2.start();
    }

    public static class IntLock implements Runnable {
        private static ReentrantLock lock1 = new ReentrantLock();
        private static ReentrantLock lock2 = new ReentrantLock();
        private int lock;

        public IntLock(int lock) {
            this.lock = lock;
        }

        @Override
        public void run(a) {
            try {
                if (lock == 1) {
                    lock1.lock();

                    try {
                        Thread.sleep(500);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }

                    lock2.lock();
                } else {
                    lock2.lock();

                    try {
                        Thread.sleep(500);
                    } catch(InterruptedException e) { e.printStackTrace(); } lock1.lock(); }}finally {
                if (lock1.isHeldByCurrentThread()) {
                    lock1.unlock();
                }
                if (lock2.isHeldByCurrentThread()) {
                    lock2.unlock();
                }
            }
        }
    }
}
Copy the code

5.2. Dump Log Analysis

Start the DeadLockRunner’s main() method, use JPS to look at the ID of the blocked JVM process, and then use JStack to look at the thread stack information. You can see two threads competing for resources and deadlock.

$ jstack -lJava HotSpot(TM) 64-Bit Server VM (25.172-B11 mixed mode): // omit... Found one Java-level deadlock: ============================="Thread-1":
  waiting for ownable synchronizer 0x000000076ad61180, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "Thread-0"
"Thread-0":
  waiting for ownable synchronizer 0x000000076ad611b0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "Thread-1"

Java stack information for the threads listed above:
===================================================
"Thread-1":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076ad61180> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at io.ostenant.deadlock.DeadLockRunner$IntLock.run(DeadLockRunner.java:47)
	at java.lang.Thread.run(Thread.java:748)
"Thread-0":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076ad611b0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at io.ostenant.deadlock.DeadLockRunner$IntLock.run(DeadLockRunner.java:37)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.
Copy the code

reference

Zhou, zhiming, understanding the Java Virtual Machine in depth: Advanced JVM features and Best Practices, China Machine Press

Welcome to pay attention to the technical public number: Zero one Technology Stack

This account will continue to share learning materials and articles on back-end technologies, including virtual machine basics, multithreaded programming, high-performance frameworks, asynchronous, caching and messaging middleware, distributed and microservices, architecture learning and progression.