Series columns: SERIES of JVM columns

  • JVM performance Tuning (1) — JVM memory model and class loading operation mechanism

  • JVM performance Tuning (2) — garbage collection algorithms and garbage collectors

  • JVM performance tuning (3) – Analyze garbage collection policies through GC logs

  • JVM performance tuning (4) – Memory allocation and garbage collection tuning

The JDK tools

We’ll start with a look at some of the common tools that can help with performance tuning and troubleshooting, and then examine tuning with a specific example combined with the tools.

The JDK tools

The JDK comes with a number of performance monitoring tools that you can use to monitor systems and troubleshoot memory performance issues.

Find out the process using JPS

Java Virtual Machine Process Status Tool JPS (Java Virtual Machine Process Status Tool) is a JDK 1.5 provided a display of all current Java Process PID command, simple and practical. This is a great way to look at simple things about a current Java process on a Linux/Unix platform.

1. Check the Java process PID

Command: JPS -l

The left column is the PID of the Java process.

2. Output the parameters passed to the JVM

Command: JPS -vl

Use jstat to view VM statistics

Using the Jstat tool, you can monitor the real-time health of your Java application, see Eden, Survivor, older memory usage within the VM, as well as the number and time taken to execute YoungGC and FullGC. Through these indicators, we can easily analyze the current operating conditions of the system, judge the current system memory usage pressure and GC pressure, as well as whether the memory allocation is reasonable.

1. See what jstat does

Command: jstat -options

Command description:

  • -class: Displays information about ClassLoad.
  • -compiler: Displays JIT compilation information.
  • -gc: Displays heap information related to GC.
  • -gccapacity: Displays the capacity and usage of each generation.
  • -gcmetacapacity: Displays the size of Metaspace;
  • -gcnew: Display information of the new generation;
  • -gcnewcapacity: shows the size and usage of Cenozoic;
  • -gcold: Displays old age and permanent generation information;
  • -gcoldcapacity: shows the size of the old age;
  • -gcutil: Displays garbage collection information.
  • -gccause: Displays information about garbage collection (same as -gcutil), along with the cause of the last or currently occurring garbage collection;
  • -printcompilation: Prints jIT-compiled method information

Of these, jstat-gc is the most complete, most commonly used, and most useful, and is almost sufficient to analyze the JVM’s health.

2. Display ClassLoader information

Jstat -class

3. Check memory usage and GC

Jstat -gc [

[

]

Statistical description:

  • S0C: Capacity of the young generation To Survivor (in KB);
  • S1C: Capacity of the young generation From Survivor (in KB);
  • S0U: To Survivor currently used space in the young generation (in KB);
  • S1U: In the young generation From Survivor currently used space (in KB);
  • EC: Capacity of Eden in the young generation (unit: KB)
  • EU: Currently used space of Eden in the young generation (unit: KB);
  • OC: Capacity of the old age (in KB);
  • OU: Space currently used in the old age (unit: KB);
  • MC: Capacity of Metaspace (in KB);
  • MU: Metaspace Currently used space (in KB)
  • CCSC: Compresses the class space size
  • CCSU: Compressing the size of class space usage
  • YGC: Number of GC counts in the young generation from the start of the application to the time of sampling
  • YGCT: Time from application startup to GC in the young generation at sampling time (s);
  • FGC: Gc number of old generation (full GC) from application startup to sampling;
  • FGCT: Time (s) for old generation (full GC) GC from the start of the application to the time of sampling;
  • GCT: Total time taken by GC from application startup to sampling time (s)

4. Check garbage collection statistics

Jstat -gcutil
[

]

Statistical description:

  • S0: Survivor0 Zone occupation percentage
  • S1: Survivor1 Zone occupation percentage
  • E: Occupancy percentage of Eden area
  • O: Percentage occupied by older years
  • M: percentage of metadata usage
  • YGC: Recycling times of young generation
  • YGCT: Indicates the reclaim time of the young generation
  • FGC: Times of recycling in old years
  • FGCT: Recycling time in the old age
  • GCT: Total GC time

Use JMap to view object distribution

Using JMAP, you can view the initial configuration information of the heap memory, the usage of the heap memory, and output information about objects in the heap memory, including the objects generated and the number of objects.

1. Check heap memory

Jmap -heap

This command prints out some of the heap memory-related parameter Settings and the state of each region, and the jstat command is usually sufficient to view this information.

2. View the object distribution during system running

Run the jmap-histo [:live] command. If live is used, only live objects are counted

This command sorts objects in descending order by the amount of memory they occupy, putting the most memory-consuming objects at the top. Using this command, you can easily check the current memory usage of objects in the JVM and which objects are occupying the most memory space.

3. Generate a heap memory dump snapshot

Jmap -dump:format=b,file=

Jmap -dump:live,format=b,file=

Jmap-dump outputs all objects in the heap; Jmap-dump :live outputs all live objects in the heap. Jmap-dump :live triggers FullGC. Format =b is output in binary format; File is the file path. The format is the hrpof suffix.

This command generates a dump.hrpof file in the current directory, which is a binary format that cannot be opened directly and can be analyzed using tools such as MAT. This command takes a snapshot of all objects in the VM heap memory at that moment and places it in a file for later analysis.

Analyze the thread stack using JStack

Jstack is a thread stack analysis tool. The most common function is to use the jstack pid command to view the stack information of a thread. It is usually combined with top-hp PID or pidstat -p pid-t to view the status of a specific thread. It is often used to check for deadlocks, threads with high CPU usage, etc.

1. Jstack parameters

Parameter Description:

  • -l: long list. prints additional information about locks, such as belonging tojava.util.concurrent 的 ownable synchronizersList.
  • -F: whenjstack [-l] pidForce printing stack information when there is no response
  • -m: Prints Java andnative c/c++All stack information for the framework.
  • -h | -help: Prints help information

2. View thread stack information

Command: jstack > stack.log

This command dumps the program’s thread stack. For each thread stack, the thread ID, thread status (wait, sleep, running, etc.), and lock status can be viewed.

Information description:

  • pool-11-thread-6: Thread name
  • # 1920: Thread number
  • prio=5: Priority of the thread
  • os_prio=0: Indicates the thread priority of the system level
  • tid=0x00007f87e028c000: the thread ID
  • nid=0x6724: Id of the native threadprintf "%x\n" <pid>Command to convert the thread ID
  • waiting on condition [0x00007f87b97d2000]: Indicates the current status of the thread

Linux command line tools

The top command

The top command is one of the most commonly used commands in Linux. It displays real-time information about CPU usage, memory usage, and system load of the executing process. The upper part displays system statistics, and the lower part displays process usage statistics.

Look at the first line: it shows the CPU load

  • 23:22:23: indicates the current time
  • up 12 days, 12:18: refers to how long the machine has been running
  • 1 user: The current machine has one user in use
  • Load average: 0.19, 0.27, 0.30: indicates the CPU load in 1, 5, and 15 minutes.

The most important thing is to look at the load average. For example, if the machine has a 4-core CPU, then 0.19, 0.27 and 0.30 indicate that none of the 4-core CPU is used up, and the 4-core CPU is basically idle. If the CPU load is 1, one of the cores is busy. If the load is 4, all four cores are full. If it exceeds 4, it indicates that the 4-core CPU is not busy enough to handle the current task, and many processes may be waiting for the CPU to perform their tasks.

View system resource usage of a specific thread

The vmstat command

Vmstat is short for Virtual Meomory Statistics. It monitors the Virtual memory, process, and CPU activities of an OPERATING system (OS).

Format: vmstat [options] [< interval >] [< times >]

Field Description:

  • Procs:
    • R: number of processes waiting to run
    • B: Number of processes in a non-interrupted sleep state
  • Memory (Kb) :
    • SWPD: virtual memory usage
    • Free: indicates free memory
    • Buff: Amount of memory used for buffering
    • Cache: The size of memory used for caching
  • Swap:
    • Si: Number of swap pages swapped from disk to memory
    • So: number of swap pages from memory swap to disk
  • IO :(current Linux version block size is 1024bytes)
    • Bi: number of blocks sent to the block device
    • Bo: Number of blocks received from a block device
  • System:
    • In: Number of interrupts per second, including clock interrupts. 【 interrupt 】
    • Cs: number of context switches per second. “The count per second”
  • CPU (expressed in percentage) :
    • Us: User CPU usage (user time)
    • Sy: Kernel CPU system time
    • Id: idle time (including IO wait time), the idle time of the CPU. Expressed as a percentage.
    • Wa: Wait for I/O time

Judgment indicators:

  • If r is often greater than 4 and ID is often less than 40, the CPU is heavily loaded.
  • If bi and BO are not equal to 0 for a long time, the memory is insufficient.
  • If disk is not always equal to 0 and the queue in B is greater than 3, I/O performance is poor.
  • Cs is used to observe the frequency of system context switching during the running of Java programs. Too high indicates that the program creates too many threads causing frequent context switches.

Pidstat command

If you are monitoring context switches for an application, you can use the pidstat command to monitor context switches for a specific process.

Pidstat is a component of Sysstat and a powerful performance monitoring tool that can be installed with the command yum install Sysstat. The top and vmstat commands both monitor memory, CPU, and I/O usage of a process, while the pidstat command goes down to the thread level.

Pidstat [options] [< interval >] [< times >]

1. Common choices

  • -u: Displays the CPU usage of each process
  • -r: Displays the memory usage of each process
  • -d: Displays the I/O usage of each process
  • -p: Specifies the process NUMBER
  • -w: Displays the context switch for each process
  • -t: Displays statistics about threads in a process
  • -T { TASK | CHILD | ALL }
    • TASKRepresents a task that reports independently;CHILDThe keyword indicates the statistics of all threads in the reporting process.ALLRepresents reporting individual tasks and all threads under the task.
    • Note: Global statistics for tasks and child threads are independent of the pidstat option. These statistics do not correspond to the current statistical interval and are collected only when the child thread kills or completes.
  • -V: the version number
  • -h: Shows all activities on one line so that other programs can parse them easily.
  • -I: In an SMP environment, indicates the CPU usage/number of cores of a task
  • -l: Displays the command name and all parameters

2. Check the CPU usage of all processes

Pidstat, pidstat -u -p ALL

Field Description:

  • PID: the process ID
  • %usr: Indicates the percentage of CPU occupied by a process in user space
  • %system: Indicates the percentage of CPU occupied by a process in kernel space
  • %guest: Indicates the percentage of CPU usage by processes on the VM
  • %CPU: Indicates the percentage of CPU usage by processes
  • CPU: INDICATES the CPU number of the processing process
  • Command: Command corresponding to the current process

3. Display context switching for each process

Pidstat -w -p < interval > < number of times >

Field Description:

  • PID: the process id
  • Cswch/s: Number of active task context switches per second
  • Nvcswch/s: Number of passive task context switches per second
  • Command: the command name

4. Display statistics about threads in the process

Command: pidstat -p -t

Visualization tool

Here are a few visual analysis tools that you can use to dump GC log files and heap dumps. For online analysis, it is generally sufficient to use the ABOVE JDK command-line tools, which can be used to do some auxiliary analysis.

Jvisualvm – JVM monitoring

Jvisualvm is a monitoring tool provided by JDK, located in %JAVA_HOME%/bin/ jVisualvm. exe, double-click to run.

VisualVM provides a visual interface for viewing detailed information about Java technology-based applications running on the JVM. VisualVM can monitor threads, memory, CPU time of methods and objects in memory, objects that have been GC, and see the allocated stack in reverse (e.g., from which 100 Strings are allocated).

Some more detailed usage can refer to this article: zhuanlan.zhihu.com/p/30837957

1. Plug-in installation

VisualVM is based on the NetBeans platform development tools and has the ability to extend functionality through plug-ins. With plug-in extension support, VisualVM can:

  • Display VM processes and their configuration and environment information (JPS, jINFO)
  • Monitor the application’s processor, garbage collection, heap, method area, and thread information (jstat, JStack)
  • Dump and analyze heap dump snapshots (JMAP, JHAT)
  • Method-level program run performance analysis to find out which methods are called the most and run the longest
  • Offline program snapshot: Collects program runtime configuration, thread dump, memory dump and other information to create a snapshot, which can be sent to the developer for Bug feedback
  • The endless possibilities brought by other plug-ins

You can install the required plug-ins by opening the plug-in panel from the tools options:

2. Monitoring panel

Select the program you want to monitor on the left, and you can view CPU, heap, thread, and other fluctuations on the right, as well as perform manual GC and heap Dump operations directly from here.

3. Thread panel

You can see all the threads and their running status. Click the thread Dump button in the panel to view the thread stack at the time of the thread. (Thread stack can also be fetched via jStack)

4. GC panel

You can easily see the GC trend chart. (You can also use the jstat tool to monitor)

5. Analyze heap dump snapshots

The heap dump snapshot can be analyzed by opening the dump heap dump file with the load Snapshot button in the upper left corner.

GCViewer – Analyze GC logs offline

GCViewer allows you to view GC logs offline at github.com/chewiebug/G… . Run the [MVN clean install-dmaven.test.skip =true] command to package and compile the GCViewer. After compiling, you will see the JAR package in the target directory. Then run the JAR package on the command line to start GCViewer.

You can then open the GC log File through File to see the GC statistics and GC situation. With the tools, we can see information such as throughput, pause times, and GC frequency, which gives us a very intuitive picture of GC performance.

GCeasy – Analyze GC logs online

GCeasy is an online version of a very intuitive GC log analysis tool. You can compress the log files and upload them to the GCeasy website to see very clear GC log analysis results.

FastThread – Analyze thread stacks

After a thread stack is dumped by using the jstack command, you can use the FastThread online tool to analyze the thread stack information. You can intuitively see the number of threads, thread pool, thread status, whether there are deadlocks and other information.

MAT – Analyzes heap dump files

The heap dump we dump using jmap can be analyzed using MAT, which objects were created, and then the reference chain of the objects can be analyzed to find the problem. MAT download address: www.eclipse.org/mat/downloa…

Main functions of MAT:

  • Find out the cause of the memory leak
  • Find classes and jars that are repeatedly referenced
  • Analyze the use of collections
  • Analyze the class loader

Performance tuning tools

1. Address of online tool

Thread Dump analysis (FastThread) : fastThread. IO /

A thread Dump analysis (PerfMa) : thread.console.perfma.com/

Memory Dump analysis (PerfMa) : memory.console.perfma.com/

JVM parameter analysis (PerfMa) : opts.console.perfma.com/

GC Log Analysis (GCEasy) : www.gceasy.io/

GC Log Analysis (GCViewer) : github.com/chewiebug/G…

Java diagnosis, Arthas: alibaba. Making. IO/Arthas

MAT: www.eclipse.org/mat/downloa…

2. Tuning tool selection

There are many tools available for tuning. For online systems, the Jstat tool is sufficient to analyze the JVM’s health, and if GC logs are available, GCeasy can be used to quickly analyze the JVM’s health.

If CPU load is too high, you can use top + pidstat to find threads with high load, or you can use Jstat directly to see if FullGC is frequent.

Jstack can be used to dump the thread stack, and FastThread can also be used to analyze the thread stack, which threads are blocked waiting, etc.

When encountering OOM problems, use Jmap to dump heap dump snapshots and use MAT to analyze which large number of objects have been created.

JVM Monitoring Platform

If you don’t have a visual monitoring platform deployed, you can use these tools to analyze the JVM’s memory model, GC status, etc. You can run Jstat on your machine and have it write monitoring information to a file and check it daily.

The monitoring platform can see through a visual interface the memory changes in various areas of the JVM, GC times and GC time, and timely alarm when performance problems occur.

Visual monitoring platforms such as Zabbix, Ganglia, Open-Falcon, and Prometheus can be deployed, and the performance of the JVM can be graphically seen directly by plugging online systems into these platforms.

The next article will explain the use of tools for JVM performance tuning in more detail