Note source: Silicon Valley JVM full tutorial, one million playback, the whole network peak (Song Hongkang detailed Java virtual machine)

Synchronous update: https://gitee.com/vectorx/NOT…

https://codechina.csdn.net/qq…

https://github.com/uxiahnan/N…

[TOC]

1. An overview of the article

1.1. Interview questions for Dachang

<mark> : </mark>

Three sides of Alipay: What is the JVM performance tuning done?

<mark> millet: </mark>

Have you done any JVM memory tuning?

From SQL, JVM, architecture, database four aspects of the optimization ideas

<mark> : </mark>

Compiler optimization for the JVM

What does JVM performance tuning do

Which JVM diagnostic tuning tools have been used?

Side 2: How is the JVM tuned? How much heap memory and stack space is appropriate

V3: Which JVM-related analysis tools have been used? What are the specific performance tuning steps

<mark> ali: </mark>

How do I tune the JVM? What are the methods?

How do you understand memory leaks? What are the conditions that can cause a memory leak? How to solve it?

<mark> byte jump: </mark>

Three sides: How to tune the JVM and how to tune the parameters?

<mark> pinduoduo: </mark>

From SQL, JVM, architecture, database four aspects of the optimization ideas

<mark>京东：</mark>

Which JVM diagnostic tuning tools have been used?

Why does a seckill system with hundreds of thousands of concurrent seckills occur so frequently?

How does a daily million-dollar trading system optimize the JVM?

How to monitor and locate online production system OOM?

How do highly concurrent systems optimize performance based on the G1 garbage collector?

1.2. Background notes

Problems in the production environment

  • How to handle memory overflow in production environment?
  • How much memory should the production environment allocate to the server?
  • How do I tune the performance of the garbage collector?
  • How to handle the high CPU load in a production environment?
  • How many threads should a production environment allocate to an application?
  • Without the log, how do you determine if a request executes a line of code?
  • How do you see the input return value of a method in real time without log?

Why do you tune it

  • Prevent OOM
  • To solve the OOM
  • Reduce the frequency of Full GC

Different stages of consideration

  • Before the launch
  • Project operation phase
  • OOM appears on the line

1.3. Overview of tuning

Basis of Monitoring

  • Run log
  • The exception stack
  • The GC log
  • Thread snapshot
  • Heap dump snapshot

The general direction of tuning

  • Write your code wisely
  • Full and reasonable use of hardware resources
  • JVM tuning is done properly

1.4. Steps for performance optimization

Step 1: Performance monitoring

  • The GC is frequent
  • CPU load is too high
  • OOM
  • Memory leaks
  • A deadlock
  • Long program response time

Step 2: Performance analysis

  • Print the GC log and analyze the exception information through GCviewer or http://gceasy.io
  • Flexible use of command line tools, jstack, jmap, jinfo, etc
  • Dump the heap file and use the memory analysis tool to analyze the file
  • Use Ali Arthas, JConsole, and JVisualVM to view the state of the JVM in real time
  • Jstack looks at stack information

Step 3: Performance tuning

  • Increase the memory appropriately and select the garbage collector according to the business context
  • Optimize code and control memory usage
  • Increase machine, disperse node pressure
  • Set the number of threads in the thread pool reasonably
  • Use middleware to improve program efficiency, such as caching, message queues, and so on
  • Other……

1.5. Performance evaluation/test indicators

Pause time (or response time)

The time elapsed between submitting a request and returning a response to that request is generally focused on the average response time. List of response times for common operations:

operation The response time
Open a site A few seconds
Database query for a record (indexed) More than ten milliseconds
A mechanical disk addresses location 4 ms
Read 1 MB of data sequentially from a mechanical disk Two milliseconds
Read 1M data sequentially from SSD disk 0.3 milliseconds
Replace a data read from remote distribution with Redis 0.5 milliseconds
Read 1M data from memory More than a dozen subtle
Java program native method calls A few subtle
The network transmits 2KB of data A subtle

In the garbage collection section:

  • Pause time: The time a program’s worker thread is paused while garbage collection is being performed.
  • -XX:MaxGCPauseMillis

throughput

  • A measure of the amount of work (requests) done per unit of time
  • In GC: Proportion of the total elapsed time of events running user code (total elapsed time: program elapsed time + memory reclaimed time)
  • Throughput is 1-1/(1+n), where -XX::GCTimeRatio=n

concurrency

  • The number of requests that are actually interacting with the server at any one time

Memory footprint

  • Size of memory occupied by the Java heap

The relationship between each other

Take freeway traffic, for example

  • Throughput: Data on the number of vehicles passing through freeway tollgates each day
  • Concurrence: The number of vehicles running on a highway
  • Response time: Speed

<hr/>

2. JVM monitoring and diagnostics tools – command line

Summary of 2.1.

Performance diagnosis is a problem that software engineers often face and solve in their daily work. In today’s world where user experience is Paramount, solving the performance problems of applications can bring great benefits.

Java as one of the most popular programming languages, its application performance diagnosis has been widely concerned by the industry. There are many factors that can cause performance problems in Java applications, such as thread control, disk reads and writes, database access, network I/O, garbage collection, and so on. To locate these problems, a good performance diagnostic tool is essential.

Experience 1: use data to explain problems, use knowledge to analyze problems, use tools to deal with problems.

Experience 2: No monitoring, no tuning!

Simple command-line tools

When we just come into contact with Java to learn, we must know the first two commands are javac, Java, so in addition, there are no other commands for us to use?

We went to the bin directory where the JDK was installed and found a number of helper tools. These assistive tools are used to capture different aspects and levels of information about the target JVM to help developers solve some of the complications of Java applications.

The official source address: http://hg.openjdk.java.net/jdk/jdk11/file/1ddf9a99e4ad/src/jdk.jcmd/share/classes/sun/tools

2.2. JPS: View running Java processes

JPS (Java Process Status) : Displays all the HotSpot virtual machine processes on the specified system (view the virtual machine Process information), which can be used to query for running virtual machine processes.

Note: For a local virtual machine process, the process’s local virtual machine ID is the same as the operating system’s process ID and is unique.

Basic usage syntax: JPS [options] [hostid]

We can also print additional information by appending parameters.

The options parameter

  • -q: Show only LVMID (Local Virtual Machine ID), which is the local virtual machine unique ID. Do not display the name of the main class, etc
  • -l: Output the full class name of the application’s main class or the full path to the JAR if the process is executing a JAR package
  • -m: Outputs the parameters passed to main() when the virtual machine process starts
  • -v: Lists the JVM parameters when the virtual machine process starts. For example: -xms20m -xmx50m is the JVM parameter specified by the launcher.

Note: The above parameters can be used in combination.

Note: If a Java process turns off the usePerfData parameter that is enabled by default (with the -XX: -usePerfData parameter), the JPS command (and jstat below) will not be able to detect the Java process.

Hostid parameters

The hostname registered in the RMI registry. If you want to remotely monitor Java programs on your host, you need to install JSTATD.

For network locations with more stringent security practices, it is possible to use a custom policy file to show access to a particular trusted host or network, although this technique is vulnerable to IP address fraud.

If the security issue cannot be handled using a custom policy file, the safest course of action is not to run the JSTATD server, but to use the jstat and JPS tools locally.

2.3. Jstat: View JVM statistics

Jstat (JVM Statistics Monitoring Tool) : A command-line Tool for Monitoring various health information about virtual machines. It can display runtime data such as class loading, memory, garbage collection, JIT compilation, etc. in local or remote virtual machine processes. On a server with no GUI graphical interface and only a plain text console environment, it will be the preferred tool for locating virtual machine performance problems at run time. It is often used to detect garbage collection problems and memory leaks.

The official document: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

Use basic grammar is: the jstat – < option > [-t] [-h < lines >] < vmid > [< interval > / < count >]]

Look at the command parameters: jstat-h or jstat-help

Where vmid is the process id number, which is the number you see after JPS, as follows:

The option parameter

The option can consist of the following values.

<mark> Class Loads Related: </mark>

  • -class: Display ClassLoader information: class load, unload number, total space, class load elapsed time, etc

<mark> Garbage Collection-related: </mark>

  • -gc: Displays heap information related to GC. Includes information about the capacity of Eden, two Survivor regions, old age, permanent generation, space used, total GC time, etc.
  • -gccapacity: The display is basically the same as -gc, but the output focuses on the maximum and minimum space used by each region of the Java heap.
  • -gcutil: The display is basically the same as -gc, but the output is focused on the percentage of the total used space.
  • -gccause: Same function as -gcutil, but with an extra print for the cause of the last or currently occurring GC.
  • -GCNEW: Display the status of the new generation GC
  • -gcnewCapacity: Display the same content as -gcnew, with the output focusing on the maximum and minimum space used
  • -geold: Show GC status in old age
  • -gcoldCapacity: Display the same content as -gcold, with the output focusing on the maximum and minimum space used
  • -GCPermCapacity: Shows the maximum and minimum space used by the permanent generation.

<mark> JIT-related: </mark>

  • -Compiler: Displays the JIT compiler compiled methods, elapsed time, etc
  • -PrintCompilation: Outputs methods that have been compiled by the JIT

jstat -class

jstat -compiler

jstat -printcompilation

jstat -gc

jstat -gccapacity

jstat -gcutil

jstat -gccause

jstat -gcnew

jstat -gcnewcapacity

jstat -gcold

jstat -gcoldcapacity

jstat -t

jstat -t -h

header Meaning (bytes)
EC The size of the Eden area
EU The size of the Eden zone that has been used
S0C Size of Survivor Zone 0
S1C Size of Survivor Zone 1
S0U Survivor 0 zone has been used in size
S1U Survivor 1 area has been used in size
MC The size of the meta space
MU The size of the meta space already used
OC The size of the old age
OU The size that has been used in old times
CCSC Compress the size of the class space
CCSU Compress the size of the class space already used
YGC Number of times Young GC from application startup to sampling
YGCT Young GC elapsed time from application startup to sampling (seconds)
FGC Number of Full GC between application startup and sampling
FGCT Full GC elapsed time (in seconds) from application startup to sampling
GCT Total time to GC from application startup to sampling time

Interval parameter: Specifies the period in milliseconds for the output of statistics. Namely: query interval

Count parameter: Specifies the total number of queries

-t parameter: You can add a TIMESTAMP column in front of the output information, showing the running time of the program. Unit: second

-h parameter: Can output a header information after the number of rows of data output in the periodic data output

Added: jstat can also be used to determine if a memory leak has occurred.

Step 1: In a long-running Java program, we can run the jstat command to retrieve multiple consecutive rows of performance data and get the minimum value of the OU column (the amount of old memory used) in those rows.

Step 2: Then, we repeat the above operation at long intervals to obtain multiple sets of OU minimum values. If these values are trending upwards, then the memory usage of the older Java program is increasing, which means that there is an increasing number of objects that cannot be recycled, and therefore a memory leak is likely.

2.4. jinfo: View and modify JVM configuration parameters in real time

JINFO (Configuration Info for Java) : View the Configuration parameters of the virtual machine. It can also be used to adjust the Configuration parameters of the virtual machine. In many cases, a Java application will not specify all of the Java Virtual Machine parameters. At this point, the developer may not know the default value for a particular Java Virtual Machine parameter. In this case, you might need to look up the document to get the default value for a parameter. This search process can be very difficult. But with the jinfo tool, developers can easily find the current values of Java virtual machine parameters.

The basic syntax is jinfo [options] pid

Description: The Java process ID must be added

options Option to show
no option Output all parameters and system properties
-flag name Output a parameter with the corresponding name
-flag [+-]name Only parameters with a manageable name that are on or off can be dynamically modified
-flag name=value Set the parameters of the corresponding name
-flags Output all parameters
-sysprops Output system properties

jinfo -sysprops

> jinfo -sysprops jboss.modules.system.pkgs = com.intellij.rt java.vendor = Oracle Corporation sun.java.launcher = SUN_STANDARD sun.management.compiler = HotSpot 64-Bit Tiered Compilers catalina.useNaming = true os.name = Windows 10 .

jinfo -flags

> jinfo -flags 25592 Non-default VM flags: -XX:CICompilerCount=4 -XX:InitialHeapSize=333447168 -XX:MaxHeapSize=5324668928 -XX:MaxNewSize=1774714880 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=111149056 -XX:OldSize=222298112 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC Command  line: - agentlib: JDWP = transport = dt_socket, address = 127.0.0.1:8040, suspend = y, server = n - Drebel. Base = C: \ Users \ Vector \. Equal - Drebel. Env. Ide. Plugin. Version = 2021.1.2 - Drebel. Env. Ide. Version = 2020.3.3 - Drebel. Env. Ide. The product = IU -Drebel.env.ide=intellij -Drebel.notification.url=http://localhost:7976 - agentpath: C: \ Users \ Vector \ AppData \ Roaming \ JetBrains \ IntelliJIdea2020.3 \ plugins \ jr - ide - the idea \ lib \ jrebel6 \ lib \ jrebel64 DLL  -Dmaven.home=D:\eclipse\env\maven - Didea. Modules. Paths. A file = C: \ Users \ Vector \ AppData \ Local \ JetBrains \ IntelliJIdea2020.3 \ Maven \ idea - projects - state - 596682 - c7. properties -Dclassworlds.conf=C:\Users\Vector\AppData\Local\Temp\idea-6755-mvn.conf -Dmaven.ext.class.path=D:\IDEA\plugins\maven\lib\maven-event-listener.jar -javaagent:D:\IDEA\plugins\java\lib\rt\debugger-agent.jar -Dfile.encoding=UTF-8

jinfo -flag

> jinfo -flag UseParallelGC 25592
-XX:+UseParallelGC

> jinfo -flag UseG1GC 25592
-XX:-UseG1GC

jinfo -flag name

> jinfo -flag UseParallelGC 25592
-XX:+UseParallelGC

> jinfo -flag UseG1GC 25592
-XX:-UseG1GC

jinfo -flag [+-]name

> jinfo -flag +PrintGCDetails 25592 > jinfo -flag PrintGCDetails 25592 -XX:+PrintGCDetails > jinfo -flag -PrintGCDetails  25592 > jinfo -flag PrintGCDetails 25592 -XX:-PrintGCDetails

Development:

  • Java-XX :+PrintFlagsInitial looks at the initial values for all JVM parameter launches

    [Global flags]
     intx ActiveProcessorCount                      = -1                                  {product}
    uintx AdaptiveSizeDecrementScaleFactor          = 4                                   {product}
    uintx AdaptiveSizeMajorGCDecayTimeScale         = 10                                  {product}
    uintx AdaptiveSizePausePolicy                   = 0                                   {product}
...

  • Java-XX :+PrintFlagsFinal looks at the final value of all JVM parameters

    [Global flags]
     intx ActiveProcessorCount                      = -1                                  {product}
...
     intx CICompilerCount                          := 4                                   {product}
    uintx InitialHeapSize                          := 333447168                           {product}
    uintx MaxHeapSize                              := 1029701632                          {product}
    uintx MaxNewSize                               := 1774714880                          {product}

  • Java-XX :+PrintCommandLineFlags View the names and values of detailed XX parameters that have been set by the user or JVM

    -XX:InitialHeapSize=332790016 -XX:MaxHeapSize=5324640256 -XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC

2.5.jmap: Export memory image files & memory usage

Jmap (JVM Memory Map) : It is used to retrieve dump files (heap dump snapshot files, binary files). It can also retrieve memory-related information about the target Java process, including usage of various areas of the Java heap, statistics of objects in the heap, class loading information, etc. Developers can enter the command “jmap-help” in the console to see how the jmap tool is used and some standard option configurations.

The official documentation: https://docs.oracle.com/en/java/javase/11/tools/jmap.html

The basic usage syntax is:

  • jmap [option] <pid>
  • jmap [option] <executable <core>
  • jmap [option] [server_id@] <remote server IP or hostname>
options role
-dump Generate a dump file (Java heap dump snapshot), -dump:live only holds live objects in the heap
-heap Output details of the entire heap space, including GC usage, heap configuration information, memory usage, and so on
-histo Outputs statistics on objects in the heap space, including classes, number of instances, and total capacity. -histo:live counts only living objects in the heap
-J <flag> Pass parameters to the JVM started by jmap
-finalizerinfo Objects displayed in the F-Queue waiting for the Finalizer thread to execute the Finalize method, Linux/Solaris only
-permstat ClassLoader is used as the statistical aperture to output the memory state information of the permanent generation. It is only valid for Linux/Solaris platforms
-F When the virtual machine process does not respond to the -dump option, the dump file is forced to be generated, only for Linux/Solaris platforms

Note: These parameters are somewhat different from the commands displayed as input under Linux, including the JDK version.

> jmap -dump:format=b,file=<filename.hprof> <pid>> jmap -dump:live,format=b,file=<filename.hprof> <pid>

Since jmap will access all objects in the heap, in order to ensure that the application thread does not interfere with the process, jmap needs to use a safety point mechanism to keep all threads in the heap without changing the state of the data. That is, the heap snapshot exported by jmap must be at the safe point location. This can lead to bias in the analysis results based on the heap snapshot.

For example, if some objects in the compiled machine code have a lifetime between two safe points, the :live option will not be able to detect these objects.

In addition, if a thread does not reach a safe point for a long time, jmap will wait forever. Unlike jstat, the garbage collector actively stores the summary data needed by jstat in a fixed location, while jstat simply reads it directly.

2.6. Jhat: The JDK comes with heap analysis tools

Jhat (JVM Heap Analysis Tool) : The jhat command provided by the Sun JDK is used in conjunction with the jmap command to analyze the Heap dump file (Heap dump snapshot) generated by jmap. Jhat has a built-in tiny HTTP/HTML server that generates the analysis results of the dump file and then allows the user to view the analysis results (analyzing the virtual machine dump snapshot information) in the browser.

With the jhat command, you start an HTTP service on port 7000, which is http://localhost:7000/.

Note: The jhat command has been removed in JDK9, JDK10, and VisualVM is officially recommended.

Basic applicable syntax: jhat <option> <dumpfile>

The option parameter role
| – stack false true Close | to turn on object allocation call stack tracing
– | refs false true Close | to turn on object reference tracing
-port port-number Set the jhat HTTP Server port number, default 7000
-exclude exclude-file Data members that need to be excluded when performing an object query
-baseline exclude-file Specify a base heap dump
-debug int Set Debug Level
-version After starting, the version information is displayed to exit
-J <flag> Pass in the boot parameter, such as -j-xmx512m

2.7. Jstack: Prints a snapshot of threads in the JVM

JStack (JVM Stack Trace) : Used to generate a thread snapshot (virtual machine Stack Trace) of the current moment of the virtual machine specified process. A thread snapshot is a collection of method stacks being executed by each thread of the specified process in the current virtual machine.

Generate the role of the thread snapshot: can be used to locate the cause of long pause threads, such as inter-thread deadlock, loop, request external resources caused by long wait, etc. These are common causes of long thread pauses. When a thread pauses, you can use jstack to show the stack of individual thread calls.

The official documentation: https://docs.oracle.com/en/java/javase/11/tools/jstack.html

In Thread Dump, look for the following states

  • Deadlock, Deadlock (focus)
  • Waiting on condition
  • Waiting on monitor entry
  • : Blocked
  • Runnable in execution
  • Suspended, Suspended
  • Object waiting, object.wait () or TIMED_WAITING
  • Stop, the Parked
The option parameter role
-F Forced to print the thread stack when a normally output request is not responded to
-l Displays additional information about the lock in addition to the stack
-m The C/C++ stack can be displayed if a native method is called

2.8. JCMD: Multi-function Command Line

Since JDK 1.7, a new command line tool, JCMD, has been added. It is a versatile tool that can be used to implement the functionality of all the previous commands except jstat. For example, use it to export the heap, memory usage, view Java processes, export thread information, perform GC, JVM runtime, etc.

The official documentation: https://docs.oracle.com/en/java/javase/11/tools/jcmd.html

Jcmd has most of the functionality of jmap, and it is recommended on the Oracle website to use JCMD instead of jmap

Jcmd-l: Lists all JVM processes

JCMD process number help: Lists all the specific commands supported for the specified process

JCMD process number specific command: Displays the data of the specified process instruction command

  • Thread.print can replace the jstack instruction
  • The gc.class_histogram can replace the -histo operation in jmap
  • Gc.heap_dump can replace the -dump operation in jmap
  • GC. Run allows you to see how the GC is performing
  • The vm. uptime can view the total execution time of the program and can replace the -t operation in the jstat directive
  • Vm.system_properties can replace the jinfo-sysprops process ID
  • Vm. flags can get configuration parameter information for the JVM

2.9. JSTATD: Remote Host Information Collection

While the previous directive dealt only with monitoring native Java applications, some of these tools also support monitoring of remote machines (e.g., JPS, jstat). To enable remote monitoring, you need to use the JSTATD tool in conjunction. The jstatd command is an RMI server-side program that acts as a proxy server to establish communication between the local computer and the remote monitoring tool. The JSTATD server passes the local Java application information to the remote machine.