Online JVM tuning

The problem found

The memory usage of an application is slowly increasing. The memory usage of the system alarm has exceeded 80% and is increasing.

Specifically, RES is increasing.

You can run the following command to check the memory status on the target host:

ps -p 1 -o pcpu,rss,size,vsize
Copy the code

RSS is Resident Set Size, which indicates the amount of memory allocated by the process.

SIZE: Address space used by the process. If the process maps 100M memory, the address space of the process is reported as 100M memory. In fact, this size is not the amount of memory a program actually uses.

VSZ represents the virtual memory allocated by the process. This includes all the memory that the process can access, including what goes into the swap partition, and the memory occupied by the shared library.

Problem analysis

From the superficial analysis, the suspicion may be that memory is leaking, and since memory is growing slowly, not rapidly and continuously, the suspicion tends to be that it is not leaking heap memory.

From the monitoring data, the ratio of heap and non-heap memory is not very high. Then you go inside the container and look at the memory of the Java process.

The JDK version we are using is AdoptOpenJDK 11.0.8, which is not the official JDK of Oracle, but the community version of openJDK.

The default GC is supposed to be G1 since java9, but when I look at the production JDK, it looks like this:

java -XX:+PrintCommandLineFlags
Copy the code

-XX:InitialHeapSize=41943040 -XX:MaxHeapSize=671088640 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseSerialGC 
Copy the code

I’m using SerialGC

Unbelievably, I thought of looking at the GC parameters in a Java process:

jhsdb jmap --heap --pid 1
Copy the code

Attaching to process ID 1, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 11.0.8+10 using Ththread -local Object allocation.  Mark Sweep Compact GCCopy the code

It was confirmed. Mark Sweep Compact GC, marker-cleaner-compression algorithm. SerialGC (-xx :+UseSerialGC) algorithm used in older eras.

To solve

To be honest, I always thought that this version of our JDK was the G1 garbage collector by default, and that we could do without it. So I looked at the memory allocation, and I gave it 2 gigabytes, and there was a problem.

The question is which GC should be used, not that it is right to choose G1 without any brains above JDK9

The above is from the third edition of Understanding the Java Virtual Machine by Zhiming Zhou

According to our actual situation, memory is about 2G, so I prefer CMS.

The JVM options parameters are as follows:

-Xms2048m -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError
-XX:NativeMemoryTracking=detail
-XX:+UseConcMarkSweepGC 
-XX:MetaspaceSize=256M 
-XX:MaxMetaspaceSize=256M
-XX:ReservedCodeCacheSize=128m 
-XX:InitialCodeCacheSize=128m
-Xss512k
-XX:+AlwaysPreTouch
Copy the code

Before explaining some of the important parameters, let’s examine the memory composition of a Java application:

Total memory = Heap + Code Cache + Metaspace + Symbol tables +
               Other JVM structures + Thread stacks +
               Direct buffers + Mapped files +
               Native Libraries + Malloc overhead + ...

Copy the code

As you can see, there are roughly two parts: heap and non-heap.

You can also run the following command to check the memory status of a specific Java process:

jcmd 1 VM.native_memory
Copy the code

The results are similar:

Native Memory Tracking: Total: reserved=1847158KB, committed=1561194KB - Java Heap (reserved=1048576KB, committed=1048576KB) (mmap: reserved=1048576KB, committed=1048576KB) - Class (reserved=405345KB, committed=170849KB) (classes #28273) ( instance classes #26587, array classes #1686) (malloc=8033KB #97026) (mmap: reserved=397312KB, committed=162816KB) ( Metadata: ) ( reserved=143360KB, Committed =142336KB) (used=138699KB) (free=3638KB) (waste=0KB =0.00%) (Class space:) (reserved=253952KB, Committed =20480KB) (Used =18340KB) (Free =2140KB) (waste=0KB =0.00%) - Thread (reserved= 68438KB, committed=17205KB) (thread #126) (stack: reserved=68072KB, committed=16604KB) (malloc=454KB #758) (arena=147KB #251) - Code (reserved=136634KB, committed=136634KB) (malloc=4538KB #15693) (mmap: reserved=132096KB, committed=132096KB) - GC (reserved=7511KB, committed=7511KB) (malloc=3575KB #5347) (mmap: reserved=3936KB, committed=3936KB) - Compiler (reserved=1027KB, committed=1027KB) (malloc=894KB #1383) (arena=133KB #5) - Internal (reserved=18773KB, committed=18773KB) (malloc=18741KB #7974) (mmap: reserved=32KB, committed=32KB) - Other (reserved=66199KB, committed=66199KB) (malloc=66199KB #85) - Symbol (reserved=32192KB, committed=32192KB) (malloc=28412KB #365132) (arena=3780KB #1) - Native Memory Tracking (reserved=8657KB, committed=8657KB) (malloc=544KB #7707) (tracking overhead=8112KB) - Arena Chunk (reserved=2036KB, committed=2036KB) (malloc=2036KB) - Logging (reserved=4KB, committed=4KB) (malloc=4KB #191) - Arguments (reserved=18KB, committed=18KB) (malloc=18KB #495) - Module (reserved=2706KB, committed=2706KB) (malloc=2706KB #10716) - Synchronizer (reserved=738KB, committed=738KB) (malloc=738KB #6245) - Safepoint (reserved=8KB, committed=8KB) (mmap: reserved=8KB, committed=8KB) - Unknown (reserved=48060KB, committed=48060KB) (mmap: reserved=48060KB, committed=48060KB)Copy the code

-XX:NativeMemoryTracking=detail

The above result is displayed because I added this parameter

-XX:MetaspaceSize

-xx :MetaspaceSize=256M -xx :MaxMetaspaceSize=256M, as shown in the following figure:

• If the open – XX: + UseCompressedOops and – XX: + UseCompressedClassesPointers (the default is open), UseCompressedOops uses a 32-bit offset to represent references to Java Objects, While UseCompressedClassPointers use the 32 – bit offset to represent the class in the process of the 64 – bit pointer. The size of this space can be set using CompressedClassSpaceSize
• If pointer compression is enabled, the CompressedClassSpace is assigned to MaxMetaspaceSize, MaxMetaspaceSize=Compressed Class Space Size + Metaspace area (excluding the Compressed Class Space) Size

So we fixed the size of compressed class space in Metaspace, because the default is 1 GB

-Xss512k

Stack size is controlled by -xss. The default value is 1M per thread, adjusted to 512K.

-XX:+AlwaysPreTouch

A quick note is that while the JVM heap size can be set with the JVM parameters -xmx and -xms, the operating system allocates only virtual memory, and only physical memory is allocated if the JVM actually wants to use it.

Object, first of all, will be allocated in the young generation, because before the assignment is virtual memory, so every time a new object need operating systems to first assignment of physical memory, allocating objects speed decreased, naturally only after the order of merit a new generation of GC, the allocated memory space has been allocated, after allocation will accelerate the speed of the object.

In the same way, when the space of the old age is really used, naturally, the object needs to be promoted to the old age. Therefore, when the object needs to be promoted from the new generation to the old age, the operating system also needs to allocate physical memory for the old age first, which indirectly affects the efficiency of the new generation GC.

The effect of using the [-xx :+AlwaysPreTouch] parameter is to actually allocate physical memory to the JVM at service startup, instead of virtual memory. The effect is to make the code run more efficiently. There are also some shortcomings. This naturally affects the JVM process startup time, resulting in a decrease of several orders of magnitude.

-XX:ReservedCodeCacheSize

• -XX:ReservedCodeCacheSize=128m
• -XX:InitialCodeCacheSize=128m

Code Cache is the so-called Code Cache, as a result of the JVM virtual machine memory by default is a size limit, so the Code Cache region must also have a certain size limit, the default is 240 m, I set the parameters of the numerical is based on the system to run the conclusion on the monitoring data, if you want to set up according to the actual situation, set up, please don’t write.

On the other hand, if code cacahe is full leave it alone and can be configured to clean up

-XX:+UseCodeCacheFlushing

If the CodeCache space is full, enable it by adding -xx :+UseCodeCacheFlushing to the boot parameter. Turning this option on will do a cleanup to remove some CodeCache code before the JIT is turned off, that is, before the CodeCache is filled. If there is still no space after cleaning, the JIT will still be off. This option is off by default

The ending

When I switched from G1 to CMS, the memory layout and size were reset based on my Settings. Memory growth is trending, but at a very slow and cyclical pace.

I have also dumped memory snapshots and have suspected memory leaks like Arthas internal JNI, but have found no direct evidence of such leaks. For now, it is better to observe and continue to analyze after a certain period of observation.