Welcome to github.com/hsfxuebao/j… , hope to help you, if you feel ok, please click on the Star

From: coderbee.net/index.php/j…

First, basic knowledge

1.1. The JVM is aware of container resources

Java applications are deployed in Kubernetes clusters, each container runs only one process, and the JVM startup commands are packaged in an image file.

The conventional approach is to specify the maximum and minimum size of the JVM heap with arguments such as -xmx4G-xMS2G, and to repackage the image if the heap size needs to be adjusted.

To avoid repacking for heap size changes, JDK 8U191 supports JVM awareness of container resource limits so that image files do not need to be repackaged when the JVM memory allocation is adjusted, using the following parameters to make the JVM aware of container resource limits at startup and set the heap size:

+ UseCGroupMemoryLimitForHeap - - XX: XX: InitialRAMPercentage = 60.00 XX, 80.00 XX MaxRAMPercentage = : MinRAMPercentage = 60.00Copy the code

Given that the maximum memory allocated to the container is 4G, the initial and minimum sizes of the JVM heap are 4G * 0.6 (2.4 GB) and the maximum size is 4G * 0.8 (3.2 GB) in the above configuration.

1.2. The JVM oomkill

After the preceding configuration is run for a period of time, the container restarts automatically. Run the dmesg command in Linux to view system logs. The following logs are displayed:

Aug  8 15:32:40 H-LDOCKER-01 kernel: [ pid ]   uid   tgid   total_vm      rss        nr_ptes   nr_pmds   swapents   oom_score_adj  name
Aug  8 15:32:40 H-LDOCKER-01 kernel: [33775]   1001  33775  7624373       2036828    4476      32        0          -998           java
Aug  8 15:32:40 H-LDOCKER-01 kernel: Memory cgroup out of memory: Kill process 33775 (java) score 0 or sacrifice child
Aug  8 15:32:40 H-LDOCKER-01 kernel: Killed process 33775 (java) total-vm:30497492kB, anon-rss:8134056kB, file-rss:13256kB
Copy the code

Note: the RSS column in the log above represents the memory occupied by the process. The corresponding value is 2036828, in 4KB, which means that the Java process occupied 7.77 GB. The maximum memory allocated by the container is 8 GB. Lines 3 and 4 indicate that the Java process has been run by oOM_killer.

OOM_killer is a self-protection measure in Linux. When the system memory is insufficient, the system wakes up OOM_killer, selects the process with the maximum value of /proc/ /oom_score, and kills it to prevent serious problems.

Since the application also prints GC logs, the JVM’s heap is far less than 7GB from the time the process was killed, and the extra memory is actually out of the heap.

1.3. JVM out-of-heap memory

The JVM’s off-heap memory consists of:

  • The space occupied by the JVM itself;
  • Thread stack allocation of occupied system memory;
  • Memory occupied by DirectByteBuffer;
  • Memory allocated in JNI;
  • Metadata space starting with Java 8;
  • NIO buffer
  • Unsafe calls allocated memory;
  • codecache

Iceberg object: An iceberg object is a small memory footprint in the JVM heap, but actually refers to a large chunk of local memory. Directbytebuffers and threads fall into this category.

Out-of-heap memory leaks are generally difficult to analyze with tools like MAT, and must be analyzed with tools at the operating system level.

NMT analyzes out-of-heap memory

Native Memory Tracking (NMT) is a feature introduced by HotSpot JVM to track the Native Memory used within the JVM. NMT data can be accessed using the JCMD tool. NMT does not currently support tracking memory allocations and JDK libraries for third party native code.

NMT does not track memory allocations for non-JVM code, and memory leaks in native code need to be located using tools supported by the operating system.

2.1 open the NMT

Enabling NMT results in a 5-10% performance penalty. NMT memory usage requires the addition of two machine words to the Malloc header of malloc memory. NMT memory usage is also tracked by NMT.

Startup command: – XX: NativeMemoryTracking = [off | summary | detail].

Off: NMT is turned off by default. Summary: Collect only the total memory usage data for the subsystem; Detail: Collect memory usage data for each call point.

2.2 JCMD Accessing NMT Data

Command: jcmd VM.native_memory [summary | detail | baseline | summary.diff | detail.diff | shutdown] [scale= KB | MB | GB]

OPTION

DESC

summary

Print summary data by category

detail

Print summary data by category Print virtual memory mapping Print summary memory usage by call point

baseling

Create a memory snapshot for subsequent comparison

summary.diff

Print a summary report based on the latest baseline

detail.diff

Print a detailed report based on the latest baseline

shutdown

Close the NMT

In the case of NMT is enabled, can through the following command line options in the JVM exits the output of the last memory usage data: – XX: XX: + UnlockDiagnosticVMOptions – + PrintNMTStatistics

2.3 Using NMT to Detect Memory Leaks

  1. To start NMT, run the following command:-XX:NativeMemoryTracking=summary|detail
  2. To create a baseline, run the following command:jcmd <pid> VM.native_memory baseline
  3. Observe memory changes:jcmd <pid> VM.native_memory detail.diff

NMT data output

Reserved memory: reserved memory, which does not indicate actual use. The most important thing is to apply for a batch of continuous address space. (OS) Commited Memory: actually used. (OS perspective) For 64-bit systems, the address space is almost unlimited, but increasingly committed memory may cause Considerations like Cleanup or local OOM.

The following example from docs.oracle.com/javase/8/do… .

-xx :NativeMemoryTracking=summary with JCMD < PID > vm. native_memory summary

Total:  reserved=664192KB,  committed=253120KB  <--- total memory tracked by Native Memory Tracking

-     Java Heap (reserved=516096KB, committed=204800KB)  <--- Java Heap
                (mmap: reserved=516096KB, committed=204800KB)

-     Class (reserved=6568KB, committed=4140KB)     <--- class metadata
            (classes #665)                          <--- number of loaded classes
            (malloc=424KB, #1000)                   <--- malloc'd memory, #number of malloc
            (mmap: reserved=6144KB, committed=3716KB)

-     Thread (reserved=6868KB, committed=6868KB)
            (thread #15)                            <--- number of threads
            (stack: reserved=6780KB, committed=6780KB) <--- memory used by thread stacks
            (malloc=27KB, #66)
            (arena=61KB, #30)                       <--- resource and handle areas

-     Code (reserved=102414KB, committed=6314KB)
           (malloc=2574KB, #74316)
           (mmap: reserved=99840KB, committed=3740KB)

-     GC (reserved=26154KB, committed=24938KB)
           (malloc=486KB, #110)
           (mmap: reserved=25668KB, committed=24452KB)

-     Compiler (reserved=106KB, committed=106KB)
               (malloc=7KB, #90)
               (arena=99KB, #3)

-     Internal (reserved=586KB, committed=554KB)
               (malloc=554KB, #1677)
               (mmap: reserved=32KB, committed=0KB)

-     Symbol (reserved=906KB, committed=906KB)
             (malloc=514KB, #2736)
             (arena=392KB, #1)

-     Memory Tracking (reserved=3184KB, committed=3184KB)
                      (malloc=3184KB, #300)

-     Pooled Free Chunks (reserved=1276KB, committed=1276KB)
                         (malloc=1276KB)

-     Unknown (reserved=33KB, committed=33KB)
              (arena=33KB, #1)
Copy the code

-xx :NativeMemoryTracking=detail and JCMD VM. Native_memory detail

Three, the analysis of the system level

Memory leaks are usually not a sudden surge to the limit, but rather a gradual increase, so that we can take two times of memory and compare it to see what is in the new memory.

3.0 the GDB way

GDB exports the contents of a block of memory in the specified address range:

sudo gdb --batch --pid 2754 -ex "dump memory a.dump 0x7f1023ff6000 0x7f1023ff6000+268435456"
Copy the code

Then use the hexdump) – C/TMP/memory. Bin or strings/TMP/memory. Bin | less to check the content in the memory block.

If the memory block contains text, you can see what it contains, but if it’s binary, you can’t read it.

3.1 jstack/jmap + core dump

Core dump = core dump; JVM heap dump = core dump;

Create core dump with gcore Zul-jdk /zulu8.40.0.25-ca-jdk8.0.222-linux_x64/bin/jstack ~/zuul-jdk/zulu8.40.0.25-ca-jdk8.0.222-linux_x64/bin/ Java core.1791 ~ / zuul - JDK/zulu8.40.0.25 - ca - jdk8.0.222 - linux_x64 / bin/jmap - dump: the format = b, the file = zuul. Jmap. Hprof ~/zuul-jdk/zulu8.40.0.25-ca-jdk8.0.222-linux_x64/bin/ Java core.1791 Jstack exec core-file jmap <options> exec core-file jstack <options> exec core-fileCopy the code

3.2 JHSDB

JHSDB: HSDB is short for HotSpot Debugger, a debugging tool introduced in JDK9.

$ jhsdb
    clhsdb              command line debugger
    hsdb                ui debugger
    debugd --help       to get more information
    jstack --help       to get more information
    jmap   --help       to get more information
    jinfo  --help       to get more information
    jsnap  --help       to get more information
Copy the code

When openJDK 11 failed to extract the real operation, some memory address read failures occurred when generating heap dump.

Extract information from core dump with jStack:

sudo jstack -J-d64 /usr/bin/java core.2316

jhsdb jstack --exe /usr/bin/java --core core.2316
Copy the code

– D64 indicates a 64-bit system. These two systems were also found online, but did not actually succeed.

Iv. Reference materials

  • Check the Case of Java out-of-heap memory growth
  • Remember a Java Native memory growth problem
  • Extract JVM information from Core dump, so-you-want-your-jVMS-heap