☕ [JVM Principle Exploration] lets you completely overcome the problem of memory overflow (OOM)

Daily sentence

Only through hell, can create the strength of heaven.

The Heap is out of memory

Error message:

java.lang.OutOfMemoryError: Java heap space
Copy the code

cause

1. There may be large object allocation in the code

2. There could be a memory leak, so that after multiple GC’s, you still can’t find a chunk of memory large enough to hold the current object.

3. Business scenarios increase object data and should increase memory space.

The solution

1. Check for large object allocation, most likely large array allocation

2. Run the jmap command to dump the heap memory and use the MAT tool to check whether there is a memory leak problem

3. If no significant memory leaks are found, use -xms / -xmx to increase heap memory

4. It is also easy to overlook the need to check if there are a large number of custom Finalizable objects, possibly provided within the framework

Method area overflow

Error message:

java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: Metaspace
Copy the code

cause

• Before JDK8, the persistent generation was the implementation of the HotSot virtual machine to the method area, which stored the virtual machine loaded class information, constants, static variables, JIT compiled code, etc.

• After JDK8, the permanent generation is replaced by the metaclass, which uses local memory, among other details:

  • String constants are moved from the permanent generation to the heap
  • JVM parameters associated with persistent generation have been removed

• A permanent generation or metaspace overflow can occur for several reasons:

  1. Prior to Java7, the string.intern method was frequently incorrectly used.
  2. A large number of proxy classes were generated, causing the method area to be overwhelmed and unable to be unloaded.
  3. The application runs for a long time without restart.

The solution

• The causes of permanent generation/meta-space overflow are relatively simple, and the solutions are as follows:

  1. Check whether the permanent space or meta space is set too small.

  2. Check if there is a lot of reflection or class loading and production class bytecode in the code.

  3. After dump, mat is used to check for a large number of proxy classes generated due to reflection

  4. Restart the JVM

GC overhead limit exceeded

Error message

Java. Lang. OutOfMemoryError: GC overhead limit is exceededCopy the code

cause

This is a new type of error in JDK6, usually caused by the heap being too small.

Sun officially defines this as throwing an exception when more than 98% of the time is spent GC and less than 2% of the heap is reclaimed.

The solution

1. Check to see if the project has a lot of dead loops or code that uses a lot of memory and optimize the code.

2. Add parameters – XX: – UseGCOverheadLimit disable this check, this parameter actually doesn’t solve the problem of memory, just put the wrong information delay, end up Java. Lang. OutOfMemoryError: Java heap space.

3. Dump the memory to check whether there is any memory leak. If not, expand the memory.

The virtual machine stack and the local method stack overflow

Since there is no distinction between the virtual machine stack and the local method stack in the HotSpot VIRTUAL machine, the -xoss parameter (setting the local method stack size) exists but is effectively invalid for HotSpot, and the stack size is only set by the -xss parameter. With respect to the virtual machine stack and the local method stack, two exceptions are described in the Java Virtual Machine specification:

• StackOverflowError is thrown if the stack depth requested by the thread is greater than the maximum depth allowed by the virtual machine.

• An OutOfMemoryError is thrown if the virtual machine cannot allocate enough memory while extending the stack.

It may seem more rigorous to divide exceptions into two cases here, but there is some overlap: when stack space can no longer be allocated, either too little memory or too much used stack space are essentially just two descriptions of the same thing.

The VM StackOverflowError is abnormal. Procedure

– the Xss argument reduces the stack memory size, and then continuously calls methods causing stack overflow, StackOverflowError.

public class JVMStackSOF { private int stacklength = 1; Public void stackPush(){stacklength++; stackPush(); } public static void main(String[] args) throws Throwable{ JVMStackSOF sof = new JVMStackSOF(); try{ sof.stackPush(); }catch(Throwable e){ System.out.println("stack length = " + sof.stacklength); throw e; }}}Copy the code

openjdk@ubuntu:~$ java -Xss256k -cp /home/openjdk/NetBeansProjects/JavaApplication1/build/classes Test_JVMStackSOF. JVMStackSOF stack length = 1888 Exception in the thread "is the main" Java. Lang, StackOverflowError ats Test_JVMStackSOF. JVMStackSOF. StackPush ats (JVMStackSOF. Java: 17) test_JVMStackSOF.JVMStackSOF.stackPush(JVMStackSOF.java:18)Copy the code

-xSS256K: Sets the memory capacity of the parameter stack to 256K

• In a single thread, the virtual machine throws StackOverflowError when memory cannot be allocated, either because the stack frame is too large or because the virtual machine stack is too small.

Use the -xss argument to reduce stack memory capacity. Result: A StackOverflowError is raised, and the output stack depth shrinks when the exception occurs.

A number of local variables are defined to increase the length of the local variable table in this method frame. Result: The output stack depth shrinks when a StackOverflowError is thrown.

---

Virtual machine stack isolation, each thread has its own virtual machine stack.

In the Java Virtual Machine specification, two exceptions are specified for this area of the virtual machine stack:

1. StackOverflowError is raised if the thread requests a stack depth greater than the virtual machine allows.

2. If the virtual stack can scale dynamically (as most Java virtual machines currently do), an OutOfMemoryError will be thrown if sufficient memory cannot be allocated while scaling.

OutOfMemoryError of the VIRTUAL machine is abnormal. Procedure

The stack memory size is increased with the -xss2m argument, and new threads are constantly opened, throwing OutOfMemoryError.

public class JVMStackOOM { private void dontStop() { while (true) { } } public static void main(String[] args) { // While (true) {new Thread(new Runnable() {@override public void run() {dontStop(); } }).start(); }}}Copy the code

The principle of

• This is mainly because the -xss parameter sets the stack size of a thread. As mentioned earlier, virtual machine stacks are thread private, i.e. each thread has its own stack.

There is a limit to how much memory an operating system can allocate per process, such as 2GB on 32-bit Windows. The Java virtual machine provides parameters to control the maximum size of memory in the Java heap and method area.

2GB (the operating system limit) minus Xmx (the maximum heap size), minus MaxPermSize (the maximum method size), the program counter is small enough to be ignored. If the memory consumed by the virtual machine process itself is not counted, the remaining memory is “split” between the virtual machine stack and the local method stack.

Therefore, the larger the stack size allocated to each thread, the smaller the number of threads that can be created, and the easier it is to run out of memory when creating threads. In the first case, StackOverflowError is raised when the stack space runs out. In the second case, OutOfMemoryError is raised when memory runs out.

---

Method stack overflow (exception belonging to the virtual machine stack)

Error message

java.lang.OutOfMemoryError : unable to create new native Thread
Copy the code

cause

This kind of exception is basically caused by the creation of a large number of threads, encountered once before, through jStack out of a total of more than 8000 threads.

The solution

1. The capacity of each thread stack size reduced by -xss

2. The total number of threads is also limited by the system’s free memory and operating system. Check to see if the system has this limit

/proc/sys/kernel/pid_max /proc/sys/kernel/thread-max max_user_process (ulimit -u) /proc/sys/vm-max_map_countCopy the code

Unconventional overflow

Below these OOM abnormal, probably most of the students have not encountered, but still need to know about it

Allocating large arrays

Error message

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Copy the code

This is typically caused by improper array allocation requests, and the JVM performs a check before allocating memory to the array. Addressable specifies whether the array to be allocated is addressable on the platform. If addressable is not, this error is thrown.

The solution is to check your code to see if there are places to create large arrays.

Swap area overflow

Error message:

java.lang.OutOfMemoryError: Out of swap space
Copy the code

This situation is generally caused by the operating system, the possible reasons are as follows:

1. The size of the swap partition is insufficient.
2. Other processes consume all the memory.

The solution

1. Other service processes can optionally be split out
2. Increase the size of swap partition or machine memory

Local method overflow

Error message:

java.lang.OutOfMemoryError: stack_trace_with_native_method
Copy the code

Local methods failed to allocate memory at runtime, unlike previous method stack overflows, which occurred at the JVM code level, where local method overflows occurred at JNI code or local methods.

Native direct memory overflow

• Direct memory can be set to size by: -xx :MaxDirectMemorySize. If this is not set, it defaults to the size of the heap at its maximum value -xmx.

• The rule for setting up native direct memory is various memory sizes + native direct memory size < machine physical memory.

The following program uses DirectByteBuffe to simulate a direct memory overflow

import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.List; public class DirectBufferOom { public static void main(String[] args) { final int _1M = 1024 * 1024; List<ByteBuffer> buffers = new ArrayList<>(); int count = 1; while (true) { ByteBuffer byteBuffer = ByteBuffer.allocateDirect(_1M); buffers.add(byteBuffer); System.out.println(count++); }}}Copy the code

Run Java -xx :MaxDirectMemorySize=10M DirectBufferOom from the command line and soon the console will get an exception

Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:695)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at DirectBufferOom.main(DirectBufferOom.java:12)
Copy the code

Instead of actually requesting memory allocation from the operating system, it calculates that memory cannot be allocated and manually throws an exception. The following program makes use of the Unsafe class to simulate a direct memory overflow

import sun.misc.Unsafe; import java.lang.reflect.Field; public class UnsafeOom { private static final int _1M = 1024 * 1024; public static void main(String[] args) throws IllegalAccessException, NoSuchFieldException { Field unsafeField = Unsafe.class.getDeclaredField("theUnsafe"); unsafeField.setAccessible(true); Unsafe unsafe = (Unsafe) unsafeField.get(null); while (true) { unsafe.allocateMemory(_1M); }}}Copy the code

On the cli, run Java -xx :MaxDirectMemorySize=10M UnsafeOom

The Exception in the thread "main" Java. Lang. OutOfMemoryError at sun. Misc. Unsafe. AllocateMemory (Native Method) at Org. Fenixsoft. Oom. DMOOM. Main (20) DMOOM. Java:Copy the code

An obvious feature of a directMemory-induced memory overflow is that there are no obvious anomalies in Heap Dump files. If the Dump file is small after OOM and NIO is used directly or indirectly, you may want to check for this.