This is the 19th day of my participation in the Genwen Challenge

If the profile

There are OOM(Out Of Memory) problems for JVM services, and solving them is a necessary practical skill for a Java technology stack staff. Some relatively general schemes are summarized here, hoping to help you.

The analysis reason

The most common reasons why a Java service appears in OOM are:

  1. It is possible that the memory allocation is really too small, and normal business uses a lot of memory (normal)

  2. An object is frequently requested but not released, and memory leaks, causing memory to run out (memory leaks, code problems)

  3. System resources are exhausted when a resource is requested frequently, for example, threads are constantly created, network connections are constantly initiated (threads are constantly created, code problems).

Screening program

Make sure that the memory itself is not too small

Jmap-heap PID

As shown in the figure above, you can view the size and usage of new generation and old generation heap memory allocation to see whether the allocation is too small.

Find the object that consumes the most memory

Methods: jmap – histo: 10765 | live more

As shown in the figure above, the command displays information about the living objects in a table, sorted by memory usage:

  • Number of instances
  • Memory size occupied
  • The name of the class

Is that intuitive? For instances/classes with large number of instances and large memory footprint, the relevant code should be reviewed accordingly.

In the preceding figure, RingBufferLogEvent occupies the largest amount of memory, occupying 18 MB of memory.

If you find that a class of objects takes up a lot of memory (for example, several gigabytes), it is likely that too many class objects have been created and have not been released. Such as:

  • After applying for resources, close() or Dispose () is not called to release resources

  • Consumers are slow to consume (or stop consuming), and producers keep sending tasks to the queue, causing the queue to accumulate too many tasks

Executing this command online forces a fullGC. Alternatively, memory can be dumped for analysis.

Check whether resources are exhausted

Tools:

  • pstree

  • netstat

Check the number of threads created by the process and the number of network connections. If resources are exhausted, OOM may appear.

Here’s another way to do it

/proc/${PID}/fd
/proc/${PID}/task
Copy the code

You can view handle details and thread counts respectively.

For example, the PID of the SSHD process on an online server is 9339

ll /proc/9339/fd
ll /proc/9339/task
Copy the code

As shown in the figure above, SSHD occupies four handles

  • 0 -> Standard input

  • 1 -> Standard output

  • 2 -> Standard error output

  • 3 -> Socket

SSHD has only one main thread PID of 9339, and no multithreading.

So, as long as

Ll/proc / ${PID} / fd | wc -l ll/proc / ${PID} / task | wc -l (pstree effect equivalent - p | wc -l)Copy the code

We know how many handles and threads the process has open.

Java Memory Overflow OOM

Two common errors in the JVM

  • StackoverFlowError: Stack overflow

  • OutOfMemoryError: Java Heap space: a heap overflow

In addition, there are the following mistakes

Java. Lang. StackOverflowError Java. Lang. OutOfMemoryError: Java heap space Java lang. OutOfMemoryError: The GC overhead limit exceeeded Java. Lang. OutOfMemoryError: Direct buffer memory Java. Lang. OutOfMemoryError: Unable to create new native thread Java. Lang. OutOfMemoryError: MetaspaceCopy the code

OutOfMemoryError and StackOverflowError are errors, not Exceptions

StackoverFlowError

Stack overflow, we have the simplest recursive call that can cause a stack overflow, that is, the depth of the method call stack is usually 512K, keep calling deep until the stack bursts

public class StackOverflowErrorDemo { public static void main(String[] args) { stackOverflowError(); } /** * stack is generally 512K, continuous deep calls, Until the stack is broken * Exception in the thread "is the main" Java. Lang. StackOverflowError * / private static void StackOverflowError () { stackOverflowError(); }}Copy the code

The results

Exception in thread "main" java.lang.StackOverflowError
  at com.moxi.interview.study.oom.StackOverflowErrorDemo.stackOverflowError(StackOverflowErrorDemo.java:17)
Copy the code

OutOfMemoryError: Java Heap space

A lot of objects are created and there is not enough heap space to store them

public class JavaHeapSpaceDemo {
  public static void main(String[] args) {
    // Size of heap space -xms10m -xmx10m
    // Create an 80M byte array
    byte [] bytes = new byte[80 * 1024 * 1024]; }}Copy the code

We create an 80M array and the Java Heap space appears directly

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Copy the code

GC overhead limit exceeded

An OutOfMemoryError is thrown if the GC collection time is too long, which is defined as more than 98% of the time spent doing GC and less than 2% of the heap memory is reclaimed

To do this quickly, we first need to set the JVM startup parameters

-Xms10m -Xmx10m -XX:+PrintGCDetails -XX:MaxDirectMemorySize=5m
Copy the code

The exception occurs as we continue to insert strings into the list until GC collection starts

public class GCOverheadLimitDemo {
  public static void main(String[] args) {
    int i = 0;
    List<String> list = new ArrayList<>();
    try {
      while(true) {
      // When the 1.6 intern() method found that the string constant pool (storage permanent generation) did not copy on the physical copy
      //1.7 when the intern() method found that the string constant pool (storage heap) did not map the actual heap memory objects on the stored address valueslist.add(String.valueOf(++i).intern()); }}catch (Exception e) {
      System.out.println("***************i:" + i);
      e.printStackTrace();
      throw e;
    } finally{}}}Copy the code

The results

[Full GC (Ergonomics) [PSYoungGen: 2047K->2047K(2560K)] [ParOldGen: [Times: 1010K -> 1010K (10168k)] 9154K->9154K(9728K), [Metaspace: 3504K->3504K(1056768K)], 0.031109secs] [Times: User =0.13 sys=0.00, real= 0.03secs] [Full GC (Ergonomics) [PSYoungGen: 2047K->0K(2560K)] [ParOldGen: 7136K->667K(7168K)] 9184K->667K(9728K), [Metaspace: 3540K->3540K(1056768K)], 0.0058093 secs] [Times: User =0.00 sys=0.00, real=0.01 secs] Heap PSYoungGen Total 2560K, used 114K [0x00000000FFD00000, 0x0000000100000000, 0x0000000100000000) eden space 2048K, 5%, informs [x00000000ffd00000 0, 0 x00000000ffd1c878, 0 x00000000fff00000) from space 512 k, 0%, informs [x0000000100000000 x00000000fff80000 0, 0 x00000000fff80000, 0) to space 512 k, 0%, informs [x00000000fff00000 0, 0 x00000000fff00000, 0 x00000000fff80000) ParOldGen total 7168 k, used 667K [0x00000000ff600000, 0x00000000ffd00000, 0x00000000ffd00000) object space 7168K, 9%, informs [x00000000ff600000 0, 0 x00000000ff6a6ff8, 0 x00000000ffd00000) Metaspace informs the 3605 k, capacity 4540 k, committed 4864K, reserved 1056768K class space used 399K, capacity 428K, committed 512K, reserved 1048576K Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Integer.toString(Integer.java:403) at java.lang.String.valueOf(String.java:3099)  at com.moxi.interview.study.oom.GCOverheadLimitDemo.main(GCOverheadLimitDemo.java:18)Copy the code

We can see multiple Full GCS without clearing space and throw an exception GC overhead limit after multiple GC operations

Direct buffer memory

Netty + NIO: This is due to NIO

  1. NIO programs often use ByteBuffer to read or write data. This is a Channel – and buffer-based I/O method that allocates out-of-heap memory directly using Native libraries

  2. This is then referenced by a DirectByteBuffer object stored in the Java heap. This can significantly improve performance in some scenarios because it avoids copying data back and forth between the Java heap and Native heap.

Bytebuffer.allocate (Capability) : The first way is to allocate JVM heap memory, which is under GC jurisdiction and is relatively slow due to copying

Bytebuffer.alloctedirect (Capability) : The second method is to allocate OS local memory, which is not GC and is relatively fast because no memory copy is required

If local memory is constantly allocated and heap memory is rarely used, then the JVM does not need to perform GC, DirectByteBuffer objects are not recycled, heap memory is sufficient, but local memory may be used up, and an OutOfMemoryError occurs when attempting to allocate local memory again. Then the program crashes.

In short, this problem occurs when there is not enough local memory but enough heap memory

We use -xx :MaxDirectMemorySize=5m to configure the available off-heap physical memory to be 5m

-Xms20m -Xmx20m -XX:+PrintGCDetails -XX:MaxDirectMemorySize=5m
Copy the code

And then we apply for a space of 6M

/ / set up 5 m of the physical memory usage, but distribution of 6 m space ByteBuffer bb = ByteBuffer allocateDirect (6 * 1024 * 1024); At this point, there will be problems with the operation

MaxDirectMemory configured: 5.0MB

[GC (system.gc ())] [PSYoungGen: 2030K->488K(2560K)] 2030K->796K(9728K), 0.0008326 secs] [Times: Sys =0.00, real=0.00 secs] [Full GC (system.gc ()) [PSYoungGen: 488K->0K(2560K)] [ParOldGen: 308K->712K(7168K)] 796K->712K(9728K), [Metaspace: 3512K->3512K(1056768K)], 0.0052052 secs] [Times: The user sys = = 0.09 0.00, real = 0.00 secs] the Exception in the thread "is the main" Java. Lang. OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:693) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at com.moxi.interview.study.oom.DIrectBufferMemoryDemo.main(DIrectBufferMemoryDemo.java:19)Copy the code

unable to create new native thread

No more threads can be created, that is, the maximum number of threads created has been reached

In high concurrency scenarios, this is applied

High concurrent requests to the server, often appear the following abnormal Java lang. OutOfMemoryError: unable to create new native thread, accurate said that the native thread anomalies related to the corresponding platform

Causes:

The application creates too many threads. One application process creates more threads than the system can handle

The server does not allow your application to create this many threads. Linux runs a single process by default and can create 1024 threads. If the application creates more than this number, Will be submitted to the Java. Lang. OutOfMemoryError: unable to create new native thread

The solution

  • Find a way to reduce the number of threads your application creates. Determine whether your application really needs to create so many threads. If not, change the code to minimize the number of threads

  • For some applications, many threads need to be created, far exceeding the default limit of 1024 threads in Linux. You can modify the Linux server configuration to expand the default limit

public class UnableCreateNewThreadDemo {
  public static void main(String[] args) {
    for (int i = 0; ; i++) {
      System.out.println("************** i = " + i);
      new Thread(() -> {
        try {
          TimeUnit.SECONDS.sleep(Integer.MAX_VALUE);
        } catch(InterruptedException e) { e.printStackTrace(); } }, String.valueOf(i)).start(); }}}Copy the code

At this point, the following error occurs, and the number of threads is approximately 900

Exception in thread "main" java.lang.OutOfMemoryError: unable to cerate new native thread
Copy the code
How do I check the number of threads
ulimit -u
Copy the code

Metaspace

The meta space memory is insufficient. The Matespace meta space uses local memory

-xx: The initial MetaspaceSize size is 20M

What is a meta-space

The meta space is our method area, which holds class templates, class information, constant pool, etc

Metaspace is implemented in HotSpot in the method area. The main differences between Metaspace and persistent generation are: Metaspace is not in virtual memory, but in local memory. In Java 8, class metadata (the Virtual Machines Internal Presentation of Java class), Is stored in a native memory called Metaspace

The permanent generation (replaced by Metaspace in the java8 meta-space) stores the following information:

  • Information about classes loaded by the VM
  • Constant pool
  • A static variable
  • Just-in-time compiled code

To simulate Metaspace space overflow, we continue to generate classes to fill the Metaspace space. Classes always take up more space than Metaspace specifies

When the code simulates exception generation, because the initialized meta space is 20M, we use JVM parameters to adjust the size of the meta space for better results

-XX:MetaspaceSize=8m -XX:MaxMetaspaceSize=8m
Copy the code

The code is as follows:

public class MetaspaceOutOfMemoryDemo {
  / / static class
  static class OOMTest {}public static void main(final String[] args) {
    // The simulation counts the number of times after an exception occurs
    int i =0;
    try {
      while (true) {
        i++;
        // Use Spring's dynamic bytecode technology
        Enhancer enhancer = new Enhancer();
        enhancer.setSuperclass(OOMTest.class);
        enhancer.setUseCache(false);
        enhancer.setCallback(new MethodInterceptor() {
          @Override
          public Object intercept(Object o, Method method, Object[] objects, MethodProxy methodProxy) throws Throwable {
            returnmethodProxy.invokeSuper(o, args); }}); }}catch (Exception e) {
      System.out.println("Number of exceptions :" + i);
      e.printStackTrace();
    } finally{}}}Copy the code

The following error occurs:

The number of exceptions: 201

java.lang.OutOfMemoryError:Metaspace
Copy the code

Pay attention to

  • Prior to JDK1.7: the persistent generation was the implementation of the method area, which housed run-time constant pools, string constant pools, static variables, and so on.

  • In JDK1.7: persistent generation is the implementation of the method area, which moves the string constant pool, static variables, and so on out of the heap memory. Run-time constant pool and the rest of the permanent generation (method area)

In JDK1.8 and later: The permanent generation is replaced by the metspace equivalent of the metspace implementation method area, where the string constant pool and static variables are still in the heap, and the runtime constant pool is still in the method area (metspace), which uses direct memory.

  • -xx :MetaspaceSize=N// Set the initial (and minimum) size of Metaspace

  • -xx :MaxMetaspaceSize=N// Setting the maximum Metaspace size is very different from the permanent generation. If the size is not specified, the virtual machine will use up all available system memory as more classes are created.