Alarm one day: a service deployed on a machine is suddenly unavailable. Keep in mind that your first reaction is to log in to your machine and check your logs. At this time, the following information was found in the machine log:

nio handle failed java.lang.OutOfMemoryError: Direct buffer memory at org.eclipse.jetty.io.nio.xxxx

at org.eclipse.jetty.io.nio.xxxx at org.eclipse.jetty.io.nio.xxxx
Copy the code

Indicates that it is indeed OOM, which area causes the problem? Direct Buffer memory, jetty method call stack, Direct Buffer memory, Jetty method call stack

Direct buffer memory

Out-of-heap memory, the area of memory outside the JVM heap memory that is not managed by the JVM, but that Java code can use outside the JVM heap. This space is called Direct Buffer memory, which is directly managed by the OS. But it’s a little strange to call it direct memory. I prefer to call it “off-heap memory.”

Jetty runs the flow of our written system as a JVM process:



This OOM is caused by Jetty using out-of-heap memory. As you can see, Jetty may be using out of heap memory constantly, and then running out of space to use more out of heap memory, so it gets OOM.

Jetty is constantly using out-of-heap memory:



Address OOM underlying technologies

Since Jetty is written in Java, how does he apply for off-heap memory using Java code? And then how does this off-heap space get freed up? This involves the NIO underlayer of Java.

JVM performance tuning is relatively easy, but if OOM is solved, except for something stupid and simple, like someone constantly creating objects in code. Many of the other OOM issues are technically difficult and require solid technology.

How is out-of-heap memory allocated and freed?

For example, if you want to use a block of out-of-heap memory in Java code, use the DirectByteBuffer class. You can use this class to build a DirectByteBuffer object, which is itself in the JVM heap memory.

But when you’re building this object, you’re going to create a chunk of memory that’s out of the heap and associate it with it, and if you look at the picture below, you get a pretty good idea of the relationship between the two.

So that’s basically the idea when you allocate out of heap memory.

How do I free off-heap memory

When your DirectByteBuffer object is not referenced and becomes garbage, it will be collected during a YGC or Full GC.

Whenever a DirectByteBuffer object is reclaimed, its associated out-of-heap memory is freed:

So why is there an out-of-heap memory overflow?

If you create a lot of DirectByteBuffer objects that take up a lot of out-of-heap memory, and then there is no GC thread to collect them, they will not be freed.

When out-of-heap memory is associated with a large number of DirectByteBuffer objects, if you use additional out-of-heap memory, you are reported to be out of memory! When does a large number of DirectByteBuffer objects live forever, resulting in a large amount of out-of-heap memory that cannot be freed?

If the system has high concurrency, it may create too much DirectByteBuffer, which consumes a large amount of out-of-heap memory. If the system continues to use out-of-heap memory, it will be OOM! But this is clearly not the case with the system.

Real out-of-heap memory overflow cause

You can use jstat to see how the system is running on the line, look at the processing time of some requests based on the logs, analyze the past GC logs, and look at the call time of various interfaces on the system.

Firstly, the interface call time. The system concurrency is not high, but it takes a long time to process each request, which takes 1s on average.

Jstat then finds that as the system is called over and over again, various objects are created, including Jetty itself, which creates DirectByteBuffer objects to claim out-of-heap memory, and YGC is triggered until Eden is full:

However, at the moment of GC, many DirectByteBuffer objects are still alive and have not been collected. Of course, many DirectByteBuffer objects can be collected after their corresponding requests have been processed.

There must be some DirectByteBuffer objects and some other objects that are alive and need to be turned into Survivor zones. I remember that when the system went online, the memory allocation was very unreasonable, so the young generation was given one or two hundred M, while the old generation was given seven or eight hundred M, resulting in only 10M Survivor in the young generation. Therefore, after YGC, some surviving objects (including some DirectByteBuffers) exceed 10M and cannot be placed into Survivor, so they are directly entered into Old:



As a result, some DirectByteBuffers slowly enter Old, and more and more Old DirectByteBuffers are associated with a lot of out-of-heap memory:

The old directBytebuffers can be recycled, but the old directBytebuffers can not be recycled because the old directBytebuffers are not full. Of course, in the old days directByteBuffers that were not collected were associated with a lot of out-of-heap memory!

Eventually, when you continue to use out-of-heap memory, all of the out-of-heap memory is occupied by a large number of directByteBuffers from the old generation, which can be recycled, but the out-of-heap memory can never be recycled because the old generation full GC is never triggered. Finally lead to OOM!

Why does Java NIO look so sand sculpted?

Didn’t Java NIO consider this to happen?

Considering it! He knows that there may be many DirectByteBuffer objects that no one is using, but that are occupying out-of-heap memory because gc is not triggered. Java NIO does the following: Every time a new out-of-heap memory is allocated, system.gc () is called to remind the JVM to actively perform the following GC to reclaim some garbage DirectByteBuffer objects that no one references, freeing up out-of-heap memory.

As long as the GC can be triggered to reclaim some of the DirectByteBuffers that no one references, some of the out-of-heap memory can be freed up and more objects can be allocated to out-of-heap memory. But since we set it in the JVM again:

-XX:+DisableExplicitGC
Copy the code

System.gc() does not take effect, resulting in OOM.

The ultimate optimization

The project has the following problems:

  • As a result, DirectByteBuffer objects are slowly aging, and the out-of-heap memory cannot be freed
  • With -xx :+DisableExplicitGC set, Java NIO cannot actively remind to recycle some garbage DIrectByteBuffer objects and cannot free out-of-heap memory

In this regard, we should:

  • Allocate memory properly, give more memory to the young generation, and make more room for Survivor regions
  • Release -xx :+DisableExplicitGC to enable system.gc ()

After optimization, DirectByteBuffer generally does not continue to age. As long as it stays in the young generation, the young GC will collect and free the out-of-heap memory normally.

As long as the -xx :+DisableExplicitGC limit is released, Java NIO will alert the JVM to active garbage collection via System.gc() and reclaim some of the DirectByteBuffer, thus freeing up out-of-heap memory.