Understand OutOfMemoryError

OutOfMemoryError should be a pretty tricky problem. JAVA programmers don’t have to manage memory manually like frustrated C programmers; the JVM helps them allocate and free up memory. However, when encountering a memory-related problem, such as OutOfMemoryError, how to troubleshoot and resolve it becomes a very troublesome problem. In JAVA, all objects are stored in the heap, and usually outofMemoryErrors are thrown if the JVM is unable to allocate new memory, memory runs out, and the garbage collector is unable to reclaim memory in time.

I was working on a tool that needed to read a large number of files, such as Word or Excel, and the maximum memory I allocated to the machine was only 2 gigabytes. As a result, many people’s machines will abort programs due to OutofMemoryErrors. Later, I found that OutOfMemoryError can be captured by Error or Throwable. OutOfMemoryError class inheritance relationship is as follows:

java.lang.Object
    java.lang.Throwable
        java.lang.Error
            java.lang.VirtualMachineError
                java.lang.OutOfMemoryError
Copy the code

Thus OutOfMemoryError is an Error, not an Exception, and as far as I can see, OutOfMemoryError cannot be thrown into the upper-level function.

private void OutOfMemoryErrorTest() { try { // do something might lead to OutOfMemoryError error } catch (Error e) { e.printStackTrace(); }}Copy the code

Cause of an OutOfMemoryError

The sooner we can figure out the cause of outofMemoryErrors, the better we can solve the problem. Is it because JAVA is overflowing or because the native heap is overflowing? To find out why, we can get hints by looking at the details of the exception.

Exception in thread thread_name: java.lang.OutOfMemoryErrorError: Java heap space

This is a very common situation, and most OutOfMemoryError exceptions are caused by this. This detail indicates that objects cannot be reallocated in the JAVA heap. This error does not necessarily mean that your program has a memory leak. It could simply be a configuration issue, and the default heap memory (the memory set by the JVM) could not meet the requirements of the application.

It is also possible that in long-running programs, references to objects that are no longer needed are kept, preventing the garbage collector from collecting memory to allocate new memory. This is tantamount to a memory leak.

Another potential cause may be the overuse of finalize methods. If a class has a Finalize method, objects belonging to that class will not reclaim space in garbage collection. Instead, objects wait in a queue for destruction after garbage collection, which usually happens later. In Oracle Sum’s implementation, finalizer is performed by a daemon thread that serves the Finalization queue. If the Finalizer thread can’t keep up with the Finalization queue speed, the JAVA heap will fill up and OutOfMemoryError will be thrown.

Exception in thread thread_name: java.lang.OutOfMemoryErrorError: GC Overhead limit exceeded

This is another common exception message that indicates that the JAVA program is running slowly and the garbage collector is always running. After garbage collection, if the JAVA process spends more than 98% of its time doing garbage collection, an OutOfMemoryError will be thrown if less than 2% of the heap memory is recovered in five consecutive garbage collections. This is usually the case because generating so much data takes up JAVA heap memory that there is no way to allocate new memory. In layman’s terms, garbage collectors can’t collect fast enough to keep up with memory allocation. This is just like a family has some property in their family, but the property is limited. Although they can get some back regularly, they cannot help but have a black sheep in their family, so they will go bankrupt sooner or later (OutOfMemoryError).

GCOverheadLimit Exceeded can be closed with the command line flag -xx: -usegCoverheadLimit, although OutOfMemoryError may still be raised.

Exception in thread thread_name: java.lang.OutOfMemoryErrorError: Requested array size exceeds VM limit

This exception message indicates that the application is trying to allocate an array that is larger than the heap size. For example, if a program tries to allocate a 512 MB array, but the maximum heap size is 256MB, OutOfMemoryError will be thrown. The exception message is usually caused by either a configuration problem (the heap is too small) or a BUG in the program that tries to allocate too large an array.

Exception in thread thread_name: java.lang.OutOfMemoryErrorError: Metaspace

Java class metadata (the internal representation of a Java class virtual machine) uses native memory (in this case, metaspace) for allocation. If metaspace for metadata is exhausted, an OutOfMemoryError with this exception information is thrown. The total number of Metaspace is limited by the MaxMetaSpaceSize parameter, which can be set on the command line. When the total amount of memory allocated to metadata primitively exceeds MaxMetaSpaceSize, an OutOfMemoryError with this exception information is thrown. MetaSpace and the JAVA heap are allocated from the same address space. Reducing the size of the JAVA heap increases MetaSpace’s space.

Exception in thread thread_name: java.lang.OutOfMemoryErrorError: request size bytes for reason. Out of swap space?

This exception message appears to be an OutOfMemoryError exception. However, the Java HotSpot VM code raises this exception when the native heap cannot allocate memory or when the native heap may be near exhaustion. The usual reason for this exception message is that the source code module reports an allocation failure, although this is sometimes the case. When this error message is thrown, the VM invokes fatal error handling (that is, it generates a fatal error log file containing useful information about the thread, process, and system at the time of the crash). Heap memory and memory mapping information in the log can be useful in the case of local heap exhaustion. If an OutOfMemoryErrorError exception is thrown, you may need to use a troubleshooting utility on your operating system to further diagnose the problem.

Exception in thread thread_name: java.lang.OutOfMemoryError: Compressed class space

On 64-bit platforms, Pointers to the metadata class can be represented with 32-bit offsets (using UseCompressedOops). This from the command line mark UseCompressedClassPointers control (the default is on). If use UseCompressedClassPointers, metadata class a fixed amount of space available is for CompressedClassSpaceSize. If the space UseCompressedClassPointers than CompressedClassSpaceSize, will throw a contains detailed Compressed class space Java. Lang. OutOfMemoryError. Increase CompressedClassSpaceSize can close UseCompressedClassPointers. There are limits to the acceptable size of CompressedClassSpaceSize. For example, -xx: CompressedClassSpaceSize = 4G, exceeding the acceptable range will result in the following message:

CompressedClassSpaceSize of 4294967296 is invalid; must be between 1048576 and 3221225472.

Note: There are many types of metadata classes – klass metadata and other metadata. Only klass metadata is stored in a space limited by CompressedClassSpaceSize. Other metadata is stored in Metaspace.

Exception in thread thread_name: java.lang.OutOfMemoryError: reason stack_trace_with_native_method

If the exception message is this and a stack trace is printed with the first frame being the native method, then the native method has encountered an allocation failure. The difference between this and the previous message is that allocation failures are detected in Java native interfaces (JNI) or native methods, rather than in JVM code. If such an OutOfMemoryError exception is thrown, you may need to use the operating system’s native utilities to further diagnose the problem.

The solution

Given the various cases of OutOfMemoryError exceptions and their possible causes, how do you resolve OutOfMemoryError exceptions? This can happen for a variety of reasons, and sometimes it can be a BUG in the program that causes a memory leak. Sometimes it may be a setup problem, the memory Settings are too small, as long as the larger Settings can be set. Sometimes it is not necessarily a memory leak, may be the program allocated memory can not handle, at this time you need to think of ways to optimize, to avoid memory consumption, or precisely to avoid allocating too much memory at once, resulting in memory allocation failure. Here are some solutions based on my own experience.

The simplest and most crude approach is to simply adjust the HEAP size of the JVM. The -xmx parameter allows you to set the maximum memory for the JAVA heap, which can be avoided if you allocate too little memory in the first place. The parameters should be set based on how the program is running and the actual memory of the machine. In general, the HEAP size of the JVM should not exceed half of the machine memory. Adjusting the parameter Settings may solve temporary problems, but often only delays outofMemoryErrors, but finding the key problems in the program and finding the key memory consumption points is essential.

Another common way to avoid exceptions is to remember to close the input stream. It is not uncommon for someone to open a file and forget to close the input stream at the end. If an exception occurs, the input stream will not be closed. It is common to close the input stream in finally, because this step is always performed in finally. In JAVA7 you can use try-with-resources to automatically close a resource:

try (FileInputStream input = new FileInputStream("file.txt")) {
    // operate the data
}Copy the code

Strings and Lists are commonly used data types in JAVA. In fact, JAVA has done a lot of built-in String optimization, the individual optimization is actually very small. The only thing developers can do is check the allocation of strings in the program to see if there is any string manipulation that is not necessary. The other is that for dynamic array type data, you can use ArrayList whenever possible. ArrayList is a data structure based on dynamic arrays, while LinkedList is a data structure based on linked lists. In general, querying an ArrayList of data is more efficient for data manipulation, but LinkedList is more efficient for deletion or insertion. The space waste of ArrayList is mainly reflected in reserving a certain amount of space at the end of list, while the space cost of LinkedList is reflected in that each element of it needs to consume a certain amount of space. Because the implementation of ArrayList is based on dynamic array, the dynamic expansion of ArrayList increases at a rate of 1.5 times, which results in the dynamic expansion of ArrayList when it is already large, it needs to allocate more space. The other small thing is that you can use trimSize to reduce the space taken up by an ArrayList, but make sure no new elements are added later.

Another common situation is to read files, such as TXT files and Excel or Word files. I develop programs that need to read a lot of files, and outofMemoryErrors are often caused by file reads. Using Scanner to read TXT files, you can control the amount of text read at a time by delimiter, thus avoiding reading a large amount of text at a time. POI is arguably the best solution for reading Word and Excel files. I wrote an article on POI best practices for reading files. This article summarized some of the pitfalls encountered in reading Word and Excel files using POI. I think can be counted on the domestic network is better about this aspect of the article. Older versions of Word or Excel were binary data, while later versions are essentially compressed files. If you open a DOCx file as a compressed file, you can observe its internal composition. So, while the size of a Word or Excel file may not be excessive, it often consumes a lot of memory when reading the reader’s memory. For the reading of Excel files, you can use the streaming mode to read and read. Large files are divided into temporary small files for reading, so as to avoid memory overflow. There is an excellent third-party library on the Internet called Excel – Streaming-Reader. Another optimization I made was that in cases where File objects were available, I used File objects to read files instead of InputStream, because using InputStream required loading it all into memory, which was very memory intensive.

There are also some development considerations. Avoid operating on the same variable for long periods of time, such as manipulating arrays and adding new elements, which can cause OutofMemoryErrors. Batch operations can be performed to avoid unlimited memory expansion and eventually running out of memory.

In summary, the causes of memory overflow can be varied, there may not be any one cause, and its performance may not be stable. This is why OutofMemoryErrors can be difficult to troubleshoot and difficult to resolve. Sometimes you may need to use a performance analysis tool, such as memory log dump or use the JDK’s JVM performance analysis tool jConsole to analyze memory usage and troubleshoot problems.

The above

Please search the wechat id mad_coder or scan the QR code to follow the official account: