This article was originally published in: Walker AI

With the continuous development of platform services and the continuous increase of platform data library, the system occasionally appears the situation of memory overflow. Compared with other exceptions, memory overflow is usually more subtle and accumulates over time. Therefore, it is important to find the root cause of the problem rather than simply locate the problem based on the occurrence of the exception. Therefore, it is necessary to know how to troubleshoot system memory overflow. This article takes a memory overflow in a production environment as an example to explain how to use Jprofiler to locate the problem.

Problem 1.

Suddenly I received a message that the platform was faulty. When I opened the homepage of the website, I found that all interfaces could not be accessed normally and the page failed to load, as follows:

The first time I checked the anomaly log, I found that the log almost contained the following content:

There is no doubt that the system ran out of memory. However, the specific cause of the log cannot be located. Generally, for a stable system, the size of the memory it occupies is generally kept within a certain range, so memory overflow is generally caused by the following reasons:

  • When programmers write code, they cause memory leaks. Unreleased memory gradually overbuilds and eventually overflows.
  • Try to apply for a large amount of memory in the code, resulting in most of the memory occupied, or directly resulting in memory overflow;
  • The system suddenly increases the flow, the original available memory is not enough to support.

The heap dump file will be automatically dumped when OOM occurs. The heap dump file will be automatically dumped when OOM occurs.

2. Preparation

2.1 tools

Jprofilor is a powerful JVM monitoring tool that provides accurate monitoring of all aspects of the JVM, including memory, GC, CPU usage, thread status, and more. For overflow, jProfiler’s heap snapshot analysis capabilities are typically used; the rest are beyond the scope of this article.

Jprofiler installation is very simple, follow the installation wizard to the next step, install the following interface:

Click “Start Center” in the upper left corner, and you can select different function modules in the pop-up window. Among them, “Quick Attach” can quickly connect the Java process that is running on the machine or other hosts for real-time monitoring.

For example, if you select “Start Center”, you can see the running IDEA Java process:

Select Start to enter the real-time monitoring mode. You can view the following information about the process:

To troubleshoot memory overflow problems, jprofiler’s heap snapshot analysis feature is often used, which is discussed later.

2.2 the JVM parameter

For a running system, using “Quick Attach” to monitor and analyze Java processes is not convenient, but the JVM has added a parameter for automatically dumping heap snapshot files when a Java process produces an “Out of Memory” exception, that is, a Memory leak. Therefore, we only need to analyze the heap memory snapshot. To enable this function, you need to add the following startup parameters when the Java process is started:

 	java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/app.hprof -jar app.jar
Copy the code

Among them,

     -XX:+HeapDumpOnOutOfMemoryError
Copy the code

Dump file is automatically generated when OOM occurs.

     -XX:HeapDumpPath
Copy the code

Specifies the path to the generated dump file.

3. Analyze the DUMP files of the JVM heap

After the above configuration, you can troubleshoot the fault when the system generates OOM dump files. This section uses an OOM dump file as an example to explain how to troubleshoot the fault.

In jprofiler’s “Start Center” check the “Open a Single Snapshot” box in the “Open Snapshots” function:

After the import is successful, the following interface is displayed:

Under the “Classes” TAB, you can select the size of each instance by number or total size. The char[]” type, as a built-in implementation of “String”, usually takes up a lot of memory, so this list alone is not good for locating problems. To do this, select the “Biggest Objects” TAB, which, as the name suggests, is aimed at large Objects. The diagram below:

Using the “Biggest Objects” function, we can quickly see that one of the String Objects is a full 110MB, which is obviously not reasonable. Next, you can try to preliminarily determine where the String was generated based on its contents; If you still can’t make it a problem, you can right-click the object and select “Use Selected Objects”:

The “Reference” option in the pop-up window is used to view the references of the object. The “Outgoing References “and “Incoming References” options refer to the references held by the selected object and the references held by the current object respectively.

Since String is an immutable type, its internal implementation is done by char[], so select “Incoming References “:

In this window, you can see not only the content and size of the object, but also the thread of the object, which feels like a step closer to the root of the problem. Then select “show more”:

At this point, the entire call stack for the resulting object is visible. In THIS OOM, the final cause is a huge SQL statement generated in the business code, which can be executed normally when the amount of data is small. As a large amount of data was added recently, the corresponding processing was not considered at the code level, so OOM appears. With this lesson in mind, you should also consider avoiding such situations when writing code in the future.

4. Conclusion

In general, memory overflow is a difficult problem to deal with. The examples in this article cover only the simplest analysis scenarios, but you can usually find most of the causes; Complex OOM scenarios also need to be combined with the actual situation. How to avoid such problems is to start with the code, after all, most of these problems are caused by programmers writing code improperly.

PS: more dry technology, pay attention to the public, | xingzhe_ai 】, and walker to discuss together!