I changed my job some time ago. I was in a team mainly responsible for DSP system, and I was mainly in charge of dist- Orders service in DSP
First, DSP is what, the project team is mainly engaged in the pharmaceutical industry, taobao, Tmall, jingdong…… A lot of channels have their own stores, they also have a set of ERP, is mainly the purchase, sale and storage that set, DSP is responsible for connecting ERP and transfer information between various channels, what orders, inventory……
A few weeks ago, the leader received an email about OOM of a certain computer sub-system online, and threw it to me casually, asking me to check it when I was free
I’ll just go to the server and start using commands like top, jstack, jinfo, jstat, jmap… The result discovers one all not line, be not command not line, it is I do not have authority to view……
I just don’t know what kind of permission to apply for, even if I know, operation and maintenance has been restarted, so what’s the point of checking
Finally, I could only put my hope on heap dump files. Fortnite could download files from the server, but the files were too large, more than 6GB, so I couldn’t get them down. I asked my colleagues in operation and maintenance team for help, sent various emails, pulled groups, and finally got them to me under my soft and hard work
No more nonsense, directly on the JDK with jVisualVM software for analysis, (JVisualVM in your JDK installation bin directory)
To jVisualVM for analysis, my computer is 16G of memory, analysis of 6G of heap dump files, almost did not give my computer away……
Nothing too important here other than the resulting OOM thread and system properties Click on system properties and see that the JVM has no parameters set (Max heap, min heap, GC parameters, etc.) Let’s look at classes
Double click on char[] and String 
After a long time of searching, I finally decided to change the software
Try IBM HeapAnalyzer (www.ibm.com/support/pag…)
Is a jar package, direct Java jar – Xmx15g XXX. Jar start (how much memory the -xmx here yourself, if OOM is you give too much memory in the process of analysis), or you can just write a startup scripts
Load in the heap dump and this is what you end up with
He has a suspected leak here, a DisOrderItemBO object, which takes up 2.8 GB of memory, or 58.74%
Looking further down the ArrayList, the approximate guess is that the ArrayList loads too many DisOrderItemBO
Let him show it as a tree structure:
There are 1,936,459 instances of DisOrderItemBO in the ArrayList, which takes up more than 2 gigabytes of memory
DisOrderItemBO is used in too many places, so I don’t know where to start
So I switched to another app
MemoryAnalyzer (download address: www.eclipse.org/downloads/d…
That’s what loading looks like
There is also a suspected memory leak here, click to view it, listing the details:
The first is which thread causes the problem, and the second is which objects cause the memory to grow and not be reclaimed
Similar to the results of the previous software analysis, it is found that there are too many DisOrderItemBO elements in the ArrayList collection
Now that I know that too many DisOrderItemBO objects are being generated (in one thread) and can’t be recycled, the ArrayList is discarded because it’s fine, it’s just carrying too many DisOrderItemBO objects, and now I just need to know which thread, It just so happens that this app can be viewed (maybe the first two can also be viewed, I haven’t found them)
The http-nio-7310-exec-9 thread is the Tomcat task thread pool thread. It is not an asynchronous task or a JOB in the system that causes OOM
The thread information gives the full package name of my Service class and the full package name of my controller
I went to the online log to touch a request message, and used postman to request it. I tried several times, but there would not be many DisOrderItemBO objects
Then I thought, if the parameter is empty (the code does not validate the required parameter), I tried it. After the console typed the SQL, I took it to the online DB to execute the SQL
Remove unnecessary query columns and order, and only look at the number (one data for one DisOrderItemBO example). Similarly, if you find the previous analysis of 1936459 examples, then it is correct
To rigorous, I have one more to create the condition of the where time and time by the heap dump file generated in the first 10 minutes (OOM occurs will slow the heap dump file) results query out 1936374 data, although I don’t have to, but the error is not a lot, now, it seems, big difference in addition to the problem here is not bad
Although found, but the question point is that this interface is to query the details of the order, the operation on the page should not appear without mandatory parameters (order subject ID) ah, this question I have not thought through……
If AFTER a few weeks and half a month, I have not updated the follow-up investigation article, it means that the problem has been solved perfectly