4.1 Procedure for Analyzing Online Logs
  • Run the top command to check the CPU status. If the CPU is high, run the top-hp command to check the running status of each thread in the current process. After finding the thread whose CPU is high, convert its thread ID to hexadecimal, and view the main work of the thread in the JStack log. There are two cases here
  • If it is a normal user thread, the stack information of the thread is used to see where the user code is running.
  • If the Thread is a VM Thread, run the jstat -gcutil command to monitor the GC status of the current system. Then run the jmap dump:format=b,file= command to export the current memory data of the system. After the export, the memory situation can be analyzed in the Mat tool of Eclipse to get the main objects in memory that consume more memory, and then the relevant code can be processed.
  • If you run the top command, the CPU is not high and the system memory usage is low. At this point, you can consider whether the problem is caused by the other three conditions. Specific analysis can be made according to the specific situation:
  • If the interface invocation is time-consuming and occurs irregularly, you can increase the occurrence frequency of choke points by using pressure measurement to view stack information and find choke points through jStack.
  • If a function suddenly stops, this situation cannot be repeated. In this case, you can export jStack logs several times to compare the user threads that have been in the waiting state. These threads are the threads that may have problems.
  • If the deadlock status is visible through JStack, you can examine the specific choke points of the two threads causing the deadlock and deal with the problem accordingly.
4.2 How to Check the Java CPU Usage
  • Top: Finds the PID of the process that occupies a high CPU usage
  • Jstack PID >> java_stack.log: Exports the stack of threads that occupy the highest CPU usage
  • Top-hp PID: identifies the tid of the thread whose PID processes occupy too much CPU. (or using the ps command – mp PID -o THREAD, dar, time | sort – rn | less)
  • Printf “%x\n” tid: converts the required thread ID to hexadecimal format.
  • Less javA_stack. log: searches for the hexadecimal thread TID, finds the corresponding thread stack, and analyzes and handles the problem.
4.3 How to Check the High Java Memory Usage
  • Top: Find the PID of the Java process with the highest memory usage (RES column).
  • Jmap-heap PID: Displays heap memory usage.
  • Jps-lv: Displays JVM parameter configurations.
  • Jstat -gc PID 1000: Collects GC information about the specific size of each area of the heap per second.
  • Jmap-dump :live,format=b,file=heap_dump.hprof PID: export heap files.
  • Use MAT to open the heap file and analyze the problem.
4.4 Procedure for Troubleshooting Java Out-of-heap Memory Leaks
  • Top: Find the PID of the Java process with a high memory footprint (RES column).
  • Jstat -gcutil PID 1000 Displays the percentage of regions in the heap per second. If gc is normal, the out-of-heap memory usage is analyzed.
  • JCMD PID VM. Native_memory detail, the command takes effect after adding the JVM parameter -xx :NativeMemoryTracking=detail and restarting the Java process. The command displays the memory usage and displays the output. Whether the total committed memory is less than physical memory (RES), because the JCMD command displays memory that includes memory in the heap, the Code area, and the memory that is applied via unsafe.allocateMemory and DirectByteBuffer. But it does not include out-of-heap memory requested by other Native Code.
  • Pmap – PID x | sort – rn – k 3: check the memory distribution, whether have the address space is not in the address space of the JCMD command given.
  • Locate out-of-heap memory with tools such as GperfTools, GDB, Strace, etc.