preface
During our development process, it was inevitable that there would be so-called junk code, resulting in the server CPU being at 100% all the time. However, our application has been online, resulting in high CPU of the server, but we do not know which side of the problem, how should we find out which side of the code problem? This article introduces several tools to quickly locate.
Demo code
We will first write the code, create a New Maven project for SpringBoot, create a Web service, introduce springboot built-in Web container, pom.xml key reference JAR package as follows:
<dependency>
<groupId>
org.springframework.boot
</groupId>
<artifactId>
spring-boot-starter-web
</artifactId>
</dependency>
Copy the codeCreate service: TestWhile
Create Controller: TestWhile
Configuration file application.properties
server.port=80
server.servlet.context-path=/api
Copy the codePackage the project and upload it to the test server
Java jar demo - 0.0.1 - the SNAPSHOT. Jar &Copy the codeOpen your browser and access the infinite loop method
Open a browser, type http://xxxx/api/user/testWhile? in the location bar Size =2 If “Hello program completed” is displayed, the call is successful. (Opens 2 endless loops)
Up to this problem code, has run on the server. We found an alarm on the server, so we went online.
A native method
This method does not require additional installation tools, and it is better to use this method when the Internet is not connected.
Top and printf are native Linux commands. Jstack and jstat are JDK built-in command tools.
Many powerful Linux and Java diagnostic tools are also packaged with top, jstack, and jstat base commands.
Note: JDK installation is required for commands such as jstack and jstat. The openJdk of Linux generally does not have such tools. You can check whether these commands exist in the bin directory of Java.
Find the process that consumes the most CPU
Run the top -c command to display the process running list
Example: top-c.
Interaction 1: Press 1 to display the multi-core CPU information. Interaction 2: Type P (capital P) and the processes are sorted by CPU usage
We see that dual-core CPU utilization has reached 100%.
The first process PID is 373 and that’s the culprit; You can see the process name in the last column in the COMMAND comment: “java-jar demo-0.0.1- snapshot.jar”.
Find the most CPU-consuming thread
Run the top -h -p [PID] command to display the running information list of a process
Example: Top-HP 373, as shown in the figure below, shows multiple threads with high CPU usage
Convert thread PID to hexadecimal
Command: printf “%x\n” [thread pid], convert multiple thread numbers to hexadecimal, used in step 4 before 0x.
Example: printf ‘%x\n’ 406 405 375 376, get 196, 195, 177, 178; As shown below:
Look at the stack and locate the thread
Command: jstack process PID 】 【 | grep hexadecimal 】 【 thread after converting, A10 – use jstack to obtain process PID stack, thread id, using grep positioning printing follow-up 10 line information.
Example: jstack 373 | grep ‘0 x196 A10 -, as shown in the figure below:
Testwhile. whileTrue (); whileTrue ();
There are also two GC threads, see “GC task thread#0 (ParallelGC)” in the figure above, representing the garbage collection thread, which is responsible for garbage collection
Storage stack, batch view
To look at the stack, we can also look at the other way, we can store the jStack stack first.
Jstack [process PID] > [file]
Example: jStack 373 > demo.dump, which stores the stack information of 373 processes.
Then use cat + grep to look up the stack information for the next few high CPU threads.
Example: the cat – n demo. Dump | grep A10 – ‘0 x196’, as shown in the figure below:
WhileTrue = whileTrue; while196 = whileTrue;
The GC to check the
ParallelGC task thread#0 (ParallelGC) is the same thread that collects the Java code. The ParallelGC task thread#0 is the same thread that collects the Java code.
Let’s take a look at the JVM’s GC information using jstat.
Command: jstat -gcutil [process PID] [milliseconds]
Example jstat -gcutil 373 2000 5 Displays GC information about process 373 every 2 seconds for five times, as shown in the following figure:
S0: current usage ratio of surviving area 1 S1: current usage ratio of surviving area 2 E: usage ratio of Eden park O: usage ratio of old age M: usage ratio of metadata area CCS: compression usage ratio YGC: garbage collection times of young generation FGC: garbage collection times of old age FGCT: garbage collection time of old age GCT: Total time spent in garbage collection
The above native method search to follow a certain step, relative to some trouble point, is there a simpler method? To look down
Arthas
Arthas is a powerful Java diagnostics tool developed by Alibaba. Arthas supports JDK 6+, Linux/Mac/Windows, command line interaction, and rich Tab auto-completion to further locate and diagnose problems. Let’s see
Download Arthas
Using arthas-Boot (recommended)
Download arthas-boot.jar and start it with java-jar:
curl -O https://alibaba.github.io/arthas/arthas-boot.jarjava -jar arthas-boot.jar
Copy the code
Press 1 to enter the Java process. The PID of the Java process has changed to 373
Enter Alsace to complete, as shown in the picture below, you can see that the login path has become [arthas@17376]$, you can enter Dashboard to enter the monitoring page.
Monitor to view
Arthas has entered the Arthas interface, entered Dashboard, and when you press Enter you will see thread and stack information, as shown in the figure below, Arthas has assigned threads with high CPU usage.
Above we can see that there are 2 high threads, as well as the number and time of GC.
Thread [ID] Displays a thread
CTRL + C exit the Dashboard interface and enter Thread 32 to view thread information, as shown in the following figure:
You can see that the PUT method in the whileTrue method in the TestWhile class is causing the CPU usage to increase.
Arthas is more than that. You can decompile and look at the code.
Jad decompiling
Using jad, Arthas’s built-in decompression method, type:
* jad com.rainbow.demo.service.TestWhile*
You can decompile Java’s class to see the code for the problem function, as shown below:
Exit arthas
Finally, now that the problem has been found, quit Arthas. Enter the quit command
Arthas is very powerful and will be briefly described here and covered in a special article next time
Gu introduced a simpler script that immediately identified the problem
show-busy-java-threads
Show-busy-java-threads. sh the show-busy-java-threads.sh tool is one of the tools in the Useful Scripts tool set.
Show-busy-java-threads is used to quickly check Java CPU performance problems (the top US value is too high). Show-busy-java-threads automatically detects the threads that consume too much CPU in running Java processes and displays the thread stacks to determine method calls that cause performance problems.
Note: The core of this tool still uses the JDK’s JStack method, but it is encapsulated on top of it.
Download to the current directory
Download: github.com/oldratlee/u…
There are a lot of tools under bin, let’s just show-busy- Java-threads.sh
Upload the show-busy- Java-Threads script to the server
Be sure to grant execution permission
chmod +x show-busy-java-threads
Run directly
./show-busy-java-threads
Copy the code
As you can see in the figure below, the top 5 threads with the highest CPU usage are found very quickly.
As you can see from the first two threads, as you can see with the native tool (JStack).
Other commands
As with Arthas, show-busy-Java-Threads also has some other nice enhancements:
show-busy-java-threads
Find the threads that consume the most CPU from all Java processes (default 5) and print out their thread stacks.
show-busy-java-threads -c 3
-c 3:3 is n, which displays the top 3 CPU consuming threads.
show-busy-java-threads -c 3 -p 17376
Show that process 17376 consumes 3 more threads than the CPU group.
-p 17376:17376 indicates the process PID. The -p parameter specifies the process PID.
Show-busy -java-threads -s show-busy-java-threads -s
For sudo operations, JAVAHOME environment variables cannot be passed to root, which is often unconfigured and inconvenient to configure, so it is more convenient to explicitly specify the path of the jStack command
show-busy-java-threads -a yao.log
Import the output to the specified yao.log file
show-busy-java-threads 3 5
The command is executed every 5 seconds for three times. The default interval is 3 seconds
conclusion
This document describes three methods for troubleshooting server CPU overload. The main process is as follows:
1. Check the processes with high CPU load.
2. View the threads with the highest load in the process.
3. Get the stack information in the process.
4, get the stack corresponding thread information, find the problem method inside.
We used not only native tools, but also enhanced tools Arthas and Show-busy – Java-Threads, which greatly simplified our process.
Enhancement tools are simply wrapped on top of native methods, and many of these tools are wrapped in native methods. That’s all for today, thank you!!
The last
Welcome everyone to exchange, like the article remember to pay attention to my like yo, thank you for your support!
Welcome everyone to pay attention to my public account [code nongqingfeng], 2020 many companies Java interview notes sorted out more than 2500 pages of PDF documents, articles will be updated in it, sorted out study notes will also be placed in it!