In general, a performance report that says CPU usage is high does not help you locate the problem. Because there are a lot of different things that can happen with a high CPU. The CPU has five states (US sy ID wa ST), which can be shown in vmstat. When the code consumes CPU (which is often encountered in performance analysis), the CPU is in the US state. Of course, there are cases where the system calls generated by the code are extremely high, and SY’s CPU is high in that case (which is relatively rare, I’ve only seen it once in my career). For the JAVA language, you don’t need a particularly sophisticated profile tool to locate code. Before writing the specific analysis method, we need to talk about the thread state transition relationship, let’s first look at the system level thread state transition relationship.

From this transition, you can see that the thread is ready after it is created. It is waiting for the CPU in this state. The runing state is actually executed on the CPU. Please note the difference. The R column displayed by VMstat contains the ready and running threads (this varies by operating system, but on most Linux systems). Note this, because there are many explanations of vmstat on the web that say the r column is the number of running processes or r is the number of running threads, which is not true. Let me give you an example (which will be covered below) :

This is top running on one of my cloud servers. You can see that only one of the tasks (process) is in the running state. What about vmstat?

The server only has two cpus, so if r refers to the number of running processes or threads, it must be incorrect because two cpus can run at most two threads at the same time. So remember, this r value includes both threads waiting for the CPU (ready) and threads running (running). There will be time to explain other system-level thread states in the future. Some people may think that other states are nothing to explain, but in performance analysis, thread states are related to some performance counters. For example, the suspended state is temporarily replaced because the CPU time slice is exhausted. Blocked is blocked waiting for a condition to be met; Both of these states may lead to high CPU usage. In the process of analysis, this information gives us a direction. So just saying that the CPU is high doesn’t help you analyze the problem, because the CPU is high for a variety of reasons.

Since we’re talking about JAVA, let’s take a look at JAVA’s thread state transition relationships.

From this figure, we can see that the JAVA process has a variety of states (please search for specific states), how to see what these states are doing, you need to print the stack to see. In the stack, you can see the corresponding code (for other compiled languages, seeing the code in action is also stack dependent). In addition, in the performance analysis, stack analysis is a very important piece of content, today because just to explain how to locate the code layer from the CPU height, so do not explain the state of the thread, later have time to write an article.

Example: Let’s start by executing a CPU-consuming JAVA instance (the instance is written by a member of the 7D Group, you can find a small example online or write one yourself). Check the vmstat status.

From the image above, you can see that the left window is executing a CPU-consuming Demo, while the right window sees that the CPU of the current system has been completely consumed. To obtain the process number, run the top command:

Let’s take a look at which threads in this process are consuming CPU. As you can see from pidstat (unfortunately, I overwrote the pidstat screenshot, so I don’t want to start again), there are 10 threads consuming CPU resources. I’ll put the commands here, so you can do it yourself if you’re interested. pidstat -p 10846 -u -d -t -w -h 1 1000

As you can see from the command above, there are multiple threads consuming the CPU. The thread ids are 10861, 10862, 10863, and so on. Do thread dump using jStack. [root@7dgroup ~]# jstack -l 10846 > 10846.

Nid refers to the native ID, corresponding to the system-level TID. Except that TID shows 10 processes and NID shows hexadecimal ones. Let’s convert a thread number to find it. [root@7dgroup ~]# printf %x’\n’ 10862a6d corresponds to threaddump.

The obvious place to look is line 13 of the cpuTestThreaddemo.java.

As you can see from this example, the analysis of high CPU consumption of Java code can be done only with system-level commands and JDK native commands. Because this example is very simple, the steps are relatively clear. However, in the actual analysis of a large number of code, logic complex applications, you may see the CPU consumption on the thread is constantly switching, so it is necessary to do more thread dump, one by one analysis. Of course, with some tools, we can often do more with less when analyzing complex applications. There is only one idea here.

A lot of people have written a lot of articles about JAVA analysis, and I’m writing this article to make it more serialized.