One night last week, when everyone was ready to go off work happily, the alarm robot in the group suddenly bubbled untimely: the CPU resource of the server suddenly increased. Normally, there should not be many users using our system at this point, which is probably caused by some scheduled task or open API.

In no way, I had to turn on the monitor that had been turned off and start investigating what the problem was.

1. Check process records

1.1 the top

The first step is to use the top command to check the CPU resource allocation and process number.

As can be seen from the figure, the %CPU value of the process with PID=31439 is 100%, indicating that the CPU usage of this process has reached 100%, and the online alarm is issued by this process.

If in doubt, you can use the JPS -l command to look at the Java project running on the server online and compare process ids to determine the source of the alert.

1.2 TOP-HP PROCESS PID

The second step is to run the top-hp process PID command to check the CPU resource usage of each thread in the process.

The purpose of this command is to determine which thread is executing the task causing the CPU alert. As you can see from the graph, the thread PID=31440 is occupying 99% of the CPU.

1.3 printf “0x%x\n” thread PID

The third step is to convert the thread PID to hexadecimal with printf “0x%x\n” thread PID.

This step can also be used in other ways, such as the Online Decimal to Hexadecimal site, which gets the hexadecimal value of the thread PID because the thread number is represented in the thread stack file in hexadecimal.

1.4 jstack process PID | vim + / hex threads PID –

The fourth step is to use jstack process PID | vim + / hex threads PID – stack information command output process, at the same time through hex threads number positioning to the specified thread stack information.

As you can see from the figure above, this thread executes code in Demo.java:4.

Finally, open the project and look at the source code of this class to see why it is consuming a lot of CPU here.

The demo.java file is the example code for this article, and it’s a beautiful endless loop.

You can also do this in two steps:

  • Jstack process PID > demo.txtExport process stack information todemo.txtfile
  • Open the stack file to search for the hexadecimal thread ID number

2. Top Execution result

The OPERATING system version is CentOS Linux release 7.3.1611 (Core), which can be queried by running the cat /etc/redhat-release command.

The top command is used to query the CPU and memory parameters. The top command is used to query the CPU and memory parameters.

Line 1: summary, the same result as the uptime command.

  • HH:mm:ss: indicates the current system time
  • Up x days, HH:mm: indicates the system running time from startup to current time (unit: minute).
  • X user: indicates the number of users currently logged in to the local PC
  • Load average: x.x, x.x, x.x: indicates the system load average within 1, 5, and 15 minutes
    • See this article for an introduction to load balancing values and algorithms
    • Of course, I didn’t finish reading the above article either (it was a bit difficult to read in English), but there is a paragraph in the article that is helpful to understand the system load average value:
      • If the average is 0.0, the system is idle
      • If the 1-minute average is greater than the 5-minute or 15-minute load value, then the load is increasing
      • If the 1-minute average is below the 5-minute or 15-minute load value, the load is decreasing
      • If these three values are higher than the number of cpus, you may have a performance problem (depending on the situation)

Line 2: Process information

  • X total: indicates the number of all processes
  • X running: indicates the number of running processes
  • X sleeping: indicates the number of sleeping processes
  • X zombie: indicates the number of zombie processes
    • Zombie process: When the child process ends before the parent process and the parent process does not reclaim the child process to release the resources occupied by the child process, the child process becomes a zombie process.

Line 3: CPU status

  • X.x us: percentage of CPU time consumed by user space
  • X. sy: percentage of CPU time consumed by the kernel space (system)
  • X. ni: indicates the percentage of CPU time consumed by niced user processes whose user mode priority has been adjusted
  • X. xwa: percentage of idle CPU time
  • X. hi: percentage of CPU time used to process hardware interrupts
  • X. si: Percentage of CPU time used to process software interrupts
  • X. st: percentage of time spent waiting for STEAL time

Line 4: Memory state

  • XXX total: indicates the total memory capacity
  • XXX free: indicates the free memory
  • XXX used: indicates the amount of used memory
  • XXX buff/cache: the amount of memory occupied by the cache and page cache

Row 5: Switch area status

  • XXX total: indicates the total memory capacity
  • XXX free: indicates the free memory
  • XXX used: indicates the amount of used memory
  • XXX Avail Mem: The amount of physical memory available for the next allocation of the process

Line 7 to end: Process details

  • PID: indicates the PID of a process
  • USER: indicates the USER name of the process owner
  • PR: indicates the priority of process scheduling
  • NI: Indicates the process priority (NICed value) from the user-space perspective. The lower the value, the higher the priority
  • VIRT: Virtual memory used by a process (KB). VIRT=SWAP+RES
  • RES: Physical memory used by the process (not swapped out) (KB), RES=CODE+DATA
  • SHR: Shared memory size (KB)
  • S: indicates the process status
    • D: A state of uninterruptible sleep
    • R: run
    • S: sleep
    • T: Trace/stop
    • Z: Zombie process
  • %CPU: indicates the CPU time occupied by a process
  • %MEM: percentage of physical memory occupied by a process
  • TIME+ : total CPU TIME occupied by a process (ms)
  • COMMAND: COMMAND used to run a process

3. Reference materials

  • Linux top command summary
  • Linux top command description
  • zombies
  • Linux “top” command: What are us, sy, ni, id, wa, hi, si and st (for CPU usage)?
  • Linux Memory Management

Copyright notice: this article is created by Planeswalker23, please bring the original link to reprint, thank you.