This is the sixth day of my participation in the First Challenge 2022
preface
- When a server is overloaded with CPU or memory, Java engineers need to monitor and locate problems such as thread deadlocks or memory overflow.
- This article documents instructions for viewing CPU, process/thread, AND JVM stack information on a Linux system for subsequent reference.
top
- Top is a command commonly used in Linux to view CPU, process, and memory performance indicators:
[root@localhost data/project]$top top-11:17:42 up 177 days, 19:34, 6 Users, Load Average: 6.58, 10.04, 9.68 Tasks: 145 total, 1 running, 144 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.2US, 10.0sy, 0.0ni, 70.5id, 0.0wa, 0.0hi, 0.3Si, 0.0st KiB Mem: 32780448 total, 4776344 free, 26326688 used, 1677416 buff/cache KiB Swap: 0 total, 0 free, 0 Use.5989168 Avail Mem PID USER PR NI VIRT RES SHR S %CPU % Mem TIME+ COMMAND 12241 root 20 0 8525184 1.8g 14780 S 46.0 5.8 1124:54 Java......Copy the code- The first line of system parameters:
- 11:17:42: indicates the current system time
- Up 177 days, 19:34: indicates the running time of the system
- 6 Users: indicates the number of login users of the system
- Load Average: 6.58, 10.04, 9.68: indicates the average load of 1 minute, 5 minutes, and 15 minutes
- The second line of process parameters:
- The system currently has 145 processes
- One of them is running
- 144 sleep states
- 0 pause states
- Zero zombie states
- Line 3 CPU parameters:
- Us: CPU usage in user mode is 19.2
- Sy: The kernel CPU ratio is 10.0
- Ni: Changes the CPU ratio of the NICE process to 0.0
- Id: Idle CPU ratio 70.5
- Wa: indicates that the CPU ratio waiting for I/OS is 0.0
- Hi: Hard interrupt CPU ratio is 0.0
- Si: soft interrupt CPU ratio 0.3
- St: indicates that the CPU usage of other VMS is 0.0
- Note: In a multi-CPU server, the average value of all cpus is displayed by default. Press 1 to display the proportion of each CPU.
[root@localhost data/project]$top top-11:17:42 up 177 days, 19:34, 6 Users, Load Average: 6.58, 10.04, 9.68 Tasks: 145 total, 1 running, 144 sleeping, 0 stopped, 0 zombie %Cpu0 : 21.6US, 12.2SY, 0.0ni, 65.9ID, 0.0wa, 0.0hi, 0.3si, 0.0st %Cpu1: 20.5us, 9.6SY, 0.0ni, 69.6ID, 0.0wa, 0.0hi, 0.3si, 0.0st %Cpu2: 25.6us, 10.2SY, 0.0ni, 63.8ID, 0.0wa, 0.0hi, 0.3si, 0.0st %Cpu3: 20.0US, 8.6SY, 0.0Ni, 71.0ID, 0.0wa, 0.0Hi, 0.3Si, 0.0st......Copy the code- Row 4/5 physical memory and disk memory parameters:
- Total: indicates the total memory capacity of 32780448 KB
- Free: free memory 4776344K
- Used: Used memory 26326688K
- Buff /cache: 1677416K of system cache /page cache
- Avail Mem: Virtual memory 5989168K
- Note: The system preferentially uses the physical memory. When the physical memory is insufficient, the system uses the disk memory. In general, the system tries to use the physical memory to improve performance.
- Line 6 Process details:
- PID: indicates the ID of a process
- USER: indicates an operation USER
- PR: indicates the process priority
- NI: nice value of the process
- VIRT: indicates the virtual memory occupied by a process
- RES: physical memory occupied by the process
- SHR: size of shared memory occupied by a process
- S: Process status: R=Running, S=interruptible sleeping, D=uninterruptible sleeping, T=Stopped, Z=zombie
- %CPU: indicates the CPU usage of the process
- %MEM: physical memory usage of a process
- TIME+ : indicates the total CPU usage TIME of a process
- COMMAND: Indicates the running COMMAND of a process
- Note: Enter capital P to sort the CPU usage, capital M to sort the physical memory usage, and capital H to display the thread information of the process
- Note: The top -c command can print the complete running command of a process
[root@localhost data/project]$top-c top-14:05:54 up 177 days, 22:22, 6 Users, Load Average: 9.02, 7.19, 6.26 Tasks: 146 total, 2 running, 144 sleeping, 0 stopped, 0 zombie %Cpu(s): 24.2US, 10.6SY, 0.0Ni, 64.8ID, 0.0wa, 0.0hi, 0.3Si, 0.0st KiB Mem: 32780448 total, 3990592 free, 26324892 used, 2464964 buff/cache KiB Swap: 0 total, 0 free, 0 Use.5981456 Avail Mem PID USER PR NI VIRT RES SHR S %CPU % Mem TIME+ COMMAND 12241 root 20 0 8525184 1.8g 14780 S 45.0 5.9 12011:04 Java -xmx2024m -xx :+UseG1GC -xx :MaxGCPauseMillis=100 -jar Java -demo.jar 2 root 20 00 0 S 0.0 0.0 [kthreadd] 3 root 00 0 S 0.0 0.0 14:58.90 [kthreadd /0] 5 root 0-20 00 S 0.0 0.0 00.00 [kworker/0:0H] ......Copy the code- Note: The top-HP PID command can view information about all threads of a process, where PID is a decimal thread ID
[root@localhost data/project]$ top -Hp 12241 top - 15:25:41 up 177 days, 23:42, 9 users, load average: Threads: 2468 total, 4 running, 2464 sleeping, 0 stopped, 0 zombie %Cpu(s): 16.5US, 19.4SY, 0.0Ni, 63.1 ID, 0.0wa, 0.0hi, 1.0Si, 0.0st KiB Mem: 32780448 total, 4689192 free, 25684352 used, 2406904 buff/cache KiB Swap: 0 total, 0 free, 0 Used.6612764 Avail Mem PID USER PR NI VIRT RES SHR S %CPU % Mem TIME+ COMMAND 13352 YOCd 20 0 8525184 1.8g 14780 S 4.8 5.9 0:24.00 Java 13422 yocd 20 0 8525184 1.8g 14780s 4.8 5.9 8:09.07 Java 13499 yocd 20 0 8525184 1.8g 14780s 4.8 5.9 0:08. 02 Java...Copy the codejstack
- Jstack: jstack-help jstack-help jstack: jstack-help jstack: jstack-help jstack-help
[root@localhost data/project]$ jstack -help
Usage:
jstack [-l] <pid>
(to connect to running process)
jstack -F [-m] [-l] <pid>
(to connect to a hung process)
jstack [-m] [-l] <executable> <core>
(to connect to a core file)
jstack [-m] [-l] [server_id@]<remote server IP or hostname>
(to connect to a remote debug server)
Options:
-F to force a thread dump. Use when jstack <pid> does not respond (process is hung)
-m to print both java and native frames (mixed mode)
-l long listing. Prints additional information about locks
-h or -help to print this help message
Copy the code- The most commonly used jstack command is jstack PID. The following parameters are optional and generally not required
- F: Force printing stack information when pid does not respond
- M: Print all stack information of Java, Native and C frameworks
- L: Prints additional information about thread locks, such as the list of java.util.Concurrent synchronizers, which causes the JVM to pause
[root@localhost data/project]$ jstack 12241 ...... "logback-9" #23 daemon prio=5 os_prio=0 tid=0x00007ff018b20800 nid=0x2fff waiting on condition [0x00007fef9597d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000081d49d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "logback-8" #22 daemon prio=5 os_prio=0 tid=0x00007fefa4003000 nid=0x2ffe waiting on condition [0x00007fefc49b4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000081d49d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Com. Alibaba. Nacos. Client. Worker. LongPolling. Fixed - 47.111.74.227 _18848-04 d07ec1-2439-4444-9498 - ff5df4d8e368" # 29 daemon prio=5 os_prio=0 tid=0x00007fef78003000 nid=0x3007 runnable [0x00007fef95177000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000000a1c82f30> (a java.io.BufferedInputStream) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) - locked <0x00000000a1c82f88> (a sun.net.www.protocol.http.HttpURLConnection) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) - locked <0x00000000a1c82f88> (a sun.net.www.protocol.http.HttpURLConnection) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.alibaba.nacos.client.config.impl.HttpSimpleClient.httpPost(HttpSimpleClient.java:119) at com.alibaba.nacos.client.config.http.ServerHttpAgent.httpPost(ServerHttpAgent.java:143) at com.alibaba.nacos.client.config.http.MetricsHttpAgent.httpPost(MetricsHttpAgent.java:64) at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateConfigStr(ClientWorker.java:386) at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateDataIds(ClientWorker.java:354) at com.alibaba.nacos.client.config.impl.ClientWorker$LongPollingRunnable.run(ClientWorker.java:521) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) .Copy the code- Jstack to find specific thread stack information command: jstack process id | grep -c 10 thread id (hexadecimal)
- Namely: jstack 12241 | grep -c 10 30 a5
- Note: In the preceding command, -c 10 indicates the 10 columns before and after the query, and -a 10 indicates the 10 columns after the query.
[root@localhost data/project]$ jstack 12241 | grep -C 10 30a5 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "ClientHouseKeepingService" #51 daemon prio=5 os_prio=0 tid=0x00007ff0185d3800 nid=0x30a7 in Object.wait() [0x00007fef7d4b5000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x0000000087d87890> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "AsyncAppender-Dispatcher-Thread-10" #49 daemon prio=5 os_prio=0 tid=0x00007ff01a0f9800 nid=0x30a5 in Object.wait() [0x00007fef7ebf8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.aliyun.openservices.shade.com.alibaba.rocketmq.logging.inner.LoggingBuilder$AsyncAppender$Dispatcher.run(LoggingBuil der.java:386) - locked <0x0000000087c18538> (a java.util.ArrayList) at java.lang.Thread.run(Thread.java:748) "Com. Alibaba. Nacos. Client. Worker. LongPolling. Fixed - 47.111.74.227 _18848-04 d07ec1-2439-4444-9498 - ff5df4d8e368" # 46 daemon prio=5 os_prio=0 tid=0x00007fef78005800 nid=0x3096 waiting on condition [0x00007fef7dcb7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)Copy the code- Part of the instruction information in the thread stack:
- Locked < address > target: owner of the monitor who successfully obtained an object lock using synchronized.
- Waiting to lock < address > Target: Fails to apply for an object lock using synchronized, and waits at a seat to turn in to a business.
- Target: After successfully applying for an object lock using synchronized, release the lock and waitin the waiting area.
- Parking to wait for
jmap
- Jmap can be used for JVM tuning monitoring to obtain information about memory allocation in a running process. Run the jmap -help command
[root@localhost data/project]$ jmap -help
Usage:
jmap [option] <pid>
(to connect to running process)
jmap [option] <executable <core>
(to connect to a core file)
jmap [option] [server_id@]<remote server IP or hostname>
(to connect to remote debug server)
where <option> is one of:
<none> to print same info as Solaris pmap
-heap to print java heap summary
-histo[:live] to print histogram of java object heap; if the "live"
suboption is specified, only count live objects
-clstats to print class loader statistics
-finalizerinfo to print information on objects awaiting finalization
-dump:<dump-options> to dump java heap in hprof binary format
dump-options:
live dump only live objects; if not specified,
all objects in the heap are dumped.
format=b binary format
file=<file> dump heap to <file>
Example: jmap -dump:live,format=b,file=heap.bin <pid>
-F force. Use with -dump:<dump-options> <pid> or -histo
to force a heap dump or histogram when <pid> does not
respond. The "live" suboption is not supported
in this mode.
-h | -help to print this help message
-J<flag> to pass <flag> directly to the runtime system
Copy the code-
The jmap commands are as follows:
- Heap: Displays Java heap details
- Histo: Displays object statistics in the Java heap
- Clstats: Displays class loading information
- Finalizerinfo: Prints information about objects waiting to complete
- Dump: Stores the Java heap in a file in hprof binary format
- F: -dump or -histo is used when pid does not respond. The mode live parameter is invalid
- J: Pass the parameters to the virtual machine running JMap
-
Instruction example 1: jmap-heap PID
[root@localhost data/project]$ jmap -heap 12241 Attaching to process ID 12241, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 25.211-B12 using Thread-local object allocation. Garbage-First (G1) GC with 4 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 2122317824 (2024.0MB) NewSize = 1363144 (1.2999954223632812MB) MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 2122317824 (2024.0MB) MaxNewSize = 1272971264 (1214.0MB) OldSize = 5452592 (5.1999969482421875MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 1048576 (1.0MB) Heap Usage: G1Heap: Regions = 2024 Capacity = 2122317824 (2024.0MB) Used = 656735728 (626.3119964599609MB) Free = 1465582096 30.94426859980044% used G1 Young Generation: Eden Space: Regions = 243 Capacity = 802160640 (765.0MB) Used = 254803968 (243.0MB) Free = 547356672 (522.0MB) 31.764705882352942% used Survivor Space: Regions = 8 Capacity = 8388608 (8.0MB) Used = 8388608 (8.0MB) Free = 0 (0.0MB) 100.0% Used G1 Old Generation: Regions = 376 Capacity = 495976448 (473.0MB) Used = 393543152 (375.31199645996094MB) Free = 102433296 (97.68800354003906MB) 79.3471451289558% Used 59711 Interned Strings evicted 6045272 bytes.Copy the code- Instruction instance 2: jmap-clSTATS PID
[root@localhost data/project]$ jmap -clstats 12241 Attaching to process ID 12241, please wait... Debugger Attached successfully. Server Compiler detected. JVM version is 25.211-B12 Finding Class Loader Instances.. .Copy the codejstat
- Jstat is a useful directive to monitor THE clasLoader, GC, and other JVM virtual machine information for JVM tuning monitoring.
- The jstat optional commands are as follows:
- Class: shows the number of loaded classes, space occupied, etc
- Compiler: Displays the number of COMPILED VMS
- Gc: displays VM GC statistics
- Gccapacity: collects statistics on object usage of young VMS, old VMS, and permanent VMS
- Gcnew: displays young GC information
- Gcnewcapacity: indicates the capacity information of the young generation when collecting GC statistics
- Gcold: Shows old age information when GC is displayed
- Gcoldcapacity: indicates the capacity of the old age when GC statistics are collected
- Gcpermcapacity: collects statistics about the permanent capacity during GC
- Gcutil: Collects heap capacity during GC
[root@localhost data/project]$jstat -GC 12241 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 0.0 8192.0 0.0 8192.0 304128.0 163840.0 214016.0 165812.3 156208.0 143758.2 17252.0 15374.6 7984 397.008 1 1.000 398.008 [root@localhost data/project]$jstat -gcutil 12241 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 43.92 78.05 92.03 89.12 8055 399.976 1 1.000 400.976 [root@localhost data/project]$jstat -gccapacity 12241 NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC 0.0 2072576.0 311296.0 0.0 8192.0 303104.0 0.0 2072576.0 215040.0 215040.0 0.0 1187840.0 156208.0 0.0 1048576.0 17252.0 8029 1Copy the code-
Output instruction details:
- SO: survivor 0 area memory usage ratio in the heap
- S0C: size of survivor 0 area in the heap
- S0U: used capacity size of survivor 0 area in the heap
- S1: Ratio of survivor 1 area memory usage in the heap
- S1C: size of survivor 1 area in the heap
- S1U: used capacity of survivor 1 area in the heap
- E: proportion of THE Edan area memory used in the heap
- EC: size of the Edan area in the heap
- EU: the used capacity of the Edan area in the heap
- O: memory usage ratio of the Old area in the heap
- OC: specifies the size of the Old area in the heap
- OU: the used capacity of the Old area in the heap
- P: Memory usage in the permanent area
- PC: Indicates the capacity of the permanent area
- PU: used capacity of the permanent area
- YGC: The number of young GC occurrences in the current JVM
- YGCT: Time in seconds for young GC to occur in the current JVM
- FGC: The number of full GC occurrences in the current JVM
- FGCT: time in seconds for current JVM full GC to occur
- GCT: The total time it takes for GC to occur in the current JVM, equivalent to YGCT+FGCT
- NGCMN: minimum initial size of young generation
- NGCMX: maximum capacity of young generation
- NGC: current capacity of young generation
- OGCMN: Initializes the size capacity of the old age
- OGCMX: Old age maximum capacity
- OGC: current capacity of the old age
- .
-
Reference links: JStack (view threads), Jmap (view memory), and Jstat (Performance Analysis)
The last
- The above commands are only a few examples. For more details about how to use these commands, please refer to the official website.
- Finally, learn modestly and make progress together