Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”

Description of disk performance indicators

Common disk performance indicators include: Usage, saturation, IOPS, throughput, and response time.

  • Usage: indicates the percentage of I/O processing time. High utilization, such as more than 80%, usually indicates a disk I/O performance bottleneck.
  • Saturation refers to how busy the disk is in processing I/O. Excessive saturation means that the disk has a serious performance bottleneck. When the saturation is 100%, the disk cannot accept new I/O requests.
  • Input/Output Per Second (IOPS) is the number of I/O requests Per Second.
  • Throughput refers to the volume of I/O requests per second, that is, disk I/O traffic per second, disk write plus read data. The unit is BPS.
  • Response time refers to the interval between sending an I/O request and receiving a response.

Relationship between IOPS and throughput

I/O Throughput per second = IOPS x Average I/O SIZE. The larger THE I/O SIZE is, the higher the IOPS is, and the higher the I/O throughput per second is. Therefore, we assume that the higher the IOPS and throughput, the better. In fact, both parameters have their maximum values for a disk, and there is a relationship between the two parameters.

IOPS can be divided into the following indicators:

  • Toatal IOPS: Disk IOPS in the case of mixed read/write and sequential random I/O loads. This parameter is most consistent with actual I/O loads. Most applications pay attention to this parameter.
  • Random Read IOPS: 100% Random Read IOPS.
  • Random Write IOPS: 100% Random Write IOPS.
  • Sequential Read IOPS, 100% IOPS for Sequential Read loads.
  • Sequential Write IOPS, 100% IOPS for Sequential Write loads.

Common commands for observing I/O performance indicators

iostat

Iostat, which monitors disk activity on the system. It features disk activity statistics, as well as CPU usage. Iostat also has a weakness in that it does not provide in-depth analysis of a process, only of the system as a whole.

Command parameters:

  • -c: displays the CPU usage
  • -d: displays the disk usage
  • -n: Displays disk array (LVM) information
  • -n: Displays NFS usage
  • -k: Displays the value in KB
  • -m: Displays the information in the unit of M
  • -t: reports the number of characters read and written to the terminal and CPU information per second
  • -v: displays the version information
  • -x: displays detailed I/O information
  • -p [disk] : displays information about disks and partitions

Remark:

If %iowait is too high, the disk has an I/O bottleneck. If %idle is too high, the CPU is idle.

If the %idle value is high but the system responds slowly, the CPU may wait to allocate memory. In this case, expand the memory capacity.

If the %idle value is lower than 10 for a long time, the CPU processing capacity of the system is relatively low, indicating that the CPU is the most important resource in the system.

CPU attribute values:

  • %user: indicates the percentage of time the CPU is in user mode
  • %nice: indicates the percentage of time the CPU is in user mode with the nice value
  • %system: indicates the percentage of time the CPU is in system mode
  • % ioWAIT: the percentage of time the CPU waits for input and output to complete
  • % STEAL: The percentage of the virtual CPU’s unconscious wait time while the hypervisor maintains another virtual processor
  • %idle: indicates the percentage of idle CPU time

Disk attribute values:

  • Device: indicates the disk name
  • TPS: number of I/O requests sent per second
  • Blk_read/s: indicates the number of blocks read per second
  • Blk_wrtn/s: indicates the number of written blocks per second
  • Blk_read: indicates the total number of blocks read
  • Blk_wrtn: indicates the total number of written blocks

Disk I/O:

  • RRQM /s: indicates the number of merge read operations per second. Namely rmerge/s
  • WRQM /s: indicates the number of merge write operations per second. Namely wmerge/s
  • R /s: indicates the number of read I/ OS completed per second. The Rio/s
  • W /s: number of complete write I/ OS per second. The wio/s
  • RkB /s: indicates the number of K bytes read per second. It is half the size of RSECT /s because each sector is 512 bytes.
  • WkB /s: number of K bytes written per second. Half of wsECT /s
  • Avgrq-sz: Average data size (sector) per device I/O operation
  • Avgqu-sz: indicates the average I/O queue length
  • Rsec /s: indicates the number of read sectors per second. Namely rsect/s
  • Wsec /s: indicates the number of sectors written per second. Namely wsect/s
  • R_await: The average time required for each read operation, including not only the read operation time of the disk device, but also the time spent waiting in the kernel queue
  • W_await: The average time required for each write operation, including not only the write time of the disk device, but also the time spent waiting in the kernel queue
  • Await: average wait time per device I/O operation (ms)
  • SVCTM: Average service time of each device I/O operation (ms)
  • %util: What percentage of a second is spent on I/O operations. That is, the percentage of CPU consumed by I/ OS

Remark:

If %util is close to 100%, too many I/O requests are generated, the I/O system is fully loaded, and there may be a bottleneck on the disk.

If SVCTM is closer to await, I/O has almost no wait time;

If await is much larger than SVCTM, it indicates that the I/O queue is too long and the I/O response is too slow, and necessary optimization is required.

If AVGQU-SZ is relatively large, it also indicates that equivalent IO is waiting.

Example: Check the CPU status

iostat -c 1 1
Copy the code

Example: View disk usage and response time

-x displays detailed information. Iostat -d -x -k 1 1Copy the code

Other examples

The interval is 1 second, a total of 5 times
iostat 1 5

Display device statistics every 2 seconds
iostat -d 2

Display device statistics every 2 seconds for a total of 3 times
iostat -d 2 3

Display the extended statistics of sDA and SDB every 2 seconds for three times
iostat -x sda sdb 2 3

Display sDA and all partition statistics every 2 seconds, total output 3 times
iostat -p sda 2 3

Display all information in M
iostat -m
Copy the code

pidstat

Pidstat is used to monitor the usage of system resources, such as CPUS, memory, threads, and device I/OS, of all or specified processes. If you run pidstat for the first time, the statistics are displayed since the system is started. If you run pidstat, the statistics are displayed since the last time. Users can obtain desired statistics by specifying the number and time of statistics.

Command parameters:

  • -u: displays CPU usage statistics of each process
  • -r: displays the memory usage statistics of each process
  • -d: displays THE I/O usage of each process
  • -p: specifies the process NUMBER
  • -w: displays the context switchover of each process
  • -t: displays additional information in addition to the statistics of the thread that selects the task
  • – T {TASK CHILD | | ALL} : TASK report says independent TASK, the CHILD ALL threads keyword process under the said report statistics. ALL indicates reporting individual tasks and ALL threads under the task. Note: Global statistics for tasks and child threads are independent of the pidstat option. These statistics do not correspond to the current statistical interval and are collected only when the child thread kills or completes
  • -v: displays the version number
  • -h: Shows all activities on one line so that other programs can easily parse them
  • -i: indicates the CPU usage/number of cores of a task in the SMP environment
  • -l: displays the command name and all parameters

Example: View the CPU usage of all processes

Pidstat -u -p ALL Linux 3.10.0-957.el7.x86_64 (VM-75-65) 04/24/2020_x86_64_ (4 CPU) 02:10:43 PM UID PID %usr %system %guest %CPU CPU Command 02:10:43 PM 01 0.00 0.01 0.00 0.01 0 Systemd 02:10:43 PM 02 0.00 0.00 0.00 0.00 1 kthreadd 02:10:43 PM 0 3 0.00 0.06 0.00 0.06 0 ksoftirqd/0 02:10:43 PM 0 7 0.00 0.00 0.00 0 Migration /0 02:10:43 PM 0 8 0.00 0.00 0.00 3 Rcu_bh 02:10:43 PM 0 9 0.00 0.16 0.00 0.16 2 Rcu_SCHed 02:10:43 PM 0 10 0.00 0.00 0.00 0 Lru-add-drain 02:10:43pm 0 11 0.00 0.00 0.00 0.00 0 watchdog/0Copy the code

Attribute value description:

  • PID: indicates the ID of a process
  • %usr: the percentage of CPU occupied by a process in user space
  • %system: indicates the percentage of CPU occupied by processes in kernel space
  • %guest: indicates the percentage of CPU usage by processes on the VM
  • %CPU: indicates the percentage of CPU occupied by a process
  • CPU: indicates the CPU number of the processing process
  • Command: indicates the Command corresponding to the current process

Example: View the memory usage of a specified process

Specify PID 29468 process memory usage, display once per second, display four times! Pidstat -r is the memory usage of the entire process.

Pidstat -r -p 29468 1 4 Linux 3.10.0-957.el7.x86_64 (VM-75-65) 04/24/2020_x86_64_ (4 CPU) 02:16:00 PM UID PID minflt/s Majflt /s VSZ RSS %MEM Command 02:16:01 PM 0 29468 0.00 413040 4104 0.03 Docker-Containe 02:16:02 PM 0 29468 0.00 0.00 413040 4104 0.03 Docker-containe 02:16:03 PM 0 29468 0.00 0.00 413040 4104 0.03 Docker-Containe 02:16:04 PM 0 29468 0.00 0.00 413040 4104 0.03 Docker-Containe Average: 0 29468 0.00 0.00 413040 4104 0.03 Docker-Containe Average: 0 29468 0.00 0.00 413040 4104 0.03 Docker-ContaineCopy the code

Attribute value description:

  • PID: indicates the process id
  • Minflt/s: Minor errors that occur per second in the task and do not require pages to be loaded from disk
  • Majflt/ S: Major errors that occur per second in the task, requiring pages to be loaded from disk
  • VSZ: virtual address size, virtual memory usage KB
  • RSS: Resident collection size, non-swap memory usage KB
  • Command: Task Command name

Pidstat -r is the memory usage of the entire process.

Example: Display the I/O usage of each process

Pidstat -d Linux 3.10.0-957.el7.x86_64 (VM-75-65) 04/24/2020_x86_64_ (4 CPU) 02:25:35 PM UID PID kB_rd/s kB_wr/s KB_ccwr /s Command 02:25:35pm 0 1 6.42 242.84 0.72 systemd 02:25:35pm 0 2432 0.00 0.00 xfSAILd /sda3 02:25:35pm 0 2501 0.00 0.00 Systemd-Journal 02:25:35pm 0 2522 0.00 0.00 lvmetad 02:25:35pm 0 2539 0.35 0.00 0.00 Systemd-udevd 02:25:35 PM 0 4863 0.00 0.17 0.00 auditd 02:25:35 PM 32 4893 0.00 0.00 0.00 rpcbind 02:25:35 PM 0 4894 0.00 0.00 SmartD 02:25:35 PM 0 4896 0.00 0.00 IRqBalanceCopy the code

Attribute value description:

  • PID: indicates the ID of a process
  • KB_rd /s: KB read from disk per second
  • KB_wr /s: Indicates the number of bytes written to the disk per second
  • KB_ccwr /s: KB written to the disk for task cancellation. This occurs when the task truncates the dirty Pagecache.
  • COMMAND: indicates the task COMMAND name

 

Example: Context switching for a process

Pidstat -w Linux 3.10.0-957.el7.x86_64 (VM-75-65) 04/24/2020_x86_64_ (4 CPU) 02:26:50 PM UID PID CSWCH /s NVCSWCH /s Command 02:26:50 PM 01 0.36 0.00 Systemd 02:26:50 PM 02 0.01 0.00 kthreadd 02:26:50 PM 0 3 10.42 0.00 ksoftirqd/0 02:26:50 PM 0 5 0.00 kworker/ 0:0h 02:26:50 PM 0 7 1.77 0.00 Migration /0 02:26:50 PM 0 8 0.00 0.00 rcu_bh 02:26:50 PM 0 9 114.23 0.00 rcu_sched 02:26:50 PM 0 10 0.00 lru-Add-drain 02:26:50 PM 0 11 0.25 0.00 watchdog/0 02:26:50 PM 0 12 0.25 0.00 watchdog/1Copy the code

Attribute value description:

  • PID: indicates the ID of a process
  • CSWCH /s: number of active task context switches per second
  • NVCSWCH /s: number of passive task context switches per second
  • Command: indicates the name of a Command

 

Example: Displays thread statistics for a particular process

Pidstat -p 12920 -t Linux 2.6.32-573.el6.x86_64 (VM-75-64) 04/24/2020_x86_64_ (8 CPU) 02:20:23 PM TGID TID %usr %system %guest %CPU CPU Command 02:20:23 PM 12920-0.00 0.00 0.00 3 Java 02:20:23 PM - 12920 0.00 0.00 3 | __java 02:20:23 PM - 12922 0.00 0.00 0.00 0.00 6 | __java 02:20:23 PM - 12923 0.00 0.00 0.00 0.00 0 | __java 02:20:23 PM - 12924 0.00 0.00 0.00 0.00 0 | __java 02:20:23 PM - 12925 0.00 0.00 0.00 0.00 2 | __java 02:20:23 PM - 12926 0.00 0.00 0.00 0.00 5 | __java 02:20:23 PM - 12927 0.00 0.00 0.00 0.00 4 | __java 02:20:23 PM - 12928 0.00 0.00 0.00 0.00 1 | __java 02:20:23 PM - 12930 0.00 0.00 0.00 0.00 5 | __java 02:20:23 PM - 12931 0.00 0.00 0.00 0.00 5 | __java 02:20:23 PM - 12932 0.00 0.00 0.00 0.00 3 | __java 02:20:23 PM - 12933 0.00 0.00 0.00 0.00 4 | __java 02:20:23 PM - 12934 0.00 0.00 0.00 0.00 4 | __java 02:20:23 PM - 12935 0.00 0.00 0.00 0.00 5 | __java 02:20:23 PM - 12936 0.00 0.00 0.00 0.00 3 | __java 02:20:23 PM - 12938 0.00 0.00 0.00 0.00 5 | __java 02:20:23 PM - 12939 0.00 0.00 0.00 0.00 1 | __java 02:20:23 PM - 12940 0.00 0.00 0.00 0.00 6 | __java 02:20:23 PM - 12941 0.00 0.00 0.00 0.00 2 | __java 02:20:23 PM - 12943 0.00 0.00 0.00 0.00 6 | __javaCopy the code

Attribute value description:

  • TGID: representation of the main thread
  • TID: indicates the id of a thread
  • %usr: the percentage of CPU occupied by a process in user space
  • %system: indicates the percentage of CPU occupied by processes in kernel space
  • %guest: indicates the percentage of CPU usage by processes on the VM
  • %CPU: indicates the percentage of CPU occupied by a process
  • CPU: indicates the CPU number of the processing process
  • Command: indicates the Command corresponding to the current process

iotop

Iotop is a top-class tool used to monitor disk I/O usage. It can monitor disk I/O information used by which program.

Command parameters:

  • -o, –only Displays only the processes or threads that are generating I/ OS. In addition to passing parameters, you can press O during runtime to take effect.
  • -b, — Batch indicates the non-interactive mode, which is used to record logs.
  • -n NUM, –iter=NUM Sets the number of times to monitor. Useful in non-interactive mode.
  • -d SEC, –delay=SEC Sets the interval for each monitoring. The default value is 1 second, and non-shaping data such as 1.1 is accepted.
  • -p PID, — PID =PID Specifies the process/thread to monitor.
  • -u USER, — USER =USER Specifies to monitor I/ OS generated by a USER.
  • -p, –processes displays only processes. By default, IOTOP displays all threads.
  • -a, — Accumulated I/O, not bandwidth.
  • Use kilobytes, not person-friendly units. Scripting is useful in non-interactive mode.
  • -t, –time plus timestamp, non-interactive non-mode.
  • -q, –quiet Disables the first few lines in non-interactive mode. There are three ways to specify this, in which -q indicates that the column name is displayed only on the first monitor, -qq indicates that the column name is never displayed, and -qqq indicates that the I/O summary is never displayed.

Interactive keys:

Similar to the top command, IOtop also supports the following interactive keys.

  • Left and right arrow keys: Change the order.
  • R: Reverse sort.
  • O: Switch to the option –only.
  • P: Switch to –processes.
  • A: Switch to — Debt option.
  • Q: Exit.
  • I: Changes the priority of the thread.

Show only the processes or threads that are generating I/O (interactive)

In addition to passing parameters, you can press O during runtime to take effect.

iotop  -o
Copy the code

Refresh at intervals (interactive)

Output 5 times every 2 seconds.

iotop  -d 2 -n 5
Copy the code

Refresh at intervals, output to screen (non-interactive)

Output 5 times every 2 seconds. It can also output log text to monitor I/O information in a certain period.

iotop -botq -n 5 -d 2 
Copy the code

Output disk I/O information for the process whose PID is 8382 (non-interactive)

iotop -botq -p 8382
Copy the code