Source :Linux enthusiast

ID:LinuxHub

When a server based on Linux operating system runs, it will also represent various parameter information. Operations and system administrators are usually very sensitive to this data, but these parameters are also very important to developers, especially if your application is not working properly. These indicators can help locate and track problems quickly.

This is a simple tool to view the system parameters. Of course, many tools also analyze /proc and /sys data to work, but more detailed, professional performance monitoring and tuning may require more specialized tools (perf, systemTap, etc.) and techniques. After all, system performance monitoring is a university in itself.

CPU and memory classes

1.1 the top

➜ ~ top

The three values after the first line are the average load of the system in the previous 1, 5 and 15. It can also be seen that the system load is rising, stable and falling. When this value exceeds the number of CPU executable units, it indicates that the CPU performance has been saturated and become a bottleneck.

The second line statistics the task status information of the system. Running is naturally needless to say, including running on the CPU and will be scheduled to run; Sleeping is usually a task that waits for an event (such as an IO operation) to complete. The breakdown can include both interruptible and uninterruptible types. Stopped are tasks that have been paused, usually by SIGSTOP or Ctrl-Z on a foreground task; Zombie task. Resources are automatically reclaimed when a process terminates, but a task descriptor that contains an exit task needs to be released by the parent process. This process is defunct. Any such process should pay special attention to whether the program is poorly designed. The CPU usage varies according to the type:

  • (us) User: indicates the time occupied by the CPU in the user state with low NICE value (high priority) (NICE <=0). Normally, as long as the server is not idle, this is where most of the CPU time should be spent executing such programs

  • (SY) System: the time occupied by the CPU in kernel mode. The operating system drops from user mode to kernel mode through system call to execute specific services. In general, this value is small, but when the server is performing IO intensive, this value can be large

  • (ni) Nice: indicates the elapsed time (NICE >0) of the CPU in user mode with high NICE value (low priority) and low priority. By default, the newly started process nice=0 is not counted here unless you manually change the program’s nice value via renice or setPriority ()

  • (ID) Idle: indicates the time occupied by the CPU in the idle state (kernel Idle handler)

  • (wa) iowait: Indicates the time spent waiting for the I/O to complete

  • (HI) IRQ: The time taken by the system to process hardware interrupts

  • (si) Softirq: The time spent by the system to process soft interrupts. Remember that softirQs are categorized as softirQs, Tasklets, and Work Queues

  • (st) steal: This parameter is valid only in the case of a VM, because the VM CPU shares the same physical CPU. Therefore, this period of time indicates the time that the VM waits for the hypervisor to schedule the CPU. It also indicates the time that the hypervisor schedules the CPU to another CPU. The CPU resources in this period were “stolen”. This value is not 0 on my KVM’S VPS machine, but it is only 0.1. Can it be used to determine VPS overbooked?

In many cases, a high CPU usage can mean something, which also points to the appropriate troubleshooting for server CPU usage:

  1. When the user usage is too high, it is usually some individual process that occupies a large amount of CPU. In this case, it is easy to find the program through top. At this time, if the program is suspected to be abnormal, you can find out the hot call function through perF and other ideas to further investigation;

  2. When the system usage is too high, a large number of I/O operations (including terminal I/O operations) may cause a high CPU usage. For example, on file server and Database server servers, Otherwise (e.g. >20%) it is very likely that some parts of the kernel, driver module problems;

  3. When the NICE usage is too high, it is usually intentional. When the initiator of a process knows that some processes occupy a large number of cpus, he sets the NICE value to ensure that the CPU usage requests from other processes will not be overwhelmed.

  4. When the IOWait usage is too high, it usually means that the I/O operation efficiency of some programs is very low, or the performance of the corresponding DEVICE is so low that the read and write operation takes a long time to complete.

  5. If the irQ /softirq usage is too high, it is likely that some peripherals are faulty, causing a large number of IRQ requests. Check the /proc/interrupts file to find out more.

  6. When the steal usage is too high, the vm is overbooked.

The fourth and fifth lines are the physical and virtual memory (swap partition) information: total = free + Used + buff/cache. Now buffers and cached Mem information is added together, but buffers and cached

Mem’s relationship is unclear in many ways. Buffers and Cached are the Buffers and Cached fields of /proc/meminfo: Buffers is a block cache for raw disks. It mainly uses raw blocks to cache file system metadata (such as super block information). This value is usually small (20 MB). Cached is used to read and cache specific files to increase file access efficiency. It is used for file caching in a file system.

Avail Mem is a new parameter that indicates how much memory you can give to a new program without swapping. It is roughly the same as free + buff/cached, which proves the above statement. Free + Buffers + cached Mem is the actual physical memory available. Swap usage is not a serious parameter, but frequent swap in/out is not a good thing. This situation should be noted as it usually indicates a physical memory shortage.

Finally, there is a resource usage list for each program, where CPU usage is the sum of all CPU core usage. When a top is executed, the program itself will read a large number of /proc operations, so the top program itself will be top.

Top, while very powerful, is usually used by the console to monitor system information in real time. It is not suitable for monitoring system load information for long periods of time (days, months), and for short-lived processes it will miss statistics.

1.2 the vmstat

Vmstat is another common system check tool besides Top. Here is a screenshot of the system load I compiled with boost using -j4.R represents the number of runnable processes, and the data are roughly consistent. B is the number of uninterruptible sleep processes. SWPD indicates the amount of virtual memory used, which is the same as the value of top-swAP-used. As described in the manual, the number of buffers is usually much smaller than cached Mem. Buffers are usually 20M. Bi and BO of the I/O domain indicate the number of blocks received and sent to disks per second (blocks/s). In for the system domain indicates the number of system interrupts (including clock interrupts) per second, and cs indicates the number of context switches due to process switches.

The -j parameter is CPU Core or CPU Core+1. In both cases, the context switch has not changed, and the vmstat monitor is enabled. The context switch will only be significantly increased if the -j value is significantly increased, so we don’t need to worry too much about this parameter, although we haven’t tested the compile time. Context Switch >100000 (context switch>100000) context Switch >100000 (context switch>100000)

1.3 pidstat

If you want to track a process in detail, there’s nothing better than Pidstat — you can see stack space, page misses, active/passive switching, and so on. The most useful argument to this command is -t, which lists the details of each thread in the process.

-r: displays page miss errors and memory usage. A page miss error is a page that a program needs to access mapped in virtual memory space but has not been loaded into physical memory. The two main types of page miss errors are

  1. Minflt /s refers to minor faults. When the physical page to be accessed already exists in physical memory for some reason (such as shared pages, caching mechanisms, etc.) but is not referenced in the page table of the current process, the MMU only needs to set up the corresponding entries. This is a fairly small cost

  2. Majflt /s refers to the major faults. The MMU requests a free physical page from the currently available physical memory (if no free physical page is available, the MMU switches other physical pages to swap space to free the physical page) and loads data from the outside into the physical page. And set the corresponding entry, this cost is quite high, and there are several data levels of difference with the former

-s: stack usage, including the stack space reserved for threads by StkSize and the stack space actually used by StkRef. Use ulimit -s to find the default stack space of CentOS 6.x is 10240K, while the default stack space of CentOS 7.x and Ubuntu is 8196K

-u: indicates the CPU usage. The parameters are similar to those before

-w: The number of thread context switches, which are also subdivided into CSWCH /s active switches due to factors such as waiting resources, and NVCSWCH/S passive switches due to CPU time

It would be troublesome to photoshop pidstat every time to get the pid of the program, so the -c can specify a string, and if the string is included in Command, the program’s information will be printed out. -l Displays the complete program name and parameters ➜ ~ pidstat -w -t -c ailaw -l

In this case, pidstat is better than ps when viewing single, especially multi-threaded tasks!

1.4 other

In addition to hTOP, you can also use mpstat to check whether the workload of each Core on the SMP processor is balanced and whether some hot threads occupy the Core. ➜ ~ mpstat -p ALL 1

If you want to directly monitor the resources occupied by a certain process, you can use the top-u taozj command to filter out other user-irrelevant processes, or you can use the following method to select resources. The ps command can customize the entries to be printed:

while :; do ps -eo user,pid,ni,pri,pcpu,psr,comm | grep ‘ailawd’; sleep 1; done

If you want to clarify the inheritance relationship, the following common parameter can be used to display the process tree structure, which is much more detailed and beautiful than pstREE

➜ ~ ps axjf

Disk I/O

Iotop can intuitively display the real-time disk read rate of each process and thread. /dev/sda1: /dev/sda1: /dev/sda1: /dev/sda1: /dev/sda1: /dev/sda1 And the addition of the +fg parameter can also show additional file open flag.

2.1 iostat

➜ ~ iostat -xz 1

Iostat -xz 1 or SAR -d 1 are the most important parameters for the disk:

  • Avgqu-s: average length of waiting queues for I/O requests sent to the device. A value greater than 1 indicates device saturation for a single disk, except for logical disks with multiple disk arrays

  • Await (r_await, w_await) : average waiting time (ms) for each device I/O request operation, including the sum of the time the request is queued and served;

  • SVCTM: Average service time (ms) of an I/O request sent to the device. If SVCTM is very close to await, it means that there is almost no I/O wait and the disk performance is good. Otherwise, the disk queue waits long and the disk responds poorly.

  • %util: the usage of the device, indicating the percentage of I/O working time per second. The performance of a single disk degrades when %util>60% (‘ await ‘increases) and the device becomes saturated when %util approaches 100%, except for logical disks with multiple disk arrays;

Also, while disk performance may be low, it does not necessarily affect application response. Kernels often use I/O asynchronously, or read/write caching to improve performance, but this is limited by the physical memory limitations described above.

These parameters are also applicable to network file systems.

Third, network class

It goes without saying that the importance of network performance for server tools iptraf can intuitive reality nic to send and receive information speed, more concise convenient through the SAR – n DEV 1 can get similar throughput information, and network with the maximum rate information, such as a rate card gigabit nics, it is easy to see the utilization rate of equipment.

Generally, the transmission rate of a network card is not the most important information in network development, but the packet loss rate, retransmission rate, and network delay rate for specific UDP and TCP connections.

3.1 the netstat

➜ ~ netstat -s

Displays the total data of each protocol since the system was started. Although the parameter information is rich and useful, the network state information of the current system can only be obtained by the cumulative value unless there is a difference between the two runs, or the value change trend can be directly observed by the watch eye. So netstat is usually used to check port and connection information:

Netstat — all(a) — Numeric (n) — TCP (t) — UDP (U) — Timers (O) — Listening (L) — Program (P)

– Timers cancel reverse query of domain names to increase the display speed. More commonly used are

➜ ~ netstat -antp # Lists all TCP connections

➜ ~ netstat -nltp # list all local TCP listening sockets without the -a parameter

3.2 the SAR

The SAR tool is too powerful to handle CPU, disk, page exchange, etc. -n is mainly used to analyze network activity, although it also subdivides NFS, IP, ICMP, SOCK, etc. We only care about TCP and UDP. In addition to displaying the normal situation of segment and datagram sending and receiving, the following command also includes

TCP➜ ~ sudo sar -n TCP,ETCP 1 

  • Active/S: a locally initiated TCP connection, such as through connect(), whose status changes from CLOSED -> SYN-sent

  • Passive/S: Remote TCP connection, for example, accept(). The STATUS of the TCP connection changes from LISTEN -> SYN-rcvd

  • Retrans/S (tcpRetransSegs) : Indicates the number of TCP retransmissions per second. Retransmissions occur according to the TCP confirmation retransmission mechanism when the network quality is poor or packets are lost after the server is overloaded

  • Isegerr/S (tcpInErrs) : Error packets received per second (such as checksum failures)

UDP➜ ~ sudo sar -n UDP 1 

  • Noport /s(udpNoPorts) : The number of datagrams received per second but not on the specified destination port of the application

  • Idgmerr /s(udpInErrors) : The number of datagrams received by the local machine but not distributed

Of course, these data can explain network reliability to some extent, but they are only meaningful when combined with specific business requirement scenarios.

3.3 tcpdump

Tcpdump has to be a good thing. Wireshark: Wireshark: wireshark: Wireshark: Wireshark: Wireshark

The references in the appendix give the following ideas: Restore the environment and use the tcpdump to capture packets. When problems occur again (such as log display or a status display), packet capture can be completed. The tcpdump has -c / -w parameters to limit the size of the file stored in captured packets. When this limit is reached, the saved packet data is automatically rotated, so the total number of captured packets is controllable. After the packet down the line, with wireshark think how to see how to see, not happy zai! Tcpdump does not have a GUI interface, but the packet capture function is not weak. You can specify filtering parameters such as network adapter, host, port, and protocol. The captured packets are complete and time-stamped, so the online application can analyze packets easily.

The following is a small test: When Chrome starts, it automatically sends three connections to the Webserver. Because the DST port parameter is restricted, the server response packet is filtered out. The process of establishing a connection through SYNC and ACK is still obvious! When using the tcpdump, you need to configure filtering conditions as soon as possible to facilitate the following analysis. On the one hand, after the tcpdump is enabled, the performance of network adapters and the system is affected, which affects the performance of online services.

In this paper, finished!

Source: taozj

https://taozj.org/201701/linux-performance-basic.html