Linux Tool Performance Tuning Series 3: Locating swap problems

First, prepare knowledge

1.1 What is swap

Swap when we refer to it as a noun, it can be either a partition or a file. It is a place in the operating system to store data that has been swapped out of memory. When we refer to a verb, we refer to the action of exchanging data from physical memory to swap.

1.2 为什么会swap

(1) When the physical memory is insufficient, part of the memory will be swapped to swap partition (accompanied by high IO at this time) according to a specific algorithm. But not all memory can be swapped to swap. (2) The KSWAPD process periodically checks the memory. If it finds that the memory is higher than the water level, swap will be triggered. This is to prevent the system from having little remaining memory and sudden large memory application. I’m not going to go into this, but I’ll cover it later.

1.3 What exactly is Swap

First of all, memory management divides memory into active and inactive, and the mappings used by the process user space include anonymous mappings (ANon) and file mappings (File). All of them have active anon,inactive anon,active file,inactive file. For file mapping, because it is a file in the disk space, it will not be swap. When it needs to be released, dirty data is directly written back to the disk, and other data can be directly released. Swap memory to swap. Inactive data must be swapped, so inactive memory is the primary memory swapped. So for the operating system, when I need to reclaim memory, you say it is better for file mapping or anonymous mapping, which involves a parameter: swapiness

1.3.1 swapiness

Swapiness is set to the value of /proc/sys/vmp/swappiness when memory reclamation is preferred to recycle file mapping or anonymous mapping. For swapiness=100, the weight between the two is consistent. The smaller the value is, the more likely it is to recycle the file mapping. However, if it reaches the high water level of the system, it will still swap, unless the system swap is directly closed by swapoff-a or other means.

1.3 The quality of Swap

The nice thing about swap is that when memory is low, you can swap some of it out without triggering oom-killer. It’s better to run slowly than never. The disadvantage of swap is that high I/OS are triggered during swap and system performance deteriorates. When we do a poor job of isolation, it affects the performance of other applications.

Second, tool selection

A single tool can be used for many purposes, but this article is only for swap

Third, case analysis

3.1 Applications Always Apply for Memory

This case is written using Golang, in an infinite loop, each time the loop requests memory, and does not release, and then releases memory after a certain number of times, waiting for GC, and then continues, the code and documents are archived in: archive.

3.1.1 Running procedures and analysis

(1) operation

root@szdc-calic-2-6:/www/linuxperformancetool/swap# ./swapexample1
Copy the code

Since all the commands mentioned above can be used for analysis, you can use them according to your preferences. Here I use top,vmstat and pidstat to analyze, you can open several terminals together to see.

(2) Vmstat is used first, as can be seen from the following, when the program occupies more and more memory, there is a high swap IO and block IO, think about, why are these two at the same time higher?

root@szdc-calic-2-6:~# vmstat -a 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free  inact active   si   so    bi    bo   incs us sy id wa st 1 0 1240208 18816440 715216 12702360 0 0 1 18 0 0 0 0 99 0 0 1 0 1240208 18086504 715220 13431988 0 0 0 0 2223 3192 14 96 0 0 1 0 1240208 17343176 715220 14173444 0 0 60 1906 3099 14 95 0 0 1 0 1240208 16618392 715260 14897140 00 0 60 2351 3552 14 96 00 1 0 1240208 15883220 715280 15632572 00 0 2187 3547 14 96 00 02 1240300 15276012 806532 16151868 0 920 2088 2365 3373 1 3 95 200 2 1265792 15270100 2023920 14946200 56 25500 100 27536 5102 9189 11 93 5 03 1265784 15274476 2023856 14944732 40 0 40 2572 2162 2425 0 0 90 10 0 0 9 1287036 15276116 2022816 14944232 88 21276 88 22228 2074 3136 0 0 79 21 0 0 9 1287020 15273676 2024776 14946436 92 0 92 2288 3563 5822 0 0 79 20 02 3 1286912 15271708 2024844 14946532 128 0 128 1432 2989 4907 0 0 84 16 0 0 3 1286912 15271692 2024572 14946520 0 0 0 2344 3098 4543 0 0 87 13 0 0 4 1312096 15265788 2022444 14951064 0 25188 0 26628 4946 9752 1 1 90 90Copy the code

After the system problem is found, we need to locate the problem. Here, we can use top, pidstat, ioTOP and other tools to locate the problem. I directly use top and press F to select swap.

Top-00:52:38 Up 253 days, 14:23, 3 Users, Load Average: 6.44, 2.60, 1.23 Tasks: 359 total, 1 running, 358 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.6us, 0.6SY, 0.0ni, 67.0 ID, 31.7wa, 0.0hi, 0.0Si, 0.0st KiB Mem: 32895096 total, 236100 free, 32011520 used, 647476 buff/cache KiB Swap: 31250428 total, 28358956 free, 2891472 Use.261616 Avail Mem PID USER PR NI VIRT RES SHR S %CPU % Mem TIME+ COMMAND SWAP 45466 root 20 0 10.076g 21624 2520 S 12.9 0.1 7881:36 etcd 19636 146 root 20 0 0 0 D 9.2 0.0 2:28.44 kswapd0 0 14179 root 20 0 31.136g 0.029t 4316s 3.3 94.8 0:31.86 main1 1.37g 147 root 20 0 0 0 0 S 1.7 0.0 1:52.23 KSWAPD1 0 10737 root 20 0 3067756 60688 5968s 0.7 0.2 3364:41 dockerd 60380 10750 root 20 0 2898452 38724 3096 S 0.7 0.1 629:49.22 docker-containe 79288Copy the code

As can be seen from the change of top, the swap of processes with PID =14179 keeps increasing and their memory usage is getting higher and higher. Although other processes have more swap, their memory usage does not increase. It can be judged that this process is the cause of swap. So, let’s take it a step further.

Using pidstat to judge the IO status of the PID, it can be seen that there is a large IO

root@szdc-calic-2-6:~# pidstat -p 14179 -d 1Linux 4.0-87- Generic (szdc-Calic-2-6.meitu-inc.com) Wednesday, December 12, 2018 _X86_64_ (24 CPU) 12:55:40 CST UID PID kB_rd/s kB_wr/s kB_ccwr/s IOdelay Command 12:55:41 CST 0 14179 71920.00 0.00 0.00 0 Main1 12:55:42 CST 0 14179 85664.00 0.00 0.00 0 Main1 12:55:42 CST 0 14179 85664.00 0.00 0.00 0 Main1 12:55:44 CST 0 14179 78128.00 0.00 0.00 0 MAIN1 12:55:45 CST 0 14179 69660.00 0.00 0 MAIN1 12:55:46 CST 0 14179 59892.00 0.00 0.00 0 main1Copy the code

3.1.2 problem

These questions will not be directly answered, if you really can’t think of it, you can make an issue or comment below the archive project

(1) Why in the example, simply applying for memory will cause swap IO and block IO to increase at the same time? (2) In the example, it is clear that there is still unused memory, but a large amount of memory has been frequently swap and reclaimed. (3) During golang gc, will the memory that has been swap out be swap into physical memory before GC? (4) What are the buffers and caches in the previous section? (Anonymous page or file mapping)

Iv. References

1, linuxperf.com/?p=142

2, cloud.tencent.com/developer/a…