background
Recently, WE are preparing to launch Cassandra, and my colleague is doing some pressure tests of small ECS(8G). It’s easier to trigger OOM Killer when pressing, killing Cassandra. The problem is that my heap(Xmx) configured for 8GB is not high (about 6.5g) and there is enough space left for the system. OOM can only be triggered if the USAGE of Java off-heap memory exceeds expectations, resulting in an increase in RES.
The investigation process
0. It is suspected that there is a DirectBuffer leak somewhere, or a problem with the JNI library. 1. Out-of-heap memory overhead was tracked conventionally by Google PerfTools, but no significant anomalies were found. 2. Then use Java NMT to have a look, also found nothing abnormal.
3. DirectBuffer has nothing to do with DirectBuffer. At this point I noticed that the process virtual memory was very high, exceeding ECS memory. I suspect there’s something wrong here.
4. Run the /proc/ppid/smaps command to check the process memory address space distribution, and a large number of MMAP files are found. These files are Cassandra’s data files.
At this time, the virtual memory of these Mmap files is 2G, but the physical memory is 0 (because I have restarted before, the memory is too low to prevent the process hang up affecting the troubleshooting).
Obviously, the memory overhead of MMAP is not controlled by the JVM heap, that is, out-of-heap memory. If mmap file data is loaded from disk into physical memory (RES increase), Java NMT and Google Perftool are not aware of this, which is the kernel’s scheduling process.
5. Considering that the problem occurred during pressure measurement, I just need to read these files and observe whether RES will increase, by how much and why, so as to infer whether the problem is here. Simply read the imported data with the following command.
cassandra-stress read duration=10m cl=ONE -rate threads=20 -mode native cql3 user=cassandra password=123 -schema keysp
ace=keyspace5 -node core-3Copy the code
6. It can be observed that during the pressure test (SAR-B), the major Page fault is significantly increased because the data is actually loaded from disk into memory.
Also observe that the mmap file physical memory is increased to 20MB:
The final process RES rises to around 7.1g, an increase of about 600M:
If the pressure is increased (50 threads), the physical memory per MMAP file will increase from 20MB to 40MB
7.Root cause Cassandra identifies whether the system is 64 or 32 to decide whether to use MMAP or not. ECS are both 64, but in fact ECS of small size does not have much memory.
conclusion
1. The cause of the problem is that the mMAP to memory overhead is not taken into account. There are many specific adjustment methods. You can reduce the heap configuration for small ECS or disable the MMAP feature (disk_access_mode=standard) 2. It is recommended to use NMT to check the out-of-heap memory. It is relatively easy to use, and can be seen by configuring JVM parameters.
Author: Guo Zehui
The original link
This article is the original content of the cloud habitat community, shall not be reproduced without permission.