Every once in a while a Docker Java application will disappear without any JVM error logs running on a virtual machine.

If the vm is 10GB, docker will allocate unlimited memory by default.

Reasons for positioning

To execute on the server: dmesg | grep Java, indeed as expected have data

/var/log/messages: oom-killer,

See related information: Java PID information,

memory: usage 2047696kB, limit 2047696kB, failcnt 23543
memory+swap: usage 2047696kB, limit 9007199254740991kB, failcnt 0
......
Free swap  = 0kB
Total swap = 0kB
......
Memory cgroup out of memory: Kill process 18286 (java) score 933 or sacrifice child
Copy the code

The Java process was killed by the Linux OOM because the OOM score was too high: 933. The higher the oOM_score, the more likely it is to be killed by OOM Killer. Use the following script to check oom Score:

# vi oomscore.sh #! /bin/bash for proc in $(find /proc -maxdepth 1 -regex '/proc/[0-9]+'); do printf "%2d %5d %s\n" \ "$(cat $proc/oom_score)" \ "$(basename $proc)" \ "$(cat $proc/cmdline | tr '\0' ' ' | head -c  50)" done 2>/dev/null | sort -nr | head -n 10 # chmod +x oomscore.shCopy the code

The solution

Add -m 4000m to docker run. If not, default swap is 8000M.

Solutions on the network: Disable OOM solutions Prevent important system processes from triggering the OOM mechanism and being killed: You can temporarily disable the OOM mechanism of the Linux kernel by setting /proc/pid/oom_adj to -17. The kernel calculates a score for each process using a specific algorithm to determine which process to kill. The oom score for each process can be found in /proc/pid/oom_score. SSHD and some management agents are generally protected in our operation and maintenance process.

To protect a process from being killed by the kernel, do this:

# echo -17 > /proc/$PID/oom_adj
Copy the code

To prevent SSHD from being killed, do this:

# pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; doneCopy the code

Add a timed task to your scheduled task to make it even safer:

#/etc/cron.d/oom_disable */1**** root pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; doneCopy the code

To avoid a restart failure, write to /etc/rc.d/rc.local

echo -17 > /proc/$(pidof sshd)/oom_adj
Copy the code

As for why -17 is used instead of another value (the default value is 0), this is defined by the Linux kernel. In Linux – 3.3.6 version of the kernel source code, for example, path for Linux – 3.6.6 / include/Linux/oom. H, read the kernel source known oom_adj can tone value for 15-16, including 15-16 minimum, largest – 17 to ban the use of oom. Oom_score is calculated as 2 to the power of n, where n is the oOM_adj value of the process, so the higher the oom_score is, the more the kernel kills it first.

You can also disable the OOM mechanism by modifying kernel parameters

# sysctl -w vm.panic_on_oom=1 vm.panic_on_oom=1 vm.panic_on_oom=1 //1 indicates that OOM is enabledCopy the code

Three things to watch ❤️

If you find this article helpful, I’d like to invite you to do three small favors for me:

  1. Like, forward, have your “like and comment”, is the motivation of my creation.

  2. Follow the public account “Java rotten pigskin” and share original knowledge from time to time.

  3. Also look forward to the follow-up article ing🚀

Reference: club.perfma.com/article/185…