Linux command pick-ups – My process is gone

Original: Coding diary (wechat official ID: Codelogs), welcome to share, reprint please reserve the source.

Introduction to the

Programmer but every work time is a bit longer, total meeting encounters some strange thing, for instance whenever you come off work, service hangs, next business classmate sought all sorts of come over, it seems that business and service program cannot leave you same. And when you log in the machine to troubleshoot the problem, and found that even the process of the machine is gone, the heart clattle panic god, process how disappeared?

Background tasks

For those who are new to Linux, they may not know the concept of foreground task and background task in Shell. Let me introduce this first, as follows:

1. When the command is executed directly, the foreground task is started and added after the command&Symbol to start a process as a background task.
2. You can use Jobs to query all background tasks started under the current shell.
3. If the process is a foreground task, pressCtrl+zYou can make it a background task, but the task also becomes paused.
4. usebgYou can make a suspended task run.
5. usefgYou can make a background task a foreground task, or a running task if it is paused.

Here’s an example:

Start the foreground process, then press Ctrl+ Z to make it a background process
$ java -jar app.jar
^Z
[1]+  Stopped                 java -jar app.jar

The background process is in the Stopped state
$ jobs -l
[1]+ 19316 Stopped                 java -jar app.jar

After using bg, you can see that the task is Running
$ bg %1
[1]+ java -jar app.jar &
$ jobs
[1]+  Running                 java -jar app.jar &

If fg is used again, you will find that the task becomes the foreground task
$ fgThe % 1# Use the ampersand symbol directly, which is the background task and Running state
$ java -jar app.jar &
[1] 19620
$ jobs
[1]+  Running                 java -jar app.jar &
Copy the code

Nohup and disown

However, if you start the process using the above method, when you turn off the computer after work, the SSH terminal will lose connection and your Java process will be killed. The process started in shell is all the child process of shell process. When SSH is disconnected, shell process will send SIGHUP signal to its child process, which will kill the child process started in shell, including background process.

The nohup command can solve this problem by ignoring the SIGHUP signal and allowing the process to survive as follows:

$ nohup java -jar app.jar &
Copy the code

If you have not previously started a process with nohup, you can use the disown command to cause the process to ignore the SIGHUP signal as follows:

1. useCtrl+zMake it run in the background while the process becomes paused.
2. usejobs -lTo query the jobid that was just transferred to the background task, such as 1, usebg %1Make it run.
3. usedisown -h %1Make the background task ignoredSIGHUPThe signal.

In addition to NohUP, pseudo-terminal software such as TMUx and Screen can also be used to prevent the process from being killed when the terminal is disconnected, as follows:

# Ubuntu install TMUx
$ sudo apt install tmux

Create a pseudo-terminal session named app
$ tmux new -s app

Start the process, notice that it is now inside the TMUX dummy terminal, this is the same as normal operation
To exit the pseudo-terminal, press Ctrl+ B then D, not Ctrl+ C
$ java -jar app.jar

# list all pseudo terminal sessions
$ tmux ls

Reattach to the app session so that you can see how the task is running
$ tmux attach -t app
Copy the code

In addition to safely starting background processes, TMUx can also realize advanced functions such as multi-window, split screen, etc. You can view them with MAN TMUx if you are interested.

The kill and signal

We know that a kill can kill a process, but in principle, a process is not killed by a kill. A kill sends a signal to the process, and when the process receives the signal, it exits. You can run the kill -l command to view the signals sent by the kill command as follows:

$ kill-l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV  12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13 48) SIGRTMIN+14  49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAXCopy the code

Kill -9 is the signal that sends SIGKILL to the process. It can also be written as kill -sigkill. The following are the common signal scenarios:

The process caused by oom disappears

In addition to the above conditions, oom(memory overflow) is also a common cause of process disappearance. The application has requested a large amount of memory, and the process dies due to insufficient memory.

1. The process died because the JVM was out of memory

For Java such program, use the memory more than the -xmx threshold, the JVM will automatically exit, and leave the Java in the standard output stream. Lang. An OutOfMemoryError.

Therefore, when starting a Java process, it is best to use redirection to save the standard output and standard error to a log file. Then you can check whether oom is available:

Save standard output and standard error to a log file when starting the process
$ nohup java -jar app.jar >stdout.log 2>stderr.log &

# search for OutOfMemoryError
$ grep -A5 'OutOfMemoryError'stdout.log stderr.log stdout.log:java.lang.OutOfMemoryError: Java heap space stdout.log- at java.base/java.lang.StringCoding.decodeUTF8_0(StringCoding.java:753) ~[na:na] stdout.log-  at java.base/java.lang.StringCoding.decodeUTF8(StringCoding.java:712) ~[na:na] stdout.log- at java.base/java.lang.StringCoding.decode(StringCoding.java:318) ~[na:na] stdout.log- at java.base/java.lang.String.<init>(String.java:592) ~[na:na] stdout.log- at java.base/java.lang.String.<init>(String.java:614) ~[na:na]Copy the code

2. The oom-killer mechanism causes the process to die

If Linux is running out of memory, it will use oom-killer mechanism to find a process that occupies a large amount of memory and sacrifice it. However, Java processes usually consume a large amount of memory, so it is often sacrificed.

When oom-killer kills a process, some information is printed in the DMESg log. Therefore, you can check whether the process is killed by oom-killer in the following ways:

$ dmesg -T | grep -A10 -i "kill"
[Fri Dec 17 22:35:47 2021] Memory cgroup out of memory: Killed process 3102186 (java) total-vm:10487368kB, anon-rss:287124kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:3764kB oom_score_adj:0
[Fri Dec 17 22:35:47 2021] oom_reaper: reaped process 3102186 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Copy the code

Note: DMESg is like a kernel log, and the kernel records some key information in it. If you don’t have a clue about something, get into the habit of reading the dMESg.

Linux command gap-Getting Started Linux command gap-text Processing Linux command gap-Software Resource Observation Linux command gap-Hardware resource observation Linux command gap-Profiling tools Linux command gap-dynamic tracking tools Linux command gap-Understanding system load Linux command gap-network packet capture tool

Content of the past

Awk is really a magic tool for Linux text command tips (top) Linux text command tips (bottom) character encoding solution