Our scheduled tasks and jar packages of asynchronous MQ will use blocking programs such as System.in.read() to prevent the program from exit. There is no problem in the local test until some students report that the code system.in.read () in the online Docker environment does not block. After executing the following program, the simplified code is shown below.
public static void main(String[] args) throws IOException, InterruptedException {
System.out.println("enter main....");
// Start a scheduled task
startJobSchedule();
System.out.println("before system in read....");
System.in.read();
System.out.println("after system in read....");
}
Copy the codeI glanced at it and thought no, the code must block at system.in.read (), and said if “after System in read….” was printed I live eat shoes. System.in.read(); Exit, execute the following statement, and immediately the shoe is served, MMM, delicious.
Here are some things you can learn from reading this article.
- The relationship between a process and the file descriptor FD
- /dev/null file context, read written kernel source code analysis
- Redirection nature
- On the concept of pipeline
Process and file descriptor fd
Let’s first look at the relationship between the process and the file descriptor FD. After a process is started, in addition to allocating heap and stack space, three file descriptor handles are allocated by default: stDIN 0, STdout 1, and stderr 2, as shown below.
System.in.read() is actually reading data from stdin with fd 0. We print out the return value of system.in.read () and the read. After testing, we return -1 and read EOF. This is strange, why does stdin return EOF?
Now let’s see what stdin with fd 0 actually points to. All open file handles of the process are stored in the /proc/pid/fd directory of the system. Use ls to view the list of open handles as shown below.
$ ls -l/proc/1/fd total 0 LRWX ------ 1 root root 64 4月 3 17:13 0 -> /dev/null l-wx------ 1 root root 64 4月 3 17:13 1 -> Pipe :[31508] l-wx------ 1 root root 64 4月 3 17:13 2 -> Pipe :[31509] L-wx ------ 1 root root 64 4月 3 17:13 3 -> Log lr-x------ 1 root root 64 4月 3 17:134 -> /jdk8/jre/lib/rt.jar lr-x------ 1 root root 64 4月 3 17:135 - > / app/system - in - read - 1.0 - the SNAPSHOT. The jarCopy the codeYou can see that fd 0 points to /dev/null. Let’s take a look at /dev/null.
/ dev/null file
What is the /dev/null file
/dev/null is a special device file and all received data is discarded. /dev/null is a black hole.
In addition to discarding all writes to this feature, reading data from /dev/null immediately returns EOF, which is why the previous system.in.read () call simply exits.
Using stat to view /dev/null, the output is as follows.
$ stat /dev/null
File: ‘/dev/null’
Size: 0 Blocks: 0 IO Block: 4096 character special file
Device: 5h/5d Inode: 6069 Links: 1 Device type: 1,3 Access: (0666/crw-rw-rw-) Uid: (0 / root) Gid: (0 / root) Context: system_u:object_r:null_device_t:s0 Access: (0666/crw-rw-rw-) Uid: (0 / root) Gid: (0 / root) Context: system_u:object_r:null_device_t:s0 Access: 2020-03-27 19:27:37.857000000 +0800 Change: 2020-03-27 19:27:37.857000000 +0800 Change: 2020-03-27 19:27:37.857000000 +0800 Change: $whO-B system Boot $whO-B system Boot $whO-B system Boot $whO-B system BootCopy the codeYou can see that the size of the /dev/null file is 0, and the creation and modification time are consistent with the kernel system startup time. It is not a disk file, but a file of type “Character Device File” that exists in memory.
All data written to this file will be discarded, write calls will always return success, and this particular file will not be filled up or its file size changed.
Another interesting observation is that tail -f /dev/null permanently blocks the strace command output as shown in the following summary.
$ strace tail -f /dev/null
open("/dev/null", O_RDONLY) = 3
read(3, "", 8192) = 0
inotify_init() = 4
inotify_add_watch(4, "/dev/null", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1
read(4,
Copy the codeYou can see that tail -f’s read call to /dev/null returns 0, indicating that it encountered EOF, and then tail creates an inotify instance using the inotify_init system call. This instance listens for IN_MODIFY, IN_ATTRIB, IN_DELETE_SELF, and IN_DELETE_SELF events on the /dev/null file. The implications of these four events are as follows.
- IN_MODIFY: The file is modified
- IN_ATTRIB: modifies file metadata
- IN_DELETE_SELF: listens for a directory or file to be deleted
- IN_MOVE_SELF: listens for a directory or file to be moved
It then blocks to wait for these events to occur, and since /dev/null does not, the tail command blocks after that.
/dev/null from the source point of view
The kernel handles /dev/null logic at github.com/torvalds/li… The code for writing data to /dev/null is in the write_null function, whose source code is shown below.
static ssize_t write_null(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
return count;
}
Copy the codeAs you can see, writing to /dev/null, the kernel does nothing but return the count value passed in.
The code read is in the read_NULL function, whose logic is shown below.
static ssize_t read_null(struct file *file, char __user *buf,
size_t count, loff_t *ppos)
{
return 0;
}
Copy the codeAs you can see, reading /dev/null immediately returns 0 for EOF.
/dev/null Why are there no problems with native tests? Because the native test uses terminal to start the JAR package, the process’s stdin is assigned to keyboard input and will always block if no characters are entered. Let’s look at how to reproduce the problem locally.
File descriptors and redirects
Standard input, standard output, and error output do not change their position in the descriptor, but their orientation can be changed. The redirection operators > and < are used to redirect the data stream. To change the standard input for the above process to /dev/null, use the < redirection character. Modify the previous code and add sleep to keep it from exiting.
public static void main(String[] args) throws IOException, InterruptedException {
System.out.println("enter main....");
byte[] buf = new byte[16];
System.out.println("before system in read....");
int length = System.in.read();
System.out.println("len: " + length + "\t" + new String(buf));
TimeUnit.DAYS.sleep(1);
}
Copy the codeRun packaged, and the output is as follows.
$ java -jar system-in-read-1.0-SNAPSHOT.jar < /dev/null
enter main....
before system in read....
len: -1
Copy the codeYou can see the same phenomenon as in the online Docker environment, system.in.read () does not block and returns -1.
View the fd list for the process as follows:
$ ls -l /proc/482/fd
lr-x------. 1 ya ya 64 4月 3 20:00 0 -> /dev/null
lrwx------. 1 ya ya 64 4月 3 20:00 1 -> /dev/pts/6
lrwx------. 1 ya ya 64 4月 3 20:00 2 -> /dev/pts/6
lr-x------. 1 ya ya 64 4月 3 20:00 3 -> /usr/local/ JDK /jre/lib/rt.jar lr-x------. 1 ya ya 64 4月 3 20:00 4 -> /home/y/system-in-read-1.0-snapshot.jarCopy the codeAs you can see, the standard input has been replaced with /dev/null. When the system.in.read () call reads the standard input, it first checks the list of file descriptors to see which stream the 0 descriptor points to, and then reads the data from the stream.
The above example redirects standard input, and standard output and standard error output can be redirected in a similar way.
1 >or>Redirects standard output2 >Redirects standard error output
Or you can use it in combination:
Java -jar system-in-read-1.0-snapshot. jar </dev/null > stdout.out 2> stderr.out $ls-l/proc/2629/fd lr-x------. 1 ya ya 64 4月 3 20:35 0 -> /dev/null l-wx------. 1 ya ya 64 4月 3 20:35 1 -> Out l-wx------. 1 ya ya 64 4月 3 20:352 -> /home/ya/stderr.outCopy the codeYou can see that this time the file descriptors with fd 0, 1, and 2 have been replaced.
What do I mean by 2>&1, which is often seen in shell scripts
2> redirects stderr, &1 stdout, and redirects stderr to stdout. For example, to redirect both standard output and standard error output to a file, you could write this.
cat foo.txt > output.txt 2>&1
Copy the codeNext, look at the concept of file descriptors as they relate to pipes.
The pipe
A pipe is a one-way flow of data, and we often use pipes on the command line to connect two commands, such as the following command.
nc -l 9090 | grep "hello" | wc -l
Copy the codeRun the above command, the actual execution process is as follows
- ZSH process created from the command line
- The ZSH process starts the NC-L 9090 process
- The ZSH process started the grep process and piped the standard output of the NC process to the standard input of the grep process
- The ZSH process starts the WC process and pipes the standard output of grep to the standard input of the WC process
Their process relationship is shown below.
PID TTY STAT TIME COMMAND
23714 ? Ss 0:00 \_ sshd: ya [priv]
23717 ? S 0:00 | \_ sshd: ya@pts/5
23718 pts/5 Ss 0:00 | \_ -zsh
4812 pts/5 S+ 0:00 | \_ nc -l 9090
4813 pts/5 S+ 0:00 | \_ grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exc
4814 pts/5 S+ 0:00 | \_ wc -l
Copy the codeThe following table lists the file descriptors for the NC and grep processes.
$ ls -l1 ya ya 64 4月 3 21:220 -> /dev/pts-/5 l-wx------. 1 ya ya 64 4月 3 21:221 -> Pipe :[3852257] LRWX ------. 1 ya ya 64 4月 3 21:172 -> /dev/pts/5 $ls-l1 ya ya 64 4月 3 21:220 -> pipe:[3852257] l-wx------. 1 ya ya 64 4月 3 21:221 -> Pipe :[3852259] LRWX ------. 1 ya ya 64 4月 3 21:172 -> /dev/pts/5 $ls-l1 ya ya 64 4月 3 21:220 -> pipe:[3852259] LRWX ------. 1 ya ya 64 4月 3 21:221 -> /dev/pts/5 LRWX ------. 1 ya ya 64 4月 3 21:172 -> /dev/pts/5Copy the codeThe relationship is shown below.
In Linux, the function to create a pipe is PIPE, and the common way to create a pipe is as follows.
int fd[2];
if (pipe(fd) < 0) {
printf("%s\n"."pipe error");
exit(1);
}
Copy the codeThe pipe function creates a pipe and returns two file descriptors, fd[0] to read data from the pipe and fd[1] to write data to the pipe. Next, let’s look at a piece of code to see how parent processes communicate through the pipe.
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#define BUF_SIZE 20
int main() {
int fd[2];
if (pipe(fd) < 0) {
printf("%s\n"."pipe error");
exit(1);
}
int pid;
if ((pid = fork()) < 0) {
printf("%s\n"."fork error");
exit(1);
}
// child process
if(pid == 0) { close(fd[0]); // Closes the child's readwhile (1) {
int n = write(fd[1], "hello from child\n", 18);
if (n < 0) {
printf("write eof\n");
exit(1);
}
sleep(1);
}
}
char buf[BUF_SIZE];
// parent process
if(pid > 0) { close(fd[1]); // Close the parent process's writewhile (1) {
int n = read(fd[0], buf, BUF_SIZE);
if (n <= 0) {
printf("read error\n");
exit(1);
}
printf("read from parent: %s", buf); sleep(1); }}return 0;
}
Copy the codeBy executing the code above, you can see that the string written from the child can be read by the parent and displayed on the terminal.
$ ./pipe_test
read from parent: hello from child
read from parent: hello from child
read from parent: hello from child
read from parent: hello from child
read from parent: hello from child
Copy the codeDocker with stdin
If you want the Docker process’s stdin to become a keyboard terminal, you can start Docker run with the it option. After running the image, reexamine the list of file descriptors opened by the process, and you can see that stdin, stdout, and stderr have all changed, as shown below.
$ docker exec -it 5fe22fbffe81 ls -l/proc/1/fd total 0 LRWX ------ 1 root root 64 4月 5 23:20 0 -> /dev/pts-0 LRWX ------ 1 root root 64 4月 5 23:20 1 -> /dev/pts-0 LRWX ------ 1 root root 64 4月 5 23:20 2 -> /dev/pts-0Copy the codeThe Java process also blocks on the System.in.read() call.
summary
From a small example, this article introduced the three basic file descriptors related to the process: stdin, STdout, and stderr, and how these file descriptors can be redirected. Along the way, I introduced the concept of plumbing. Okay, full shoes, sleep.
If you have any questions, you can scan the following TWO-DIMENSIONAL code to follow my official number to contact me.
