HongMeng kernel source code analysis (pipe file) | | how to reduce the data flow cost hundreds of blog analysis OpenHarmony source | v70.02

The Master said, “Being upright, he did not obey orders; Its body is not straight, although the order does not follow.” The Analects of Confucius: Sub-Chapter

A hundred blog series. This is:

V70. Xx HongMeng kernel source code analysis (pipe file) | how to reduce the data flow cost

File system related sections are as follows:

• V62. Xx HongMeng kernel source code analysis concept (file) | why everything is the file
• V63. Xx HongMeng kernel source code analysis (file system) | said in books management file system
• V64. Xx HongMeng kernel source code analysis (inode) | who is the most important concept of file system
• V65. Xx HongMeng kernel source code analysis (mount directory) | why need to mount the file system
• V66. Xx HongMeng kernel source code analysis (the root file system) | on first/File system on
• V67. Xx HongMeng kernel source code analysis (character device) | bytes read/write device for the unit
• V68. Xx HongMeng kernel source code analysis file system (VFS) | the foundation of the harmonious coexistence
• V69. Xx HongMeng kernel source code analysis (file handle) | why do you call a handle?
• V70. Xx HongMeng kernel source code analysis (pipe file) | how to reduce the data flow cost

What is a pipe

• Pipe | pipesThe earliest and clearest statement comes fromMcIlroyby1964An internal document written in 1998. The document proposes to link programs together like garden hoses. The document is as follows:

  Summary--what's most important.
      To put my strongest concerns into a nutshell:
  1. We should have some ways of coupling programs like
  garden hose--screw in another segment when it becomes when
  it becomes necessary to massage data in another way.
  This is the way of IO also.
  2. Our loader should be able to do link-loading and
  controlled establishment.
  3. Our library filing scheme should allow for rather
  general indexing, responsibility, generations, data path
  switching.
  4. It should be possible to get private system components
  (all routines are system components) for buggering around with.
  
                                                  M. D. McIlroy
                                                  October 11, 1964 
  Copy the code

• UnixThe architect of theKen ThompsonIt took only an hour to implement the system call to the pipe in the operating system. He said it was a piece of cake because the I/O redirection mechanism was the foundation of the pipeline, but the effect was impressive. The essence of a pipeline isI/ORedirection is the constant editing, constant flow of data, only such data is valuable.
• As mentioned in the Document Concept section,UnixThe “everything is a file” argument stems from the commonality of inputs and outputs, which can and should be abstracted into unified file management and flow. Take the development of cities for example, the more frequent the population flow and capital flow, the more developed the city must be. Please carefully appreciate this truth. The planning of the city should make the cost of flow lower and the time shorter, rather than checking the ID card and household registration book everywhere. For kernel designers, too, a system that makes the cost of data flow extremely simple and convenient must be a good architecture,UnixOne of the big reasons it’s been so successful for so many years is that it only uses one|Symbols implement the problem of fluidity between files. It is a great innovation, and deserves a special chapter on it.

Pipe symbol|

Pipe symbol is a vertical bar | between two commands, simple and elegant, for example, the ls is used to display a directory file, wc for statistical rows. Ls | wc represents a certain number of files in the directory Take a look at a complex:

$ < colors.txt sort | uniq -c | sort -r | head -3 > favcolors.txt
Copy the code

• colors.txtFor the original file content, output tosortTo deal with
• sort 对 colors.txtThe contents are sequentially edited and output touniqTo deal with
• uniqTo de-edit the content and output tosort -rTo deal with
• sort -rEdit content in reverse order and output tohead -3To deal with
• head -3Take the first three edits of the content and output tofavcolors.txtSave the file.
• The lastcat favcolors.txtView the results

  $ cat favcolors.txt
      4 red
      3 blue
      2 green
  Copy the code

Classic Pipe Case

The following is a classic example of a Linux official pipeline. Check the pipe

#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
int main(int argc, char *argv[])
{
    int pipefd[2];
    pid_t cpid;
    char buf;

    if (argc != 2) {
        fprintf(stderr, "Usage: %s <string>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    if (pipe(pipefd) == -1) {
        perror("pipe");
        exit(EXIT_FAILURE);
    }

    cpid = fork();
    if (cpid == -1) {
        perror("fork");
        exit(EXIT_FAILURE);
    }

    if (cpid == 0) {    /* Child reads from pipe */
        close(pipefd[1]);          /* Close unused write end */

        while (read(pipefd[0], &buf, 1) > 0)
            write(STDOUT_FILENO, &buf, 1);

        write(STDOUT_FILENO, "\n", 1);
        close(pipefd[0]);
        _exit(EXIT_SUCCESS);

    } else {            /* Parent writes argv[1] to pipe */
        close(pipefd[0]);          /* Close unused read end */
        write(pipefd[1], argv[1], strlen(argv[1]));
        close(pipefd[1]);          /* Reader will see EOF */
        wait(NULL);                /* Wait for child */
        exit(EXIT_SUCCESS);
    }
}
Copy the code

Interpretation of the

• pipe(pipefd)For the system call, two file handles are applied and piped to the two files. The system call in hongmeng pipeline isSysPipeTo implement it, look down.
• mainprocessfork()A subprocess, specificforkProcess, please go to [v45. Xx (Fork) | one call, two return] turn. The child will copy the parent’s file resources. So the child processcpidAlso has thepipefdTwo handles, meaning you can manipulate thempipefdCorresponding file
• if (cpid == 0)Represents the return of the child process,
  • close(pipefd[1])Shut down thepipefd[1]File handle. Because the program is designed for child processes to read files, it does not require operationspipefd[1]
  • while (read(pipefd[0], &buf, 1)The child process continuously reads the filepipefd[0]The content of the.
  • It’s supposed to keep readingpipefd[0]The data shows that there is a process going onpipefd[0]But the idea of the pipe is topipefd[1]To write data, data can run topipefd[0]In the.
• (cpid > 0)That’s in the code} else { Represents the parent processmainThe return.
  • close(pipefd[0])Shut down thepipefd[0]File handle. Because the program is designed to write files to the parent process, it does not need to operatepipefd[0]
  • write(pipefd[1], argv[1], strlen(argv[1]))The parent process topipefd[1]Data will appear in thepipefd[0]For the child process to read.

HongMeng implementation

The implementation of the pipe’s function-level call relationship is as follows:

SysPipe // system call AllocProcessFd // allocate two process descriptors pipe // true implementation of the underlying pipe pipe_ALLOCATE pipe "/dev/pipe%d" // generate create pipe file path, used to create two system file handles Pipecommon_allocdev // Allocate space shared by the piperegister_driver // Register the pipedevice driver open // Open two system file handles fs_getFilep // Obtain the entity object 'file' for two system file handles AssociateSystemFd // Binding of process and system file handlesCopy the code

Pipe is the most important one, it is the implementation of the pipe idea of the implementation of the implementation of the code, a little bit of code, but understand this function to understand the pipe completely, before reading the file system is recommended to read the length of the code and interpretation of the foundation is easy to understand.

int pipe(int fd[2]) { struct pipe_dev_s *dev = NULL; char devname[16]; int pipeno; int errcode; int ret; struct file *filep = NULL; size_t bufsize = 1024; /* Get exclusive access to the pipe allocation data */ ret = sem_wait(&g_pipesem); if (ret < 0) { errcode = -ret; goto errout; } /* Allocate a minor number for the pipe device */ pipeno = pipe_allocate(); if (pipeno < 0) { (void)sem_post(&g_pipesem); errcode = -pipeno; goto errout; } /* Create a pathname to the pipe device */ snprintf_s(devname, sizeof(devname), sizeof(devname) - 1, "/dev/pipe%d", pipeno); /* No.. Allocate and initialize a new device structure instance */ dev = pipecommon_allocdev(bufsize, devname); if (! dev) { (void)sem_post(&g_pipesem); errcode = ENOMEM; goto errout_with_pipe; } dev->d_pipeno = pipeno; /* Check if the pipe device has already been created */ if ((g_pipecreated & (1 << pipeno)) == 0) { /* Register the pipe  device */ ret = register_driver(devname, &pipe_fops, 0660, (void *)dev); if (ret ! = 0) { (void)sem_post(&g_pipesem); errcode = -ret; goto errout_with_dev; } /* Remember that we created this device */ g_pipecreated |= (1 << pipeno); } else { UpdateDev(dev); } (void)sem_post(&g_pipesem); /* Get a write file descriptor */ fd[1] = open(devname, O_WRONLY); if (fd[1] < 0) { errcode = -fd[1]; goto errout_with_driver; } /* Get a read file descriptor */ fd[0] = open(devname, O_RDONLY); if (fd[0] < 0) { errcode = -fd[0]; goto errout_with_wrfd; } ret = fs_getfilep(fd[0], &filep); filep->ops = &pipe_fops; ret = fs_getfilep(fd[1], &filep); filep->ops = &pipe_fops; return OK; errout_with_wrfd: close(fd[1]); errout_with_driver: unregister_driver(devname); errout_with_dev: if (dev) { pipecommon_freedev(dev); } errout_with_pipe: pipe_free(pipeno); errout: set_errno(errcode); return VFS_ERROR; }Copy the code

Interpretation of the

• The amount of pipe in Hongmeng is also limited and managed by bitmaps, with a maximum of 32 supported, using a 32-bit variableg_pipesetEnough, bitmap how to manage please see bitmap management. To use it, you have to applypipe_allocateBe responsible for.

  #define MAX_PIPES 32 static sem_t g_piPESem = {NULL}; static uint32_t g_pipeset = 0; // Static uint32_t g_pipecreated = 0; static inline int pipe_allocate(void) { int pipeno; int ret = -ENFILE; for (pipeno = 0; pipeno < MAX_PIPES; pipeno++) { if ((g_pipeset & (1 << pipeno)) == 0) { g_pipeset |= (1 << pipeno); ret = pipeno; break; } } return ret; }Copy the code

• What looks like two files on the outside of the pipe is actually reading and writing to a block of memory. Operation memory will need to apply for memory blocks, hongmeng default use1024 | 1KMemory, you need file paths to manipulate files/dev/pipe%d.

  size_t bufsize = 1024;  
  snprintf_s(devname, sizeof(devname), sizeof(devname) - 1, "/dev/pipe%d", pipeno);
  dev = pipecommon_allocdev(bufsize, devname);
  Copy the code

• This is followed by providing action files/dev/pipe%dtheVFSThat is, register file system driver, upper read and write operations, to the bottom of the real read and write is bypipecommon_readandpipecommon_writeFall to the ground.

  ret = register_driver(devname, &pipe_fops, 0660, (void *)dev);
  static const struct file_operations_vfs pipe_fops =
  {
  .open = pipecommon_open,      /* open */
  .close = pipe_close,          /* close */
  .read = pipecommon_read,      /* read */
  .write = pipecommon_write,    /* write */
  .seek = NULL,                 /* seek */
  .ioctl = NULL,                /* ioctl */
  .mmap = pipe_map,             /* mmap */
  .poll = pipecommon_poll,      /* poll */
  #ifndef CONFIG_DISABLE_PSEUDOFS_OPERATIONS
  .unlink = pipe_unlink,        /* unlink */
  #endif
  };
  Copy the code

  pipecommon_readThere’s a lot of code, I won’t show it here, but there’s a lot of semaphore in the code, just to make sure that this shared memory works.

• To operate on both file handles, you have to open both files, but one to read and one to write,pipeThe default is tofd[1]As written,fd[0]In order to read, we can scroll back here to look at the reading process of the classic pipe case.

  fd[1] = open(devname, O_WRONLY);
  fd[0] = open(devname, O_RDONLY);
  Copy the code

• The final bindingfileThe file interface operation, as detailed in the file handle section, that the application operates onFd | file handle, to the kernel is required to passfdfindfile, and then findfile->opsTo actually manipulate the file.

  ret = fs_getfilep(fd[0], &filep);
  filep->ops = &pipe_fops;
  
  ret = fs_getfilep(fd[1], &filep);
  filep->ops = &pipe_fops;
  Copy the code

Intensive reading of the kernel source code

Four code stores synchronous annotation kernel source code, >> view the Gitee repository

Analysis of 100 blogs. Dig deep into the core

Add comments to hongmeng kernel source code process, sort out the following article. Content based on the source code, often in life scene analogy as much as possible into the kernel knowledge of a scene, with a pictorial sense, easy to understand memory. It’s important to speak in a way that others can understand! The 100 blogs are by no means a bunch of ridiculously difficult concepts being put forward by Baidu. That’s not interesting. More hope to make the kernel become lifelike, feel more intimate. It’s hard, it’s hard, but there’s no turning back. 😛 and code bugs need to be constantly debug, there will be many mistakes and omissions in the article and annotation content, please forgive, but will be repeatedly amended, continuous update. Xx represents the number of modifications, refined, concise and comprehensive, and strive to create high-quality content.

HongMeng station | into a little bit every day, the original is not easy, welcome to reprint, please indicate the source.