Flutter terminal emulator to explore (2) of article | complete terminal emulator

Last time, I opened an open source for a simple Terminal simulator, which I knew was not standard, but I had been using it for a long time. Then I found some tricky problems, so I went to study the source code of some complete terminals, including Termux and Android Terminal, and finally succeeded in implementing their principles into Flutter

Dart :ffi: dart:ffi: dart:ffi: Dart :ffi: Dart: FFi: Dart: FFi: Dart: FFi: Dart: FFi: Dart: FFi: Dart: FFi I completely refactored the C part of Termux for Use with Flutter, since Flutter used by the UI framework has been tested to run on Macos!!

Where does the Process class stdout come from?

I met this thorny problem in use, or due to my lack of experience, I went to Zhihu to ask the problem I met. After discussing with my classmates about zhihu transmission, we can know that stdout in the Process comes from pipe, and that stdout also has the method of pipe. A pipe is buffered, for example 🌰

use

cp -rv sourceDir targetDir
Copy the code

Command, due to open the -v parameter, so in the standard terminal, will the cp command line print out is copying files, and when to perform this operation in the Process of the dart, you will be received in listening to stdout is not a line of the callback, but a pile of callback, it is because the pipeline is buffer mechanism, You can only get it once the buffer limit is reached, or you can get it when the buffer is full after the program ends. Let’s switch to the standard terminal emulator

cp -rv sourceDir targetDir | xargs echo
Copy the code

We also use pipes in the terminal, and print them through XARgs. When we print them, we see that the number of prints is the same as the number of stdout callbacks in DART. Not only dart but also Java Runtime input streams can’t get unbuffered output.

Buffer difference between terminal and pipe

Terminal also has a buffer, terminal line buffer, pipeline buffer for the whole, the line buffer, the output of the meeting to end with a newline \ n time, or active in C call fflush () method, which will have in the contents of the buffer output at a time, if there is no more than two conditions, it was only when the buffer full 1024 bytes, can output at a time

How does a standard terminal get the line buffered output?

The quickest way I can think of is to look at some open source libraries for standard terminals. There are some excellent ones now, including Termux and Android Terminal. Termux is the most powerful Terminal on Android, with a large number of extensible resources. After all, Termux is still a large repository and has annotations. However, I could not find the key place where Flutter could be implemented. Finally, I located the UI to get input, including synchronizing the output to the screen. This series all points to JNI, which is a Java to C/C ++ channel, and that’s where I started to know when THE C language in the project was used.

Standard terminal implementation principle

This terminal is called a pseudo terminal (PTY)

Must first read a wave of popular science from the Internet

Pseudo terminal (sometimes called pTY) refers to a pair of character devices called pseudo master and slave. The slave corresponds to a file in the /dev/pts/ directory, while the master is identified in memory as a file descriptor (FD). The pseudo-terminal is provided by the terminal emulator, which is an application running in user mode.

The Master terminal is closer to the user’s monitor and keyboard, and the slave terminal is the Command Line Interface (CLI) program running on the virtual terminal. The pseudo-terminal driver of Linux forwards the data written by the master (such as the keyboard) to the slave for input, and forwards the data written by the slave to the master for reading (such as the monitor driver). Please refer to the diagram below (from the Internet) :

The Terminal desktop programs we open, such as GNOME Terminal, are actually Terminal emulation software. When terminal emulation software runs, it creates a master and slave pair of pseudo terminals by opening the /dev/ptmx file and having the shell run on the slave side. When a user presses a keyboard key in terminal emulation software, it generates a byte stream that is written to the master, and the shell process reads input from the slave. The shell and its subroutines write output to the slave, and terminal emulation software prints characters into the window.

Text descriptors! ? From Baidu:

** everything in Linux is a file, such as C++ source files, video files, Shell scripts, executables, etc., even the hardware devices such as keyboards, monitors, and mice are files.

A Linux process can open hundreds or thousands of files, and to represent and distinguish opened files, Linux assigns each File a number (an ID), which is an integer, called a File Descriptor. 支那

The following operations are only available on Unix systems

So basically you know that the text descriptor is an int, and you can read and write from that, so write(fd, STR, Length) in C, you can write directly to the text descriptor, and Java also has a FileDescriptor class for reading and writing text descriptors, Dart doesn’t have that, But it can be fixed. To recap the terminal principle, calling open(“/dev/ PTMX “) in C will get a text descriptor, and at the same time a file under /dev/pts-/ will be generated with the names 0,1,2,3, which will be assigned to you in order. /dev/ptmx is a character device file. When a process opens the /dev/ptmx file, The process gets both a pseudoterminal Master (PTM) file descriptor and a pseudoterminal slave(PTS) device created in the /dev/pts directory. Each file descriptor obtained by opening the /dev/ptmx file is a separate PTM with its own associated PTS to look directly at my changed implementation

int get_ptm_int(
    int rows,
    int columns)
{
    // Calling open returns a random integer greater than 0
    int ptm = open("/dev/ptmx", O_RDWR | O_CLOEXEC);
    // The value is incremented from 0
    // if (ptm < 0) return throw_runtime_exception(env, "Cannot open /dev/ptmx");
#ifdef LACKS_PTSNAME_R
    char *devname;
#else
    char devname[64];
#endif
    if (grantpt(ptm) || unlockpt(ptm) ||
#ifdef LACKS_PTSNAME_R
        (devname = ptsname(ptm)) == NULL
#else
        ptsname_r(ptm, devname, sizeof(devname))
#endif
    )
    {
        // return throw_runtime_exception(env, "Cannot grantpt()/unlockpt()/ptsname_r() on /dev/ptmx");
    }

    // Enable UTF-8 mode and disable flow control to prevent Ctrl+S from locking up the display.
    struct termios tios;
    tcgetattr(ptm, &tios);
    tios.c_iflag |= IUTF8;
    tios.c_iflag &= ~(IXON | IXOFF);
    tcsetattr(ptm, TCSANOW, &tios);

    /** Set initial winsize. */
    struct winsize sz = {.ws_row = (unsigned short)rows, .ws_col = (unsigned short)columns};
    ioctl(ptm, TIOCSWINSZ, &sz);
    return ptm;
}
Copy the code

This function is mainly used to get the text descriptor for PTM, with some terminal in between. For the sake of time, I annotated the Java callback error and replaced it with the Dart callback. So once we get this PTM descriptor, we can read and write to the PTM descriptor, and then we can read everything that we write into it, feel like we’re doing something about it? It’s not that any binary program is writing into it, and your terminal UI is just reading all the time. Look at the implementation of Termux in Java

        new Thread("TermSessionInputReader[pid=" + mShellPid + "]") {
            @Override
            public void run(a) {
                try (InputStream termIn = new FileInputStream(terminalFileDescriptorWrapped)) {
                    final byte[] buffer = new byte[4096];
                    while (true) {
                        int read = termIn.read(buffer);
                        if (read == -1) return;
                        if(! mProcessToTerminalIOQueue.write(buffer,0, read)) return; mMainThreadHandler.sendEmptyMessage(MSG_NEW_INPUT); }}catch (Exception e) {
                    // Ignore, just shutting down.
                }
            }
        }.start();

        new Thread("TermSessionOutputWriter[pid=" + mShellPid + "]") {
            @Override
            public void run(a) {
                final byte[] buffer = new byte[4096];
                try (FileOutputStream termOut = new FileOutputStream(terminalFileDescriptorWrapped)) {
                    while (true) {
                        int bytesToWrite = mTerminalToProcessIOQueue.read(buffer, true);
                        if (bytesToWrite == -1) return;
                        termOut.write(buffer, 0, bytesToWrite); }}catch (IOException e) {
                    // Ignore.
                }
            }
        }.start();
Copy the code

Two infinite loops, one that reads the PTM and synchronizes what it reads to the UI and the other that writes the class capacity of the input queue into the PTM

Looking at one of the key functions in Termux (which I changed)

void create_subprocess(char *env,
                       char const *cmd,
                       char const *cwd,
                       char *const argv[],
                       char **envp,
                       int *pProcessId,
                       int ptmfd)
{
#ifdef LACKS_PTSNAME_R
    char *devname;
#else
    char devname[64];
#endif

#ifdef LACKS_PTSNAME_R
    devname = ptsname(ptmfd);
#else
    ptsname_r(ptmfd, devname, sizeof(devname));
#endif
    // Create a process and return its PID
    pid_t pid = fork();
    if (pid < 0)
    {
        // return throw_runtime_exception(env, "Fork failed");
    }
    else if (pid > 0)
    {
        *pProcessId = (int)pid;
    }
    else
    {
        // Clear signals which the Android java process may have blocked:
        sigset_t signals_to_unblock;
        sigfillset(&signals_to_unblock);
        sigprocmask(SIG_UNBLOCK, &signals_to_unblock, 0);

        close(ptmfd);
        setsid();
        //O_RDWR reads and writes,devname is /dev/pts/0,1,2,3...
        int pts = open(devname, O_RDWR);
        if (pts < 0)
            exit(- 1);
        // Copy stdin,stdout,stderr to PTS
        //ptmx,pts pseudo terminal master and slave
        dup2(pts, 0);
        dup2(pts, 1);
        dup2(pts, 2);
        //Linux API, open a folder
        DIR *self_dir = opendir("/proc/self/fd");
        if(self_dir ! =NULL)
        {
            // the file descriptor is changed to a file descriptor
            int self_dir_fd = dirfd(self_dir);
            struct dirent *entry;
            while((entry = readdir(self_dir)) ! =NULL)
            {
                int fd = atoi(entry->d_name);
                if (fd > 2&& fd ! = self_dir_fd) close(fd); } closedir(self_dir); }// Clear environment variables
        // clearenv();

        if (envp)
            for (; *envp; ++envp)
                putenv(*envp);

        if(chdir(cwd) ! =0)
        {
            char *error_message;
            // No need to free asprintf()-allocated memory since doing execvp() or exit() below.
            if (asprintf(&error_message, "chdir(\"%s\")", cwd) == - 1)
                error_message = "chdir()";
            perror(error_message);
            fflush(stderr);
        }
        // Execute the program
        execvp(cmd, argv);

        // Show terminal output about failing exec() call:
        char *error_message;
        if (asprintf(&error_message, "exec(\"%s\")", cmd) == - 1)
            error_message = "exec()";
        perror(error_message);
        _exit(1); }}Copy the code

In fact, termux’s create_subprocess is split into two parts to match the Dart part, and the logic is not changed. Note that the create_subprocess is called once. After this function is called, a fork() process will be executed twice. PProcessId (pProcessId, pProcessId); pProcessId (pProcessId, pProcessId); pProcessId (pProcessId, pProcessId, pProcessId, pProcessId); Ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r = ptsname_r Therefore, the output of binary called by exec is written into PTS, and written into PTS can be written out from PTM, thus realizing pseudo terminal

What if Dart can’t read or write text descriptors?

With DART: FF, C is readable and does not exist

void write_to_fd(int fd, char *str)
{
    write(fd, str, strlen(str));
}
char *get_output_from_fd(int fd)
{
    int flag = - 1;
    flag = fcntl(fd, F_GETFL); // Get the current flag
    flag |= O_NONBLOCK;        // Set the new falg
    fcntl(fd, F_SETFL, flag);  / / update the flag
    // Dynamically request space
    char *str = (char *)malloc((4097) * sizeof(char));
    The read function returns the length of the character read from the fd
    // The read content is stored in STR. 4096 means 4096 bytes are read at this time. If only 10 bytes are read at this time, length is 10
    int length = read(fd, str, 4096);
    if (length == - 1)
    {
        free(str);
        return NULL;
    }
    else
    {
        str[length] = '\ 0';
        returnstr; }}Copy the code

The implementation of Flutter is also complicated, because it is not easy to rewrite a complete set of terminal sequence. Termux, as an Android native project, has a large number of community resources and the support of third-party developers, so it has been relatively complete now. You can also refer to my previous posts about Dart calling FFI

Effect!!

Python use:Cursor movement:

Color output of ls and other commands:

Open source address