No Dishes Studio – Marklux Source: marklux.cn/blog/56 All rights reserved

Files and file systems

Files are the most important abstraction in Linux. In most cases, you can think of everything in Linux as a file. A lot of interaction is actually done by reading and writing files.

File descriptor

In the Linux kernel, a file is represented by an integer called a file descriptor. In layman’s terms, you can think of it as a file ID (unique identifier).

Common file

Ordinary files are byte stream organized data.
Files are not associated with file names, but with inodes, which have unique integer values (INO) assigned by the file system to normal files and hold some of the metadata associated with the file.

Contents and Links

Normally files are opened by filename.
A directory is a mapping between a readable name and an index number, and the pairing between a name and an index node is called a link.
A directory can be thought of as a normal file, except that it contains a mapping of file names to index nodes (links)

process

A process is an abstraction second only to a file. Simply understood, a process is executing object code, an active, running program. In complex cases, however, processes can also contain a wide variety of data, resources, states, and even virtual machines.

You can think of a process this way: it is the basic unit of competition for computer resources.

Processes, programs, and threads

The program

Programs, simply binary files on disk, are code that can be executed by the kernel
process

When a user starts a program, a space is opened in memory, which creates a process that contains a unique PID, the executor’s permission attribute parameters, and the code and associated data needed for the program.

Processes are the basic unit of resource allocation in the system.

A process can spawn other child processes, and the related permissions of the child process will be the same as the parent process.
thread

Each process contains one or more threads, which are units of activity within the process and abstractions responsible for executing code and managing the running state of the process.

The thread is the basic unit of independent operation and scheduling.

Hierarchy of processes (parent and child)

During the execution of a process, other processes may be spawned, called child processes, which have a PPID that identifies the parent process PID. The child process can inherit the environment variables and permission parameters of the parent process.

Thus, the Linux system is born in the process hierarchy – process tree.

The root of the process tree is the first process (the init process).

Procedure call flow: fork & exec

To create a child process, the system forks a parent process and generates a temporary process. The difference between the parent process and the temporary process is its PID and ppID. The system then exec the temporary process to load the actual program to run. The existence of a subprocess.

End of process

When a process terminates, rather than being immediately removed from the system, the kernel stores parts of the process in memory, allowing the parent process to query its status (this is called a pending process).

When the parent determines that the child has been terminated, the child is deleted completely.

However, if a child process has terminated and the parent does not know its status, the process becomes a zombie

Services and Processes

Daemons are simply in-memory processes that are usually started at startup with a script in init.d.

Process of communication

There are several basic ways for processes to communicate: pipes, semaphores, message queues, shared memory, and fast user controls mutually exclusive.

Programs, processes, and threads

Now let’s discuss these three concepts again in detail

Program (the program)

A program is a compiled, executable binary code that is stored on a storage medium and not run.

Process (process)

A process is a running program.

Processes include many resources and have their own separate memory space.

thread

A thread is a unit of activity within a process.

Includes its own virtual memory, such as stacks, process states such as registers, and instruction Pointers.

In a single-threaded process, a thread is a process. In multi-threaded processes, multiple threads will share the same memory address space
Refer to the reading

PID

Refer to the basic concepts section above.

In C, PID is represented by the data type pid_t.

Running a process

Creating a process is divided into two processes on Unix systems.

Load the program into memory and perform the operation of the program image :exec
Create a new process :fork

exec

The simplest exec system call function:`execl()`

Function prototype:

int execl(const char * path,const chr * arg,...)Copy the code

The execl() call will load an image of the path indicated by path into memory, replacing the image of the current process.

The arg argument takes the first argument, whose content is mutable, but must end with NULL.

For example:

int ret;

ret = execl("/bin/vi"."vi",NULL);

if (ret == -1) {
    perror("execl");
}Copy the code

The above code will replace the currently running program with /bin/vi

Note that the first argument vi is the default convention on Unix systems. When a process is created and executed, the shell puts the last part of the path into the first argument of the new process, which causes the process to resolve the name of the binary image file.

int ret;

ret = execl("/bin/vi"."vi"."/home/mark/a.txt",NULL);

if (ret == -1) {
    perror("execl");
}Copy the code

The above code is a very typical operation, which is equivalent to you executing the following command at the terminal:

vi /home/mark/a.txtCopy the code

The return value:

Execl () normally does not return and will jump to a new program entry point after a successful call.

A successful call to execl() changes the address space and the process image, as well as many other attributes of the process.

However, the PID,PPID, priority and other parameters of the process will be preserved, and even the open file descriptor will be preserved (which means that it can access all the files that the original process opened).

On failure -1 is returned and errno is updated.

Other exec function

Omitted, search when using

fork

With the fork() system call, you can create a child process that looks exactly like the current process image.

The function prototype

pid_t fork(void)Copy the code

After the call is successful, a new process (child process) is created, and both processes continue to run.

The return value

If the call succeeds, fork() returns the pid of the child, in which case it returns 0; If that fails, -1 is returned and errno is updated, and the child process is not created.

For example,

Let’s look at the code below

#include <unistd.h>
#include <stdio.h>
int main() { pid_t fpid; Int count=0;printf("this is a process\n");

    fpid=fork();

    if (fpid < 0)
        printf("error in fork!");
    else if (fpid == 0) {
        printf("i am the child process, my process id is %d\n",getpid());
        printf("I am the son of my father \ N");
        count++;
    }
    else {
        printf("i am the parent process, my process id is %d\n",getpid());
        printf("I am the father of the child \ N");
        count++;
    }
    printf("The statistical result is: %d\n",count);
    return 0;
}Copy the code

The code runs like this:

This is a process I am the parent process, my process ID is 21448 1 I am the child process, my process id is 21449Copy the code

After fork(), the program has two processes, with the parent and child continuing to execute the code into different if branches.

How to understand that Pids are different in parent-child processes?

The parent process’s PID points to the child process’s PID, because the child process has no children, so its PID is 0.

When writing copy

The traditional fork mechanism is that when fork is invoked, the kernel copies all the internal data structures, copies the process’s page table entries, and copies the parent’s address space to the child process on a page-by-page (very time-consuming) basis.

Modern fork mechanisms adopt an optimization strategy of an inert algorithm.

In order to avoid the overhead of copying, the “copy” operation is minimized. When multiple processes need to read copies of their own part of the resource, they do not copy multiple copies, but set a file pointer for each process and let them read the same actual file.

Obviously, this approach will create conflicts (similar to concurrency) at write time, so when a process wants to modify its own copy, it will copy the resource (only copy on write, so it is called copy-on-write), which reduces the frequency of replication.

Joint instance

Create a child process in an application to open another application.

pid_t pid;

pid = fork();

if (pid == -1)
    perror("fork"); / / the child processif(! pid) { const char * args[] = {"windlass",NULL}; int ret; // Parameters are passed in array ret = execv("/bin/windlass",args);

    if (ret == -1) {
        perror("execv");
        exit(EXIT_FAILURE); }}Copy the code

The above program creates a child process and makes it run the /bin/windlas program.

Terminate the process

exit()

The function prototype

void exit (int status)Copy the code

This function is used to terminate the current process. The status parameter is only used to identify the exit status of the process. This value will be passed to the parent of the current process for judgment.

There are a few other terminating functions that won’t be covered here.

Wait for the child process to terminate

How do I notify the parent process that the child process has terminated? This can be achieved through signalling mechanisms. But in many cases, the parent process needs to know more detailed information about the child process (such as the return value), and a simple signal notification is useless.

If the child process is completely destroyed by the time it terminates, the parent process cannot obtain any information about the child process.

So Unix originally made such design, if a child process finished before the parent process, the kernel is to the child process is set as a special kind of running state, this state of the process known as zombies, it retains only the smallest profile, waiting for the parent process after the access to the information, will be destroyed.

wait()

The function prototype

pid_t wait(int * status);Copy the code

This function can be used to get information about terminated child processes.

The PID of the terminated child process is returned on success, and -1 on error. If no child process terminates, the call will block until one child process terminates.

waitpid()

The function prototype

pid_t waitpid(pid_t pid,int * status,int options);Copy the code

Waitpid () is a more powerful system call that supports more fine-grained control.

Some other wait functions that you might encounter

wait3()
wait4()

Simply put, wait3 waits for the termination of any child process and wait4 waits for the termination of a specified child process.

Create and wait for a new process

Most of the time we will encounter the following situation:

You create a new process, and you want to wait for it to finish before continuing to run your own process, that is, create a new process and immediately start waiting for its termination.

An appropriate choice is system():

int system(const char * command);Copy the code

The system() function will call the commands provided by command, which is typically used to run simple tools and shell scripts.

On success, the return status is the result of executing command.

You can use fork(), exec(), waitpid() to implement a system().

Here is a simple implementation:

int my_system(const char * cmd)
{
    int status;
    pid_t pid;

    pid = fork();

    if (pid == -1) {
        return- 1; }else if (pid == 0) {
        const char * argv[4];

        argv[0] = "sh";
        argv[1] = "-c";
        argv[2] = cmd;
        argv[3] = NULL;

        execv("bin/sh",argv); // The call seems to have type conversion problemsexit(1); }// child process // parent processif (waitpid(pid,&status,0) == -1)
        return- 1;else if (WIFEXITED(status))
        return WEXITSTATUS(status);

    return- 1; }Copy the code

Ghost process

We talked about zombie processes above, but if the parent process does not wait for the child process to operate, then all its children will become ghost processes, and the ghost process will always exist (because it will not terminate until the parent process calls), causing the system to slow down.

Normally we should not allow this to happen, but if the parent terminates before the child terminates, or before the parent has a chance to wait for its zombie child, ghost processes inevitably result.

The Linux kernel has a mechanism to prevent this from happening.

Whenever a process terminates, the kernel iterates through all its children and resets their parent to init, which periodically waits for all children to make sure there are no long-lived ghost processes.

Processes and Permissions

Omitted, to be added

Sessions and process groups

Process group

Each process belongs to a process group, which consists of one or more processes related to each other for job control.

The ID of a process group is the PID of the first process in the process group.

The significance of a process group is that signals can be sent to all processes in the process group. This enables simultaneous operations on multiple processes.

The session

A session is a collection of one or more process groups.

Generally speaking, there is no fundamental difference between a session and a shell.

We usually describe a session using the example of a user logging into a terminal to perform a series of operations.

For example,

$cat ship-inventory.txt | grep booty|sortCopy the code

This is a shell command in a session that produces a group of three processes.

Daemon (Service)

Daemons run in the background and are not associated with any control terminal. It is usually started when the system is started by calling the init script.

In Linux, there is no difference between a daemon and a service.

There are two basic requirements for a daemon: it must run as a child of the init process, and it must not interact with any control terminal.

A process that generates a daemon

callfork()To create a child process (which will soon become a daemon)
Called in the parent of the processexit()This ensures that the parent exits when its child terminates, that the daemon’s parent is no longer running, and that the daemon is not the primary process. (It inherits the process group ID of the parent process and is definitely not the leader)
callsetsid()Create a new process group and a new session for the daemon as the first process of both. This ensures that there is no control terminal associated with the daemon.
callchdir()To change the current working directory to the root directory. This is to prevent the daemon from running in a random directory opened by the parent of the original fork for easy management.
Close all file descriptors.
Open file descriptors 0,1,2 (stdin,stdout,err) and redirect them to/dev/null.

daemon()

Use the above to generate a daemon

The function prototype

int daemon(int nochdir,int noclose);Copy the code

If the nochdir parameter is a non-zero value, the working directory is not directed to the root directory. If noclose is a non-zero value, all open file descriptors are not closed.

Returns 0 on success, -1 on failure.

Note that the resulting function is a copy of the parent (fork), so the resulting daemon looks exactly like the parent. In general, this means writing code in the parent to run functions in the background and then calling daemon() to wrap those functions into a daemon.

This appears to wrap the currently executing process as a daemon, but in fact wraps a copy of its derivation.

thread

Basic concept

A thread is a unit of execution (one level lower than a process) within a process, including virtual processors, stacks, program state, etc.

Threads can be thought of as the smallest execution unit of operating system scheduling.

Modern operating systems make two basic abstractions of user space: virtual memory and virtual processors. This makes the process internally “feel” like it owns machine resources.

Virtual memory

The system allocates a separate memory space for each process, which makes it think it has all the RAM to itself.

However, all threads within the same process share that process’s memory space.

Virtual processor

This is a thread-specific concept that makes each thread “feel” like it has the CPU to itself. In fact, the same is true for processes.

multithreading

Benefits of multithreading

Programming abstraction

Modular design patterns
concurrent

True concurrency can be achieved on multi-core processors, increasing system throughput
Improve responsiveness

Prevent serial arithmetic from dying
Prevent I/O congestion

Avoid I/O operations in a single thread that cause the entire process to block. It can also be solved by asynchronous I/O and non-blocking I/O.
Reduce context switching

Multithreaded switching costs far less performance than inter-process context switching
Memory sharing

Because threads within the same process can share memory, these features can be exploited in some scenarios to replace multiple processes with multiple threads.

The cost of multithreading

Debugging is extremely difficult.

Concurrent read and write operations in the same memory space can cause various problems (such as dirty data), it can be difficult to synchronize resources in multi-process scenarios, and the unpredictable timing and sequence of multiple independently running threads can cause all kinds of strange problems.

Consider the problem of concurrency.

Threading model

The concept of threads exists in both kernel and user space.

Kernel-level threading model

Each kernel thread is converted directly into a user-space thread. That is, kernel threads: user-space threads = 1:1

User-level threading model

Under this model, a user process that protects n threads will map to only one kernel process. N: 1.

You can reduce the cost of context switching, but it doesn’t make sense under Linux, where context switching between processes is inherently wasteful and therefore rarely used.

Hybrid threading model

A mixture of these two models, the N: M type.

Difficult to implement.

* Coroutine

Provides a lighter execution unit than threads.

The threaded

There is one thread for each connection

That is, blocking I/O, which is essentially single-threaded mode

Threads run in a serial fashion. When a thread encounters an I/O, the thread must be suspended until the operation is complete.

Event-driven threading pattern

The operation of the single thread model, most of the system load is waiting for I/O operations (especially), so in event-driven mode, the wait operation from the thread stripping away of the execution of the process, by sending asynchronous I/O request or I/O multiplexing, introducing the event loop and callback to handle the relationship between the thread and I/O.

For several modes of I/O, see here

Briefly, there are four kinds:

Blocking IO: serial processing, single thread, synchronous wait
Non-blocking I/O: After the thread initiates an I/O request, it will receive the result immediately instead of waiting. If the I/O is not finished, an ERROR will be returned. The thread needs to continuously request the Kernel to determine whether the I/O is complete
Asynchronous I/O: After the thread initiates an I/O request, the Kernel receives the result immediately. After the I/O is complete, the Kernel sends a SIGNAL to notify the thread
Event-driven IO: an upgrade to non-blocking I/O, which allows the Kernel to monitor multiple sockets (each socket is a non-blocking I/O) and continue to execute any socket that has a result.

Concurrency, parallelism, competition!

Concurrency and parallelism

Concurrency refers to the need to run (process) multiple threads at the same time.

Parallelism means that multiple threads are running at the same time.

Essentially, concurrency is a programming concept, while parallelism is a hardware property that can be implemented in parallel or without parallelism (single CPU).

competition

The biggest challenge of concurrent programming is competition, mainly because of the unpredictable order of execution results when multiple threads are executing simultaneously

For the simplest demonstration, refer to the basic example in Java concurrent programming.

Take a look at this line of code:
```
  x++;Copy the code
```
Assuming that the initial value of x is 5, and we execute this line of code with two threads at the same time, there are many different results, that is, x may be 6 or 7 at the end of the run. (This is a basic demonstration of concurrency, and it’s easy to understand if you understand it yourself.)

The reasons are briefly described as follows:

A thread executes x++ in 3 steps:
1. Load x into the register
2. Add the value of the register to 1
3. Write the register value back into x
  
  When two threads compete, it is the unpredictability of the three-step execution process in time. Suppose that x is 5 when thread 1 and thread 2 load x into the register, but when thread 1 writes back to X, x becomes 6, and when thread 2 writes back to X, x is still 6, which is contrary to the original intention.
  
  If there are more threads the results become more unpredictable.

The solution to competition: synchronization

To put it simply, it is to eliminate concurrency and use synchronous access and operation on competing resources.

The lock

The most common mechanism for dealing with concurrency is locking, although system-level locking is simpler than other complex systems such as DBMSS (there are no shared locks, exclusive locks, and other more complex concepts).

But locks cause two problems: deadlocks and starvation.

Solving these two problems requires some mechanics and design philosophy. Refer to the DBMS’s concurrency notes for details on locking.

One thing to remember about locks.

Resources are locked, not code

Keep this in mind when writing code.

System thread implementation: PThreads

In the original Linux system call, there was no complete thread library like C++11 or Java.

Overall, the PThread API is redundant and complex, but the basic operations are also creation, exit, and so on.

One thing to keep in mind is that threads in Linux have a state called Joinable. Here’s a brief overview:

The Join and Detach

The concept is very similar to that of the parent process, which waits for the child to exit (a series of wait functions).

In Linux, threads have two different states: Joinable and UnJoinable.

If a thread is marked as joinable, its stack resources and process descriptors are not released (similar to zombie processes), even if its thread function finishes executing or it is terminated with pthread_exit(). In this case, the thread creator should call pthread_join() to wait for the thread to terminate and reclaim its resources (similar to the wait function). By default, threads are created in this state.

If a thread is marked as unjoinable, it is said to have been detach. If the thread terminates, all its resources are automatically reclaimed. Save it the trouble of cleaning up its ass.

Since all created threads are joinable by default, either call pthread_detach(thread_id) on the parent thread to separate them, or call pthread_detach(pthread_self()) on the internal thread to mark itself as separate.

Introduction to Linux system programming

Files and file systems

File descriptor

Common file

Contents and Links

process

Processes, programs, and threads

Hierarchy of processes (parent and child)

Procedure call flow: fork & exec

End of process

Services and Processes

Process of communication

Programs, processes, and threads

Program (the program)

Process (process)

thread

PID

Running a process

exec

The simplest exec system call function:execl()

Other exec function

fork

When writing copy

Joint instance

Terminate the process

exit()

Wait for the child process to terminate

wait()

waitpid()

Some other wait functions that you might encounter

Create and wait for a new process

Ghost process

Processes and Permissions

Sessions and process groups

Process group

The session

Daemon (Service)

A process that generates a daemon

daemon()

thread

Basic concept

Virtual memory

Virtual processor

multithreading

Benefits of multithreading

The cost of multithreading

Threading model

Kernel-level threading model

User-level threading model

Hybrid threading model

* Coroutine

The threaded

There is one thread for each connection

Event-driven threading pattern

Concurrency, parallelism, competition!

Concurrency and parallelism

competition

The solution to competition: synchronization

The lock

System thread implementation: PThreads

The Join and Detach

Related Posts

Diagram + pseudocode, understand the super difficult to understand distributed Paxos algorithm

Programmer’s inner core method -AVL tree

105 Construct binary tree by traversing sequence of preorder and middle order

The simplest exec system call function:`execl()`