03 Go Combat: Hello World file system

Original is not easy, more dry goods, public concern: Qiya cloud storage

[toc]

Antecedents feed

Finally, the hands-on part, today we directly set up a file system called Hello World, attached with all the code implementation, and can test.

Environment to prepare

Environment Preparation:

Go programming environment (prepare an environment of go 1.13 or higher)
Get a Random Linux VIRTUAL machine (fuse support)

Verify that the Linux kernel supports FUSE

root@ubuntu:~# modprobe fuse
Copy the code

If the command does not return an error, then the kernel supports FUSE and is loaded.

The homemade file system starts

Step 1: Parse the FUSE protocol

In 02 FUSE Framework, we introduced the FUSE protocol and talked about the three components of the FUSE framework: the kernel FUSE file system, the user-mode Libfuse library, and the fusermount tool.

The kernel has only one FUSE file system, which accepts VFS requests, encapsulates the FUSE protocol package, and forwards user mode through the channel established by /dev/fuse. The task of the user mode is to parse and process the FUSE protocol package, then wrap the request response in the FUSE protocol, pass the /dev/fuse channel back to the kernel, and the VFS passes it back to the user.

So, we see that the user-mode libfuse library is really just for FUSE protocol parsing and encapsulation. Hey, kids, pay attention. Here’s the point. Highlight: As long as the data protocol, there is one characteristic: language – independent. The data protocol format is nothing more than a means of analyzing byte streams

FUSE is the FUSE protocol library implemented in c. The official Github address is github.com/libfuse/lib… . Go cannot use the Libfuse library directly, because the Libfuse library is all encapsulated as C constructs.

This is the first hurdle we need to cross, which is parsing the FUSE protocol in the Go language. All right, here we go. Take a look at FUSE’s packet format first:

Consider a question: What does Libfuse do?

Then, to build and/dev/fuseChannel;
Then, you implement a Server Server that listens on this channel, thus establishing contact with kernel FUSE to receive and send messages;
Then, different parsing is done for different requests;
- For example, read is Opcode, write is Opcode;
- Write requests carry user data. Other requests do not carry user data.
The parsed FUSE IO request is then forwarded to the user-mode file system;
Then, the IO response from the user-mode file system is received and encapsulated in FUSE response format.
And then, different requests have different responses, do different processing, forget it, it’s too much trouble,
- The response to read requests carries user data; other requests do not;

Go’s FUSE protocol library does the same. In fact, parsing the data format of any protocol is never interesting, because the logical functions implemented by the code are defined. A FUSE library for Go is recommended: Bazil/FUSE, a FUSE protocol parsing library written purely in Go that does exactly the same thing as libfuse, which is written purely in C.

bazil.org/fuse is a Go library for writing FUSE userspace filesystems.

With the Go FUSE protocol parsing library, you’re ready to start writing file system programs. It’s the parts we can participate in creating that are really interesting.

Step 2: Go homemade file system

Helloworld file system helloFS implements the following functions:

The mount point root directory has only one calledhelloFile (note: do not need to create oh, directly after mounting);
catthishelloWill returnhello, worldThe content of the;
Mount point directory properties: inode is 20210601, mode is 555.
helloThe inode value is 20210606 and the mode value is 444.

Create a file called Helloword. go with me and write the following code:

// Implement a filesystem called hellfs
package main

import (
    "context"
    "flag"
    "log"
    "os"
    "syscall"

    "bazil.org/fuse"
    "bazil.org/fuse/fs"
    _ "bazil.org/fuse/fs/fstestutil"
)

func main(a) {
    var mountpoint string
    flag.StringVar(&mountpoint, "mountpoint".""."mount point(dir)?")
    flag.Parse()

    if mountpoint == "" {
        log.Fatal("please input invalid mount point\n")}// Create a FUSE request listening channel object that parses and encapsulates it;
    c, err := fuse.Mount(mountpoint, fuse.FSName("helloworld"), fuse.Subtype("hellofs"))
    iferr ! =nil {
        log.Fatal(err)
    }
    defer c.Close()

    // Register the FS structure with the server so that it can be called back to process requests
    err = fs.Serve(c, FS{})
    iferr ! =nil {
        log.Fatal(err)
    }
}

// Hellofs file system body
type FS struct{}

func (FS) Root(a) (fs.Node, error) {
    return Dir{}, nil
}

// In the hellofs filesystem, Dir is the body of the directory operation
type Dir struct{}

func (Dir) Attr(ctx context.Context, a *fuse.Attr) error {
    a.Inode = 20210601
    a.Mode = os.ModeDir | 0555
    return nil
}

// When the ls directory is set, the ReadDirAll call is triggered, which returns the specified content, indicating that there is only one hello file;
func (Dir) Lookup(ctx context.Context, name string) (fs.Node, error) {
    // Only one entry file called Hello is processed, and everything else returns not exist
    if name == "hello" {
        return File{}, nil
    }
    return nil, syscall.ENOENT
}

// Define the behavior of Readdir, which returns a fixed inode:2 file called hello. The behavior of the corresponding user is generally ls directory.
func (Dir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
    var dirDirs = []fuse.Dirent{{Inode: 2, Name: "hello", Type: fuse.DT_File}}
    return dirDirs, nil
}

The helloFS File structure implements the File invocation in the File system
type File struct{}

const fileContent = "hello, world\n"

// When stat the file, inode is 2 and mode is 444
func (File) Attr(ctx context.Context, a *fuse.Attr) error {
    a.Inode = 20210606
    a.Mode = 0444
    a.Size = uint64(len(fileContent))
    return nil
}

// When you cat the file, the file contents return Hello, world
func (File) ReadAll(ctx context.Context) ([]byte, error) {
    return []byte(fileContent), nil
}
Copy the code

A brief description of what it does:

Defines the root directoryreaddir 和 getattrBehavior callback;
Defines the Hello filereadall 和 getattrBehavior callback;

Step 3: Let the file system Go

Now, the exciting part is, let’s compile this program and run it with a minimalist file system. Small as a sparrow is, it has all the organs.

Compile the helloworld. Go

root@ubuntu:~/gopher/src# go build -gcflags "-N -l" ./helloworld.go 
Copy the code

Compile successfully and get the binary helloWorld.

Create an empty directory

To create an empty directory as a mount point, I created a directory called myfs under/MNT.

root@ubuntu:~# mkdir /mnt/myfs/
Copy the code

Mount the run

Okay, so now we have our user file system program ready, our mount points ready, and we’re all set and ready to run. The command is as follows:

root@ubuntu:~/gopher/src# ./helloworld --mountpoint=/mnt/myfs --fuse.debug=true
Copy the code

Parameter Description:

mountpoint: Specifies the mount point directory, which is the empty directory created above/mnt/myfs/ ；
fuse.debugTo better understand the user file system, you can set this switch totrueSo that theWhat back-end logic does the request sent by the user correspond toIt’s pretty obvious;

Once the test runs, if there are no exceptions, HelloWorld is executed as a daemon, with no log. Until the request is received.

At this time, we do not move the terminal window (you can see the log later), and then open a new terminal for testing.

Step 4: Minimalist file system`hellofs`The test of

System Angle detection

Now let’s test helloFS from multiple angles and get a feel for what our first user file system looks like.

First, the file system must be mounted to use, so the df command can see how it is mounted:

root@ubuntu:~# df -aTh|grep helloHelloworld fuse. Hellofs 0.0K 0.0K 0.0K - / MNT /myfsCopy the code

See? There is a file system called HelloWorld of type fuse.hellofs. Both of these names are specified in the code.

Then, if the FusectL file system (kernel fuse file system) is mounted, you can also see one more numerically named directory at /sys/fs/ FUSE /connections than before.

File operation probe

Let’s probe the HelloFS file system with ls, stat, cat, etc.

First question:statTake a look at the mount pointsstat /mnt/myfs? What data do you get?

root@ubuntu:~# stat /mnt/myfs/
  File: '/mnt/myfs/'Size: 0 Blocks: 0 IO Block: 4096 directory Device: 29h/41d Inode: 20210601 Links: 1 Access: (0555/dr-xr-xr-x) Uid: (0/ root) Gid: (0/ root) Access: 2021-06-06 13:49:02.463926775 +0800 Modify: 2021-06-06 13:49:02.463926775 +0800 Change: 2021-06-06 13:49:02.463926775 +0800 Birth: -Copy the code

We see a special inode: 20210601, permission: 555 (recall the code implementation above, directory inode 20210601 is what we specified). As follows:

Note that the user file system prints a log along with stat/MNT /myfs:

root@ubuntu:~/code/gopher/src/myfs# ./helloworld --mountpoint=/mnt/myfs --fuse.debug=true
2021/06/06 13:49:04 FUSE: <- Getattr [ID=0x2 Node=0x1 Uid=0 Gid=0 Pid=891] 0x0 fl=0
2021/06/06 13:49:04 FUSE: -> [ID=0x2] Getattr valid=1m0s ino=20210601 size=0 mode=dr-xr-xr-x
Copy the code

This log explicitly tells us that a Getattr request was received, the parameters of the request were specified, and then the response was returned after the helloFS processing was completed.

Second question:ls /mnt/myfsThe reaction?

root@ubuntu:~# ls -l /mnt/myfs/
total 0
-r--r--r-- 1 root root 13 Jun  6 13:49 hello
Copy the code

We see a Hello file (remember, we didn’t create this file).

So, what does stat/MNT /myfs/hello get?

root@ubuntu:~# stat /mnt/myfs/hello 
  File: '/mnt/myfs/hello'Size: 13 Blocks: 0 IO Block: 4096 regular file Device: 29h/41d Inode: 20210606 Links: 1 Access: (0444/-r--r--r--) Uid: (0/ root) Gid: (0/ root) Access: 2021-06-06 13:49:02.463926775 +0800 Modify: 2021-06-06 13:49:02.463926775 +0800 Change: 2021-06-06 13:49:02.463926775 +0800 Birth: -Copy the code

We see the special inode 20210606.

Third question:cat /mnt/myfs/helloThis file?

root@ubuntu:~# cat /mnt/myfs/hello 
hello, world
Copy the code

We see hello World return, even though you never wrote this file.

The fourth question: please feel hello file, this file and you usually see the file what is the difference?

This is an additional question and one that readers should focus on. Consider the uniqueness of the hello file:

This file shows up when you didn’t create it?
Does this file exist on disk?
Is this document writable?

Have you figured out the above questions? You can think about it, or talk to me.

Remember that the concept of a file has always been a logical object. Is an abstract object that the file system gives you. In other words, whatever the file represents is just what the file system wants to show you.

What you see is what FS wants you to see!!

Original is not easy, more dry goods, public concern: Qiya cloud storage

conclusion

FUSE framework has three components: kernel FUSE module, user-mode FUSE protocol parsing library, fusermount tool;
FUSE protocol parsing itself is language-independent and can be implemented in C, Go, or even Python;
libfuseFUSE is the FUSE protocol parsing library for a pure C implementation if you want to implement a user file system in C.
bazil.org/fuseFUSE is the FUSE protocol library for the pure Go implementation. We implement the user file system in the Go language, so this is the right choice;
How easy is it to implement a user file system? We only need to define the processing logic of FS, Dir and FileHello, worldFile system;

Afterword.

The helloWorld user mode file system has been implemented. After implementing HelloFS, do you understand what a ** “file” ** is?

With this in mind, the next time we implement a more complex file system: an encrypted distributed file system, stay tuned.

Original is not easy, more dry goods, public concern: Qiya cloud storage

03 Go Combat: Hello World file system

Antecedents feed

Environment to prepare

The homemade file system starts

Step 1: Parse the FUSE protocol

Step 2: Go homemade file system

Step 3: Let the file system Go

Compile the helloworld. Go

Create an empty directory

Mount the run

Step 4: Minimalist file systemhellofsThe test of

System Angle detection

File operation probe

conclusion

Afterword.

Related Posts

Java multithreading — Explicit vs. implicit locking

Bye, Spring Security OAuth!!

Easy to understand data structure and algorithm series (bidirectional linked list)

Step 4: Minimalist file system`hellofs`The test of