Golang File system event listener – FSnotify

The paper

If the Golang program wants to listen for changes to certain files on the file system, the most common way to do this is to use the FSnotify library. At first, it was developed by Chris Howey(Github account: Howeyc). Later, it was loved by the majority of developers, so it was established separately. To date, its warehouses have received 5.9K Star, which is testament to its popularity. To learn more about the history of FSnotify, check out the fsNOTIFY website.

Source code analysis

The following source code analysis based on the git commit version is: 7 f4cf4dd2b522a984eaca51d1ccee54101d3414a

1. Code statistics

Cloc –by-file-by-lang — exclud-dir =. Github — exclud-lang =YAML,Markdown [project-dir], The results are as follows (omit the statistics for markup languages such as YAML):

The total number of GO lines of FSnotify code is 2922 lines, of which 1627(=923+358+190+105+51) lines of test class code, the actual valid code is only 1295 lines. With so little code supporting Windows/Linux/MAC, it’s a fairly lean library.

2. Example

To get a sense of the code, let’s start with the official example, which looks like this:

package main

import (
    "log"

    "github.com/fsnotify/fsnotify"
)

func main(a) {
    watcher, err := fsnotify.NewWatcher() // Initialize an empty watcher
    iferr ! =nil {
        log.Fatal(err)
    }
    defer watcher.Close() // Close watcher at the end of the program

    done := make(chan bool)
    go func(a) { // Start a coroutine to handle events sent by watcher separately
        for {
            select {
            case event, ok := <-watcher.Events: // Normal event processing logic
                if! ok {return
                }
                log.Println("event:", event)
                if event.Op&fsnotify.Write == fsnotify.Write {
                    log.Println("modified file:", event.Name)
                }
            case err, ok := <-watcher.Errors: // Processing logic when an error occurs
                if! ok {return
                }
                log.Println("error:", err)
            }
        }
    }()

    err = watcher.Add("/tmp/foo") // Enable watcher to monitor/TMP /foo
    iferr ! =nil {
        log.Fatal(err)
    }
    <-done // Make the main coroutine not exit
}
Copy the code

The usage is very simple:

1. Example Initialize an empty FSnotify watcher
2. Write a coroutine to handle events pushed by Watcher
3. Tell the watcher which files or directories to listen to

3. Build constraints

Fsnotify is a cross-platform library. The source code contains both Linux, MAC and Windows implementation logic. This is where the problem comes in:

After a developer references this library, how can he or she ensure that the compiled executable contains only implementations for the corresponding target platform and not implementations for unrelated platforms? For example, if the developer is compiling for Linux, how do you remove the implementation code for unrelated platforms such as MAC and Windows?

The good news is that Golang provides Build Constraints, which can be used as follows:

// +build linux,386 darwin,! cgo
Copy the code

The above comment is not an ordinary comment, but a build constraint. If written at the top of the code file (above the package declaration), it will be compiled into the executable by the compiler at compile time according to the target platform. The above line of build constraints means (Linux AND 386) OR (Darwin AND (NOT CGO)).

Now that we know how to use build constraints, we can look at the fsnotify source code and read the implementation in detail depending on the platform we care about.

4. Read it in detail — the Linux part

The most commonly used part is the Linux implementation. The underlying part is the Linux-based inotify mechanism, and the associated logic is in the inotify.go file in the library.

A. General idea

Watcher, err := fsnotify.newwatcher (), so let’s see what new watcher contains.

// NewWatcher establishes a new watcher with the underlying OS and begins waiting for events.
func NewWatcher(a) (*Watcher, error) {
    // Create inotify fd
    fd, errno := unix.InotifyInit1(unix.IN_CLOEXEC)
    if fd == - 1 {
        return nil, errno
    }
    // Create epoll
    poller, err := newFdPoller(fd)
    iferr ! =nil {
        unix.Close(fd)
        return nil, err
    }
    w := &Watcher{
        fd:       fd,
        poller:   poller,
        watches:  make(map[string]*watch),
        paths:    make(map[int]string),
        Events:   make(chan Event),
        Errors:   make(chan error),
        done:     make(chan struct{}),
        doneResp: make(chan struct{})},go w.readEvents()
    return w, nil
}
Copy the code

The general idea of the above code:

1. Create an inotify instance

   The inotify instance is returned to the caller in the form of a file descriptor (FD), from which events can be read if the file we watch changes. The problem is that we need to read the file descriptor ourselves, so we need to have some sort of rotation mechanism, which brings us to the epoll register below.

2. Listen for events on the instance using epoll

   Register the fd with ePoll, and ePoll will receive and return the data to us as soon as the data from the FD arrives.

3. {watches} is used to store the watch object and event is used to push events

4. Start the listening coroutine

B. Event listening coroutine

The above code finally starts a listening coroutine go w.readevents (). Let’s see what happens here. The code looks like this:

Redundant code is omitted for brevity

func (w *Watcher) readEvents(a) {
    var(...).// Variables
    defer close(...).// Close the context's resources

    for {
        if w.isClosed() { return }

        ok, errno = w.poller.wait() // The program blocks at this line until ePoll listens for the relevant event
        if. {continue } // Error handling logic
    
        n, errno = unix.Read(w.fd, buf[:]) // This is where an event occurs, so this is where the event is read into the buffer and handled below
        if. {continue } // Error handling logic
        if n < unix.SizeofInotifyEvent { // When the read event is less than 16 bytes (the unit size of an event structure), exception handling logic.continue
        }

        var offset uint32
        // We don't know how many events were read into the buffer
        // So we use offset to record the current position offset until we finish reading
        // The for loop ends when offset has been added to such a value that there are not enough bytes left to read an entire inotify event structure
        for offset <= uint32(n-unix.SizeofInotifyEvent) {
            // Force the address value to be converted into an inotify structure
            raw := (*unix.InotifyEvent)(unsafe.Pointer(&buf[offset]))

            mask := uint32(raw.Mask) // The event that occurred is represented as a mask
            nameLen := uint32(raw.Len) // When listening to a directory, the name of the event in the directory is included in the structure, where len is the length of the filename

            ifmask&unix.IN_Q_OVERFLOW ! =0{... }// If the mask format is incorrect, send the event to Errors chan

            w.mu.Lock() // Locked because the context may be deleted
            // Wd is the file descriptor for what we watch and what happened this time
            // We can fetch the file name corresponding to the file descriptor from the built context
            name, ok := w.paths[int(raw.Wd)] 
            // If a delete event occurs, the filename is also deleted from the context
            if ok && mask&unix.IN_DELETE_SELF == unix.IN_DELETE_SELF {
                delete(w.paths, int(raw.Wd))
                delete(w.watches, name)
            }
            w.mu.Unlock() / / unlock

            if nameLen > 0 { // When watch is a directory, an event occurs in the file below it, causing the nameLen to be greater than 0
                // At this point, read the name of the file (right after the inotify event structure) and force the address value to be converted into an array of 4096 bytes
                bytes := (*[unix.PathMax]byte)(unsafe.Pointer(&buf[offset+unix.SizeofInotifyEvent]))[:nameLen:nameLen]
                // Concatenation path (filename will end with \000, so leave it out)
                name += "/" + strings.TrimRight(string(bytes[0:nameLen]), "\ 000")
            }

            event := newEvent(name, mask) // Generate an event
            if! event.ignoreLinux(mask) {// If the event is not ignored, send it to Events chan
                select {
                case w.Events <- event:
                case <-w.done:
                    return}}// Move the offset to the next inotify event structure
            offset += unix.SizeofInotifyEvent + nameLen
        }
    }
}
Copy the code

C. Add the watch path

Err = watcher.add (“/ TMP /foo”) to make watcher go to the watch path/TMP /foo.Add is to register the path in inotify.

// Add starts watching the named file or directory (non-recursively).
func (w *Watcher) Add(name string) error {
    name = filepath.Clean(name) // Get the standard path. For example, / TMP //////too becomes/TMP /too after Clean
    if w.isClosed() {
        return errors.New("inotify instance already closed")}const agnosticEvents = unix.IN_MOVED_TO | unix.IN_MOVED_FROM |
        unix.IN_CREATE | unix.IN_ATTRIB | unix.IN_MODIFY |
        unix.IN_MOVE_SELF | unix.IN_DELETE | unix.IN_DELETE_SELF

    var flags uint32 = agnosticEvents

    w.mu.Lock()
    defer w.mu.Unlock()
  watchEntry := w.watches[name] // Retrieve the watch path from the context (if it exists)
    ifwatchEntry ! =nil {
        flags |= watchEntry.flags | unix.IN_MASK_ADD
    }
    wd, errno := unix.InotifyAddWatch(w.fd, name, flags) // Add the watch path
    if wd == - 1 {
        return errno
    }

    if watchEntry == nil { // If this path does not exist in the context, it indicates that this is a new watch, add it to the context
        w.watches[name] = &watch{wd: uint32(wd), flags: flags}
        w.paths[wd] = name
    } else { // Update the context if it exists
        watchEntry.wd = uint32(wd)
        watchEntry.flags = flags
    }

    return nil
}
Copy the code

D. Delete the watch path

// Remove stops watching the named file or directory (non-recursively).
func (w *Watcher) Remove(name string) error {
    name = filepath.Clean(name) // Get the standard path

    // Multiple coroutines may be involved in writing the same watch item at the same time, so lock
    w.mu.Lock()
    defer w.mu.Unlock() // Unlock at last
    watch, ok := w.watches[name]

    if. {... }// Error handling

    // Remove the corresponding watch item from the context
    delete(w.paths, int(watch.wd))
    delete(w.watches, name)
    // Delete the fd of watch in inotify
    success, errno := unix.InotifyRmWatch(w.fd, watch.wd)
    if. {... }// Error handling
  
    return nil
}
Copy the code

E. Poller section (based on epoll)

Func NewWatcher() (*Watcher, error) calls poller, err := newFdPoller(fd), which registers inotify fd on epoll to implement efficient FS listening. The code is as follows:

Redundant code is omitted for brevity

func newFdPoller(fd int) (*fdPoller, error) {
    var errno error
    poller := emptyPoller(fd)
    defer func(a) {
        iferrno ! =nil {
            poller.close()
        }
    }()
    poller.fd = fd

    // To use epoll, create a separate fd for it using the EpollCreate function
    poller.epfd, errno = unix.EpollCreate1(unix.EPOLL_CLOEXEC)
    if. {return. }/ / error handling
  
    // To achieve an elegant exit, create a pipe, pipe[0] for reading and pipe[1] for writing
  // We will analyze this logic when we introduce watcher's Close function
    errno = unix.Pipe2(poller.pipe[:], unix.O_NONBLOCK|unix.O_CLOEXEC)
    if. {return. }/ / error handling

    // Register inotify fd to epoll
    event := unix.EpollEvent{
        Fd:     int32(poller.fd),
        Events: unix.EPOLLIN,
    }
    errno = unix.EpollCtl(poller.epfd, unix.EPOLL_CTL_ADD, poller.fd, &event)
    if. {return. }/ / error handling

    // Register the pipe fd to epoll
    event = unix.EpollEvent{
        Fd:     int32(poller.pipe[0]),
        Events: unix.EPOLLIN,
    }
    errno = unix.EpollCtl(poller.epfd, unix.EPOLL_CTL_ADD, poller.pipe[0], &event)
    if. {return. }/ / error handling

    return poller, nil
}
Copy the code

The function func newFdPoller(fd int) (*fdPoller, error) registers two files on epoll’s fd, one inotify and the other pipe[0] it uses to implement an elegant exit.

The ok, errno = w.poller.wait() statement we mentioned in the * event-listening coroutine func (w *Watcher) readEvents()* section above blocks until the event is received. To see how the poller(epoll above) handles events, the code looks like this:

Redundant code is omitted for brevity

func (poller *fdPoller) wait(a) (bool, error) {
    // Listen to two fd's in total: 1. Inotify 2. Pipe required for graceful exit [0]
    There are three possible events per fd, so up to six events can be fired
    // Prepare a slice larger than 6
    events := make([]unix.EpollEvent, 7)
    for {
        // Block wait on the fd of epoll. The -1 parameter indicates that the wait will not time out
        // When an event is generated, it will be sent to events
        n, errno := unix.EpollWait(poller.epfd, events, - 1) 
        if. {... }// All exception handling

        // The following is the processing of normal events received
        ready := events[:n]
        epollhup := false
        epollerr := false
        epollin := false
        for _, event := range ready {
            if event.Fd == int32(poller.fd) {
                ifevent.Events&unix.EPOLLHUP ! =0 {
                    // This should not happen, but if it does, treat it as a wakeup.
                    epollhup = true
                }
                ifevent.Events&unix.EPOLLERR ! =0 {
                    // If an error is waiting on the file descriptor, we should pretend
                    // something is ready to read, and let unix.Read pick up the error.
                    epollerr = true
                }
                ifevent.Events&unix.EPOLLIN ! =0 {
                    // Inotify has events
                    epollin = true}}if event.Fd == int32(poller.pipe[0]) {
                ifevent.Events&unix.EPOLLHUP ! =0 {
                    // Write pipe descriptor was closed, by us. This means we're closing down the
                    // watcher, and we should wake up.
                }
                ifevent.Events&unix.EPOLLERR ! =0 {
                    // If an error is waiting on the pipe file descriptor.
                    // This is an absolute mystery, and should never ever happen.
                    return false, errors.New("Error on the pipe descriptor.")}ifevent.Events&unix.EPOLLIN ! =0 {
                    // Receive an elegant exit event from the program and call clearWake to empty the pipe
                    err := poller.clearWake()
                    iferr ! =nil {
                        return false, err
                    }
                }
            }
        }

        if epollhup || epollerr || epollin {
            return true.nil
        }
        return false.nil}}Copy the code

ClearWake function, code as follows

func (poller *fdPoller) clearWake(a) error {
   // You have to be woken up a LOT in order to get to 100!
   buf := make([]byte.100)
   n, errno := unix.Read(poller.pipe[0], buf) // Read the exit signal in pipe[0]
   if. {... }// Error handling
   return nil
}
Copy the code

So how does the signal in pipe[0] come from? This means that there must be a place to write data to pipe[1]. In fact, as our sample code invokes watcher.close () in deferred mode, the most important step is to call w.poller.wake() as follows:

Redundant code is omitted for brevity

// Close the write end of the poller.
func (poller *fdPoller) wake(a) error {
    buf := make([]byte.1)
    // Here we write a character in pipe[1] as an exit signal
    n, errno := unix.Write(poller.pipe[1], buf)
    if. {... }// Error handling
    return nil
}
Copy the code

Aside: The early design for this elegant exit isn’t really like this, but the idea is similar. Check out fsnotify’s early submissions if you’re interested

At this point, the implementation of FSnotify on Linux is analyzed.