“This is the 21st day of my participation in the 21st Century Gwen Challenge. See details: The last Gwen Challenge 2021”

In this article, I want to show Pipe, a function of the Go library IO

func Pipe(a) (*PipeReader, *PipeWriter)
Copy the code

Pipe creates a synchronous in-memory pipe. It can be used to connect code expecting an io.Reader with code expecting an io.Writer.

Reads and Writes on the pipe are matched one to one except when multiple Reads are needed to consume a single Write. That is, each Write to the PipeWriter blocks until it has satisfied one or more Reads from the PipeReader that fully consume the written data. The data is copied directly from the Write to the corresponding Read (or Reads); there is no internal buffering.

It is safe to call Read and Write in parallel with each other or with Close. Parallel calls to Read and parallel calls to Write are also safe: the individual calls will be gated sequentially.

According to the documentation, IO.Pipe creates a synchronous memory Pipe that can be used to connect code that needs IO.Reader to code that needs IO.

When called, io.pipe () returns a PipeReader and a PipeWriter. They are connected (pipes), so everything written to the PipeWriter can be read from the PipeReader.

Here are three chestnuts to illustrate the use of IO.Pipe and the features it brings when combined with I/O.

Example1: JSON to HTTP Request

When we encode some data as JSON and want to send it to the Web via HTTP. Post, we usually encode the memory data through json.Encoder first, and then feed the result to http.Post as input. But the JSON encoder needs an IO.Writer, and the HTTP Post method needs an IO.Reader as input, so we can’t just concatenate them together. We need to convert between them with []bytes. We can also use IO.Pipe to do this.

pr, pw := io.Pipe()

go func(a) {
    // close the writer, so the reader knows there's no more data
    defer pw.Close()

    // write json data to the PipeReader through the PipeWriter
    if err := json.NewEncoder(pw).Encode(&PayLoad{Content: "Hello Pipe!"}); err ! =nil {
        log.Fatal(err)
    }
}()

// JSON from the PipeWriter lands in the PipeReader
/ /... and we send it off...
if _, err := http.Post("http://example.com"."application/json", pr); err ! =nil {
    log.Fatal(err)
}
Copy the code

First, we encode the structure PayLoad as JSON and write the data to the PipeWriter created by calling IO.Pipe. After that, we create an HTTP POST request that gets its data from the Piper Ader. The PipeReader is filled with data written to the PipeWriter.

The important thing to note here is that we must code asynchronously to prevent deadlocks, because if we don’t have a reader, we will write without a reader.

This practical example is a good example of the versatility of IO.Pipe. It really incentivizes Gophers to build components using IO.Reader and IO.Writer without having to worry about using them together.

Split up Data with TeeReader

I found another way to combine IO.Pipe with TeeReader, which produces a similar effect to streaming images. Pipe and TeeReader are used to transfer a video file to another format and upload the original file at the same time, with minimal overhead and complete parallelism.

The simplified code looks like this:

pr, pw := io.Pipe()

// we need to wait for everything to be done
wg := sync.WaitGroup{}
wg.Add(2)

// we get some file as input
f, err := os.Open("./fruit.txt")
iferr ! =nil {
    log.Fatal(err)
}

// TeeReader gets the data from the file and also writes it to the PipeWriter
tr := io.TeeReader(f, pw) 

go func(a) {
    defer wg.Done()
    defer pw.Close()

    // get data from the TeeReader, which feeds the PipeReader through the PipeWriter
    _, err := http.Post("https://example.com"."text/html", tr)
    iferr ! =nil {
        log.Fatal(err)
    }
}()

go func(a) {
    defer wg.Done()
    // read from the PipeReader to stdout
    if_, err := io.Copy(os.Stdout, pr); err ! =nil {
        log.Fatal(err)
    }
}()

wg.Wait()
Copy the code

We have some kind of input IO.Reader, in this case a file, and create a TeeReader, which returns a Reader that will write to the Writer you provide, and everything it reads from the Reader you provide.

Now we start two Goroutines, one that simply prints the data to STdout and the other that sends the data to the HTTP endpoint. TeeReader uses IO.Pipe to split the given input. When TeeReader is used, the PipeReader also receives these same bytes.

Example 3: Piping the output of Shell commands

The third way is to combine IO.Pipe with OS.exec. Basically, it does what the task runner in most CI services (such as Jenkins or Travis CI) does, which is execute some shell commands and display their output on some web site.

The simplified code looks like this:

pr, pw := io.Pipe()
defer pw.Close()

// tell the command to write to our pipe
cmd := exec.Command("cat"."fruit.txt")
cmd.Stdout = pw

go func(a) {
    defer pr.Close()
    // copy the data written to the PipeReader via the cmd to stdout
    if_, err := io.Copy(os.Stdout, pr); err ! =nil {
        log.Fatal(err)
    }
}()

// run the command, which writes all output to the PipeWriter
// which then ends up in the PipeReader
iferr := cmd.Run(); err ! =nil {
    log.Fatal(err)
}
Copy the code

First, we define the command — cat — a file called fruit.txt, which puts the contents of the file on standard output. We then set the standard output of the command to our PipeWriter.

So we redirect the output of the command to our pipe as before, which will allow us to read it at another point through our PipeReader.