Problem specification

Recently, DUE to the need of the project, I have been engaged in some golang development and encountered many problems. In addition to the pit about ActiveMQ I stepped on before, I also stepped on a pit in the HTTP library when I was using Gin for Web development recently.

Gin itself is a simple wrapper, and most of its functionality is based on the NET/HTTP standard library, including Request.

The two problems I encountered this time were caused by one reason: I sometimes lost request parameters when I was processing POST requests to receive file uploads (upload files can be received, but the POST parameters along with them are empty).

It was found that the python requests were handled properly, but the Java requests were not.

To debug the problem, I tried to log the request and ran into the first problem:

Request.Body is a ReaderCloser buffer that can only be read once. If I read it to log, the following arguments will be empty, and if I take arguments, the buffer will be empty.

The second problem, of course, is that the POST request is missing, for reasons explained below.

A lossless way to read Request.Body

I have tried both the official documentation and Stackoverflow methods, such as write back after reading, or Bind, but I can’t get the content properly. Finally, I saw a method mentioned in the comments of a foreigner’s BLOG:

httputil.DumpRequest

I finally solved the problem and wrote a debug-specific method:

func req_body(c *gin.Context) string {
    body, err := httputil.DumpRequest(c.Request, true)
    iferr ! = nil { println(err) }return string(body[:1024])
}
Copy the code

The POST parameter is missing

With the above method, I finally found the problem. Note the differences between python and Java requests:

python:

Connection: close Accept: */* Accept-Encoding: gzip, deflate Connection: close Content-Length: xxx Content-Type: multipart/form-data; A boundary = 5 b9ca15e3fd14316b6a4b03cb4ee4de2 the user-agent: python - requests / 2.12.3 X-ray Forwarded - For: 120.36 xx X - Real - Ip: 120.36 xx - 5 b9ca15e3fd14316b6a4b03cb4ee4de2 Content - Disposition: the form - data; name="index"

1
--5b9ca15e3fd14316b6a4b03cb4ee4de2
Content-Disposition: form-data; name="checksum"

testchecksum
Copy the code

Java:

Connection: close Connection: close Content-Length: xxx Content-Type: multipart/form-data; Boundary = jQsrKZ2MjZuub4YOADN-Fvui_VSjn0SXHHzz D User-Agent: Apache-httpClient /4.5.2 (Java/1.7.0_79) X-Forwarded-For: 1.192.xx X-REAL-IP: 1.192.xx -- JQSrKZ2MJZuub4YOADN-FVUi_VSjn0SXHHZZd Content-disposition: form-data; name="index"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

2
--JqsRkZ2MjzUUb4YOaDn-FvuI_vsjn0sXhhZZd
Content-Disposition: form-data; name="checksum"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

testchecksum
Copy the code

Obviously, the Java version has something in the header of each part that should be causing HTTP multipart parsing to fail.

But I’ve tried PostForm and MultipartForm again and again, and I can’t get the POST argument — PostForm gets empty, and MultipartForm gets empty form.Value.

Finally, I hit upon the idea of trying the form.File of MultipartForm, and found that all the parameters were parsed here… I don’t know how the HTTP standard library was designed, but did so many Go users not notice? Or are there just not that many users of Go?

In order to solve this problem, I had to write some code to transfer the parameter of File to Value:

func req_getpart(v *multipart.FileHeader) (string, error) {
    f, err := v.Open()
    iferr ! = nil { println("Open fail")
        return "", err
    }
    buf, err := ioutil.ReadAll(f)
    defer f.Close()
    iferr ! = nil { println("Read fail")
        return "", err
    }
    return string(buf), nil
}

func req_multipart(c *gin.Context) *multipart.Form {
    form, _ := c.MultipartForm()
    if len(form.Value) == 0 && len(form.File) > 0 {
        for k, v := range form.File {
            if len(v) > 0 {
                buf, err := req_getpart(v[0])
                iferr ! = nil {continue
                }
                form.Value[k] = append(form.Value[k], buf)
            }
        }
        for k, _ := range form.Value {
            delete(form.File, k)
        }
    }
    return form
}
Copy the code