IO.ReadAll is used to read the resp.Body returned by the remote interface.ReadAll is used to read the resp.
With this in mind, let’s take a closer look at the IO.ReadAll source code:
1. Demo reads the file content
package main
import (
"fmt"
"io"
"os"
)
func main(a) {
// Read the contents of the file
fileInfo, err := os.Open("./abc.go")
iferr ! =nil {
panic(err)
}
contentBytes, err := io.ReadAll(fileInfo)
iferr ! =nil {
panic(err)
}
fmt.Println(string(contentBytes))
}
Copy the codeIO.ReadAll reads all the data in the stream. Sliding Windows? Linearly/exponentially increasing read? Talk is cheap. Show me the code.
2. io.ReadAll Code
Go1.16 / SRC/IO/IO. Go# L626
// ReadAll reads from r until an error or EOF and returns the data it read.
// A successful call returns err == nil, not err == EOF. Because ReadAll is
// defined to read from src until EOF, it does not treat an EOF from Read
// as an error to be reported.
func ReadAll(r Reader) ([]byte, error) {
b := make([]byte.0.512)
for {
if len(b) == cap(b) {
// Add more capacity (let append pick how much).
b = append(b, 0)[:len(b)]
}
//println(cap(b))
n, err := r.Read(b[len(b):cap(b)])
b = b[:len(b)+n]
iferr ! =nil {
if err == EOF {
err = nil
}
return b, err
}
}
}
Copy the codeCap = 512 ([]byte); cap = 512 ([]byte); cap = 512 ([]byte); cap = 512 ([]byte); cap = 512 ([]byte); cap = 512 ([]byte); cap = 512 ([]byte); In the for loop, len(b) == cap(b) first determines whether b is full. If it is, append(b, 0) with an element.
We know that the capacity of a slice is insufficient and needs to be expanded, but what is the expansion mechanism? Go ahead and Show me the code.
3. Slice expansion mechanism
Go1.16 / SRC/runtime/slice. Go# L125
// growslice handles slice growth during append.
// It is passed the slice element type, the old slice, and the desired new minimum capacity,
// and it returns a new slice with at least that capacity, with the old data
// copied into it.
// The new slice's length is set to the old slice's length,
// NOT to the new requested capacity.
// This is for codegen convenience. The old slice's length is used immediately
// to calculate where to write new values during an append.
// TODO: When the old backend is gone, reconsider this decision.
// The SSA backend might prefer the new length or to return only ptr/cap and save stack space.
func growslice(et *_type, old slice, cap int) slice{... newcap := old.cap
doublecap := newcap + newcap
//println("newcap: ", newcap)
//println("cap: ", cap)
if cap > doublecap {
newcap = cap
} else {
if old.cap < 1024 {
newcap = doublecap
} else {
// Check 0 < newcap to detect overflow
// and prevent an infinite loop.
for 0 < newcap && newcap < cap {
newcap += newcap / 4
}
// Set newcap to the requested cap when
// the newcap calculation overflowed.
if newcap <= 0 {
newcap = cap}}}... }Copy the codeSlice expansion algorithm: 1). When the required capacity (CAP) exceeds twice of the original slice capacity (Doublecap), the required capacity will be used as the new capacity (NewCap). 2). When the original section capacity < 1024, the capacity of the new section (Newcap) will directly double (Doublecap); 3). When the original slice capacity >= 1024, the original slice capacity will be repeatedly increased by 1/4 until the new capacity (Newcap) exceeds the required capacity;
For example, in the IO.ReadAll source code, the initial slice cap = 512.
512
1024(doublecap)
1280(1024 + 1024/4)
1600(1280 + 1280/4)
2000(1600 + 1600/4)...Copy the codeIs the actual expansion CAP like this? Let’s verify:
before newcap: 1024
-after newcap: 1024
before newcap: 1280
-after newcap: 1280
before newcap: 1600
-after newcap: 1792
before newcap: 2240
-after newcap: 2304
Copy the codeStrange? After NewCap does not expand the memory size as expected. After NewCap does not expand the memory size as expected. After NewCap does not expand the memory size as expected.
Go1.16 / SRC/runtime/slice. Go# L198
println("before newcap: ", newcap)
var overflow bool
var lenmem, newlenmem, capmem uintptr
// Specialize for common values of et.size.
// For 1 we don't need any division/multiplication.
// For sys.PtrSize, compiler will optimize division/multiplication into a shift by a constant.
// For powers of 2, use a variable shift.
switch{...case isPowerOfTwo(et.size):
var shift uintptr
if sys.PtrSize == 8 {
// Mask shift for better code generation.
shift = uintptr(sys.Ctz64(uint64(et.size))) & 63
} else {
shift = uintptr(sys.Ctz32(uint32(et.size))) & 31
}
lenmem = uintptr(old.len) << shift
newlenmem = uintptr(cap) << shift
capmem = roundupsize(uintptr(newcap) << shift) // Enter the memory block allocation
overflow = uintptr(newcap) > (maxAlloc >> shift)
newcap = int(capmem >> shift)
...
}
println("after newcap: ", newcap)
Copy the codeInto the memory block (block) memory allocation: go1.16 / SRC/runtime/msize go# L13
// Returns size of the memory block that mallocgc will allocate if you ask for the size.
func roundupsize(size uintptr) uintptr {
if size < _MaxSmallSize {
if size <= smallSizeMax- 8 - {
return uintptr(class_to_size[size_to_class8[divRoundUp(size, smallSizeDiv)]])
} else {
return uintptr(class_to_size[size_to_class128[divRoundUp(size-smallSizeMax, largeSizeDiv)]])
}
}
if size+_PageSize < size {
return size
}
return alignUp(size, _PageSize)
}
Copy the codeFor spanClass corresponding size: go1.16 / SRC/runtime/sizeclasses go# L84
const (
_NumSizeClasses = 68
)
var class_to_size = [_NumSizeClasses]uint16{0.8.16.24.32.48.64.80.96.112.128.144.160.176.192.208.224.240.256.288.320.352.384.416.448.480.512.576.640.704.768.896.1024.1152.1280.1408.1536.1792.2048.2304.2688.3072.3200.3456.4096.4864.5376.6144.6528.6784.6912.8192.9472.9728.10240.10880.12288.13568.14336.16384.18432.19072.20480.21760.24576.27264.28672.32768}
Copy the codeAs you can see from the 68 spanclasses above, we want 1600 to be aligned to 1792,2240 to 2304, in line with the following verification results:
before newcap: 1024
-after newcap: 1024
before newcap: 1280
-after newcap: 1280
before newcap: 1600
-after newcap: 1792
before newcap: 2240
-after newcap: 2304
Copy the code4. Summary
IO.ReadAll uses the increased capacity to read the entire IO stream using the slice Append automatic expansion + memory alignment mechanism. The slice Append expansion algorithm is as follows: 1). When the required capacity (CAP) exceeds twice of the original slice capacity (Doublecap), the required capacity will be used as the new capacity (NewCap); 2). When the original section capacity < 1024, the capacity of the new section (Newcap) will directly double (Doublecap); 3). When the original slice capacity >= 1024, the original slice capacity will be repeatedly increased by 1/4 until the new capacity (Newcap) exceeds the required capacity;
There will be more articles on memory allocation, GC mechanism, GPM scheduling, interview series, K8s series, ETCD series, etc. Finally, I wish everyone a happy Dragon Boat Festival ~