Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”
This article also participated in the “Digitalstar Project” to win a creative gift package and creative incentive money
introduce
This article describes how to adjust the gRPC data transfer size limit by RK-boot.
The GRPC size limit exists on the receiver, meaning there is no limit to how much data can be sent. The default size for receiving data is 4MB. The example uses google.golang.org/grpc v1.38.0.
What is gRPC data transfer size limit?
The default maximum data transfer size of gRPC server is 4MB. Sometimes, we need to transfer larger data, such as large images.
Please visit the following address for the full tutorial:
- rkdocs.netlify.app/cn
The installation
go get github.com/rookie-ninja/rk-boot
Copy the code
Quick start
Rk-boot supports adjusting size limits by means of code & YAML files.
For the full demonstration, we create a Greeter API.
1. Create a protobuf file
To compile protobuf using the buf command line, we need to create the following files.
The file name | describe |
---|---|
api/v1/greeter.proto | Protobuf file |
buf.yaml | Tell buf command line where to look for protobuf files |
buf.gen.yaml | Tell buf command line how to compile protobuf files |
- api/v1/greeter.proto
syntax = "proto3";
package api.v1;
option go_package = "api/v1/greeter";
service Greeter {
rpc Greeter (GreeterRequest) returns (GreeterResponse) {}}message GreeterRequest {
bytes msg = 1;
}
message GreeterResponse {}
Copy the code
- buf.yaml
version: v1beta1
name: github.com/rk-dev/rk-demo
build:
roots:
- api
Copy the code
- buf.gen.yaml
version: v1beta1
plugins:
# protoc-gen-go needs to be installed, generate go files based on proto files
- name: go
out: api/gen
opt:
- paths=source_relative
# protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
- name: go-grpc
out: api/gen
opt:
- paths=source_relative
- require_unimplemented_servers=false
Copy the code
- Compile the protobuf file
$ buf generate
Copy the code
2. Create the boot. Yaml
Boot. yaml allows us to unlimit the size, but not adjust the size.
To adjust limits, you can do so in code, which we’ll cover below.
---
grpc:
- name: greeter # Name of grpc entry
port: 8080 # Port of grpc entry
enabled: true # Enable grpc entry
noRecvMsgSizeLimit: true
Copy the code
3. Create a server. Go
We implemented the Greeter interface.
// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main
import (
"context"
"github.com/rookie-ninja/rk-boot"
"github.com/rookie-ninja/rk-demo/api/gen/v1"
"google.golang.org/grpc"
)
// Application entrance.
func main(a) {
// Create a new boot instance.
boot := rkboot.NewBoot()
// Get grpc entry with name
grpcEntry := boot.GetGrpcEntry("greeter")
grpcEntry.AddRegFuncGrpc(registerGreeter)
// Bootstrap
boot.Bootstrap(context.Background())
// Wait for shutdown sig
boot.WaitForShutdownSig(context.Background())
}
func registerGreeter(server *grpc.Server) {
greeter.RegisterGreeterServer(server, &GreeterServer{})
}
type GreeterServer struct{}
func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
return &greeter.GreeterResponse{}, nil
}
Copy the code
4. Create client. Go
We’re trying to transfer 10 megabytes of data.
// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main
import (
"context"
"github.com/rookie-ninja/rk-demo/api/gen/v1"
"google.golang.org/grpc"
"log"
)
func main(a) {
opts := []grpc.DialOption{
grpc.WithInsecure(),
grpc.WithBlock(),
}
// 1: Create grpc client
conn, client := createGreeterClient(opts...)
defer conn.Close()
kb := 1024
mb := 1024*kb
// 2: Call server with 10mb size of data
if _, err := client.Greeter(context.Background(), &greeter.GreeterRequest{Msg: make([]byte.10*mb, 10*mb)}); err ! =nil {
panic(err)
}
}
func createGreeterClient(opts ... grpc.DialOption) (*grpc.ClientConn, greeter.GreeterClient) {
// 1: Set up a connection to the server.
conn, err := grpc.DialContext(context.Background(), "localhost:8080", opts...)
iferr ! =nil {
log.Fatalf("Failed to connect: %v", err)
}
// 2: Create grpc client
client := greeter.NewGreeterClient(conn)
return conn, client
}
Copy the code
5. Folder structure
. ├ ─ ─ API │ ├ ─ ─ gen │ │ └ ─ ─ v1 │ │ ├ ─ ─ greeter. Pb. Go │ │ └ ─ ─ greeter_grpc. Pb. Go │ └ ─ ─ v1 │ └ ─ ─ greeter. Proto ├ ─ ─ Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass ExercisesCopy the code
6. Verify
There won’t be any mistakes.
$ go run server.go
$ go run client.go
Copy the code
Since the server is only allowed to receive 4MB of data by default, if we set noRecvMsgSizeLimit to false in boot.yaml, we will get the following error.
rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10485765 vs. 4194304)
Copy the code
Adjust the size of the transmitted data
In the last example, we used the noRecvMsgSizeLimit option to remove the size limit on the gRPC server. This time, we’ll try to resize. Use the above protobuf file again.
1. Modify the boot. Yaml
This time we set noRecvMsgSizeLimit to false.
---
grpc:
- name: greeter # Name of grpc entry
port: 8080 # Port of grpc entry
enabled: true # Enable grpc entry
noRecvMsgSizeLimit: false
Copy the code
2. Modify server. Go
We use the AddServerOptions() function to set the maximum value the server can receive.
// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main
import (
"context"
"github.com/rookie-ninja/rk-boot"
"github.com/rookie-ninja/rk-demo/api/gen/v1"
"google.golang.org/grpc"
)
// Application entrance.
func main(a) {
// Create a new boot instance.
boot := rkboot.NewBoot()
// Get grpc entry with name
grpcEntry := boot.GetGrpcEntry("greeter")
grpcEntry.AddRegFuncGrpc(registerGreeter)
/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / /
// *** Set server receive size to 20MB ***
/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / /
kb := 1024
mb := 1024*kb
grpcEntry.AddServerOptions(grpc.MaxRecvMsgSize(20*mb))
// Bootstrap
boot.Bootstrap(context.Background())
// Wait for shutdown sig
boot.WaitForShutdownSig(context.Background())
}
func registerGreeter(server *grpc.Server) {
greeter.RegisterGreeterServer(server, &GreeterServer{})
}
type GreeterServer struct{}
func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
return &greeter.GreeterResponse{}, nil
}
Copy the code
3. Verify
There won’t be any mistakes.
$ go run server.go
$ go run client.go
Copy the code
If we send more than 20MB of data, the following error occurs.
rpc error: code = ResourceExhausted desc = grpc: received message larger than max (31457285 vs. 20971520)
Copy the code
Adjust the size of data transferred from [client]
If the data returned by the server is larger than 4MB, we need to resize it on the client side.
kb := 1024
mb := 1024*kb
opts := []grpc.DialOption{
grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(20*mb)),
grpc.WithInsecure(),
grpc.WithBlock(),
}
Copy the code