Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”

This article also participated in the “Digitalstar Project” to win a creative gift package and creative incentive money

introduce

This article describes how to adjust the gRPC data transfer size limit by RK-boot.

The GRPC size limit exists on the receiver, meaning there is no limit to how much data can be sent. The default size for receiving data is 4MB. The example uses google.golang.org/grpc v1.38.0.

What is gRPC data transfer size limit?

The default maximum data transfer size of gRPC server is 4MB. Sometimes, we need to transfer larger data, such as large images.

Please visit the following address for the full tutorial:

  • rkdocs.netlify.app/cn

The installation

go get github.com/rookie-ninja/rk-boot
Copy the code

Quick start

Rk-boot supports adjusting size limits by means of code & YAML files.

For the full demonstration, we create a Greeter API.

1. Create a protobuf file

To compile protobuf using the buf command line, we need to create the following files.

The file name describe
api/v1/greeter.proto Protobuf file
buf.yaml Tell buf command line where to look for protobuf files
buf.gen.yaml Tell buf command line how to compile protobuf files
  • api/v1/greeter.proto
syntax = "proto3";

package api.v1;

option go_package = "api/v1/greeter";

service Greeter {
  rpc Greeter (GreeterRequest) returns (GreeterResponse) {}}message GreeterRequest {
  bytes msg = 1;
}

message GreeterResponse {}
Copy the code
  • buf.yaml
version: v1beta1
name: github.com/rk-dev/rk-demo
build:
  roots:
    - api
Copy the code
  • buf.gen.yaml
version: v1beta1
plugins:
  # protoc-gen-go needs to be installed, generate go files based on proto files
  - name: go
    out: api/gen
    opt:
     - paths=source_relative
  # protoc-gen-go-grpc needs to be installed, generate grpc go files based on proto files
  - name: go-grpc
    out: api/gen
    opt:
      - paths=source_relative
      - require_unimplemented_servers=false
Copy the code
  • Compile the protobuf file
$ buf generate
Copy the code

2. Create the boot. Yaml

Boot. yaml allows us to unlimit the size, but not adjust the size.

To adjust limits, you can do so in code, which we’ll cover below.

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    noRecvMsgSizeLimit: true
Copy the code

3. Create a server. Go

We implemented the Greeter interface.

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
)

// Application entrance.
func main(a) {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get grpc entry with name
	grpcEntry := boot.GetGrpcEntry("greeter")
	grpcEntry.AddRegFuncGrpc(registerGreeter)

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{}, nil
}
Copy the code

4. Create client. Go

We’re trying to transfer 10 megabytes of data.

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
	"log"
)

func main(a) {
	opts := []grpc.DialOption{
		grpc.WithInsecure(),
		grpc.WithBlock(),
	}

	// 1: Create grpc client
	conn, client := createGreeterClient(opts...)
	defer conn.Close()

	kb := 1024
	mb := 1024*kb

	// 2: Call server with 10mb size of data
	if _, err := client.Greeter(context.Background(), &greeter.GreeterRequest{Msg: make([]byte.10*mb, 10*mb)}); err ! =nil {
		panic(err)
	}
}

func createGreeterClient(opts ... grpc.DialOption) (*grpc.ClientConn, greeter.GreeterClient) {
	// 1: Set up a connection to the server.
	conn, err := grpc.DialContext(context.Background(), "localhost:8080", opts...)
	iferr ! =nil {
		log.Fatalf("Failed to connect: %v", err)
	}

	// 2: Create grpc client
	client := greeter.NewGreeterClient(conn)

	return conn, client
}
Copy the code

5. Folder structure

. ├ ─ ─ API │ ├ ─ ─ gen │ │ └ ─ ─ v1 │ │ ├ ─ ─ greeter. Pb. Go │ │ └ ─ ─ greeter_grpc. Pb. Go │ └ ─ ─ v1 │ └ ─ ─ greeter. Proto ├ ─ ─ Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass Exercises ── Bass ExercisesCopy the code

6. Verify

There won’t be any mistakes.

$ go run server.go
$ go run client.go
Copy the code

Since the server is only allowed to receive 4MB of data by default, if we set noRecvMsgSizeLimit to false in boot.yaml, we will get the following error.

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10485765 vs. 4194304)
Copy the code

Adjust the size of the transmitted data

In the last example, we used the noRecvMsgSizeLimit option to remove the size limit on the gRPC server. This time, we’ll try to resize. Use the above protobuf file again.

1. Modify the boot. Yaml

This time we set noRecvMsgSizeLimit to false.

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    noRecvMsgSizeLimit: false
Copy the code

2. Modify server. Go

We use the AddServerOptions() function to set the maximum value the server can receive.

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.
package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-demo/api/gen/v1"
	"google.golang.org/grpc"
)

// Application entrance.
func main(a) {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Get grpc entry with name
	grpcEntry := boot.GetGrpcEntry("greeter")
	grpcEntry.AddRegFuncGrpc(registerGreeter)
	
	/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / /
	// *** Set server receive size to 20MB ***
	/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / /
	kb := 1024
	mb := 1024*kb
	grpcEntry.AddServerOptions(grpc.MaxRecvMsgSize(20*mb))

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

func registerGreeter(server *grpc.Server) {
	greeter.RegisterGreeterServer(server, &GreeterServer{})
}

type GreeterServer struct{}

func (server *GreeterServer) Greeter(ctx context.Context, request *greeter.GreeterRequest) (*greeter.GreeterResponse, error) {
	return &greeter.GreeterResponse{}, nil
}
Copy the code

3. Verify

There won’t be any mistakes.

$ go run server.go
$ go run client.go
Copy the code

If we send more than 20MB of data, the following error occurs.

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (31457285 vs. 20971520)
Copy the code

Adjust the size of data transferred from [client]

If the data returned by the server is larger than 4MB, we need to resize it on the client side.

kb := 1024
mb := 1024*kb

opts := []grpc.DialOption{
	grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(20*mb)),
	grpc.WithInsecure(),
	grpc.WithBlock(),
}
Copy the code