GRPC, critical praise

Original link: gRPC, awesome

GRPC is a great technology, with strict interface constraints and high performance, and is used in K8S and many microservices frameworks.

As a programmer, it’s the right thing to do.

Having written some gRPC services in Python, I’m ready to use Go to get a taste of gRPC programming.

The feature of this article is to speak directly with the code, through the complete code out of the box, to introduce the various use of gRPC.

The code has been uploaded to GitHub, so let’s get started.

introduce

GRPC is a cross-language open source RPC framework developed by Google based on Protobuf. GRPC is designed based on the HTTP/2 protocol and can provide multiple services based on a single HTTP/2 link, making it more mobile-friendly.

An introduction to

The first step is to define the PROTO file. Since gRPC is also a C/S architecture, this step is equivalent to specifying the interface specification.

proto

syntax = "proto3";

package proto;

// The greeting service definition.
service Greeter {
    // Sends a greeting
    rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
    string name = 1;
}

// The response message containing the greetings
message HelloReply {
    string message = 1;
}
Copy the code

Generate gRPC code using protoc-gen-Go built-in gRPC plug-in:

protoc --go_out=plugins=grpc:. helloworld.proto
Copy the code

After executing this command, a helloworld.pb.go file is generated in the current directory, which defines the server and client interfaces respectively:

// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type GreeterClient interface {
	// Sends a greetingSayHello(ctx context.Context, in *HelloRequest, opts ... grpc.CallOption) (*HelloReply, error) }// GreeterServer is the server API for Greeter service.
type GreeterServer interface {
	// Sends a greeting
	SayHello(context.Context, *HelloRequest) (*HelloReply, error)
}
Copy the code

Next is to write the server and client side of the code, respectively, to implement the corresponding interface.

server

package main

import (
	"context"
	"fmt"
	"grpc-server/proto"
	"log"
	"net"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"
)

type greeter struct{}func (*greeter) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {
	fmt.Println(req)
	reply := &proto.HelloReply{Message: "hello"}
	return reply, nil
}

func main(a) {
	lis, err := net.Listen("tcp".": 50051")
	iferr ! =nil {
		log.Fatalf("failed to listen: %v", err)
	}

	server := grpc.NewServer()
	// Register the reflection service required by Grpcurl
	reflection.Register(server)
	// Register business services
	proto.RegisterGreeterServer(server, &greeter{})

	fmt.Println("grpc server start ...")
	iferr := server.Serve(lis); err ! =nil {
		log.Fatalf("failed to serve: %v", err)
	}
}
Copy the code

client

package main

import (
	"context"
	"fmt"
	"grpc-client/proto"
	"log"

	"google.golang.org/grpc"
)

func main(a) {
	conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
	iferr ! =nil {
		log.Fatal(err)
	}
	defer conn.Close()

	client := proto.NewGreeterClient(conn)
	reply, err := client.SayHello(context.Background(), &proto.HelloRequest{Name: "zhangsan"})
	iferr ! =nil {
		log.Fatal(err)
	}
	fmt.Println(reply.Message)
}
Copy the code

This completes the development of the basic gRPC service, and we continue to enrich this “basic template” to learn more features.

Flow way

Let’s look at streaming. As the name suggests, data can be sent and received continuously.

Flow can be divided into unidirectional flow and bidirectional flow. Here, we directly take bidirectional flow as an example.

proto

service Greeter {
    // Sends a greeting
    rpc SayHello (HelloRequest) returns (HelloReply) {}
    // Sends stream message
    rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {}
}
Copy the code

Add a stream function SayHelloStream to specify stream properties by stream keyword.

You need to regenerate the helloWorld.pb. go file, which I won’t go into here.

server

func (*greeter) SayHelloStream(stream proto.Greeter_SayHelloStreamServer) error {
	for {
		args, err := stream.Recv()
		iferr ! =nil {
			if err == io.EOF {
				return nil
			}
			return err
		}

		fmt.Println("Recv: " + args.Name)
		reply := &proto.HelloReply{Message: "hi " + args.Name}

		err = stream.Send(reply)
		iferr ! =nil {
			return err
		}
	}
}
Copy the code

Add the SayHelloStream function to the base template, nothing else needs to be changed.

client

client := proto.NewGreeterClient(conn)

/ / stream processing
stream, err := client.SayHelloStream(context.Background())
iferr ! =nil {
	log.Fatal(err)
}

// Send a message
go func(a) {
	for {
		if err := stream.Send(&proto.HelloRequest{Name: "zhangsan"}); err ! =nil {
			log.Fatal(err)
		}
		time.Sleep(time.Second)
	}
}()

// Receive the message
for {
	reply, err := stream.Recv()
	iferr ! =nil {
		if err == io.EOF {
			break
		}
		log.Fatal(err)
	}
	fmt.Println(reply.Message)
}
Copy the code

Messages are sent through a Goroutine and received by the main program’s for loop.

The execution program will find that there are constant printouts from both the server and the client.

The validator

Next comes the validator, a requirement that comes naturally to mind as it involves requests between interfaces, so it is necessary to properly validate the parameters.

We use Protoc-gen-GoValidators and Go-grpc-Middleware to do this.

First installation:

go get github.com/mwitkow/go-proto-validators/protoc-gen-govalidators

go get github.com/grpc-ecosystem/go-grpc-middleware
Copy the code

Next modify the proto file:

proto

The import "github.com/mwitkow/[email protected]/validator.proto"; Message HelloRequest {string name = 1 [(validator.field) = {regex: "^[z]{2,5}$"}]; }Copy the code

Here, the name parameter is verified to meet the requirements of the re before normal requests can be made.

There are other validation rules, such as validation of numeric size, which I won’t cover here.

Next generate the *.pb.go file:

Protoc \ -- proto_path = ${GOPATH} / PKG/mod \ -- proto_path = ${GOPATH} / pkg/mod/github.com/gogo/[email protected] \ --proto_path=. \ --govalidators_out=. --go_out=plugins=grpc:.\ *.protoCopy the code

Execute after success, will be more than a helloworld directory. The validator. Pb. Go file.

It is important to note that the simple command used before does not work, requiring multiple proto_path parameters to specify the directory to import the proto file.

There are two dependency cases officially given, one for Google Protobuf and one for Gogo Protobuf. I’m using the second one here.

Even with the above command, it is possible to get this error:

Import "github.com/mwitkow/go-proto-validators/validator.proto" was not found or had errors
Copy the code

But don’t panic, it’s probably a reference path problem, be careful about your installed version, and the specific path in GOPATH.

Finally, server-side code modification:

The introduction of package:

grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
grpc_validator "github.com/grpc-ecosystem/go-grpc-middleware/validator"
Copy the code

Then add the validator function at initialization:

server := grpc.NewServer(
	grpc.UnaryInterceptor(
		grpc_middleware.ChainUnaryServer(
			grpc_validator.UnaryServerInterceptor(),
		),
	),
	grpc.StreamInterceptor(
		grpc_middleware.ChainStreamServer(
			grpc_validator.StreamServerInterceptor(),
		),
	),
)
Copy the code

After we start the program, we will receive an error when we request the same client code:

2021/10/11 18:32:59 rpc error: code = InvalidArgument desc = invalid field Name: Value 'zhangsan' must be a string corner-point to regex "^[z]{2,5}$" exit status 1Copy the code

Because name: zhangsan does not meet the requirement of correct service, but if name: ZZZ is passed, it can return normally.

Token authentication

Token authentication is a Token authentication.

With the previous experience with validators, you can do the same thing by writing an interceptor and then injecting it when initializing the server.

Authentication function:

func Auth(ctx context.Context) error {
	md, ok := metadata.FromIncomingContext(ctx)
	if! ok {return fmt.Errorf("missing credentials")}var user string
	var password string

	if val, ok := md["user"]; ok {
		user = val[0]}if val, ok := md["password"]; ok {
		password = val[0]}ifuser ! ="admin"|| password ! ="admin" {
		return grpc.Errorf(codes.Unauthenticated, "invalid token")}return nil
}
Copy the code

Metadata. FromIncomingContext from the context reading user name and password, and then compare the actual data, determine whether through certification.

The interceptor:

var authInterceptor grpc.UnaryServerInterceptor
authInterceptor = func(
	ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,
) (resp interface{}, err error) {
	// Intercepts normal method requests to validate tokens
	err = Auth(ctx)
	iferr ! =nil {
		return
	}
	// Continue processing the request
	return handler(ctx, req)
}
Copy the code

Initialization:

server := grpc.NewServer(
	grpc.UnaryInterceptor(
		grpc_middleware.ChainUnaryServer(
			authInterceptor,
			grpc_validator.UnaryServerInterceptor(),
		),
	),
	grpc.StreamInterceptor(
		grpc_middleware.ChainStreamServer(
			grpc_validator.StreamServerInterceptor(),
		),
	),
)
Copy the code

AuthInterceptor is the Token authentication interceptor.

Finally, the clients need to implement the PerRPCCredentials interface.

type PerRPCCredentials interface {
    // GetRequestMetadata gets the current request metadata, refreshing
    // tokens if required. This should be called by the transport layer on
    // each request, and the data should be populated in headers or other
    // context. If a status code is returned, it will be used as the status
    // for the RPC. uri is the URI of the entry point for the request.
    // When supported by the underlying implementation, ctx can be used for
    // timeout and cancellation.
    // TODO(zhaoq): Define the set of the qualified keys instead of leaving
    // it as an arbitrary string.
    GetRequestMetadata(ctx context.Context, uri ...string) (
        map[string]string,    error,
    )
    // RequireTransportSecurity indicates whether the credentials requires
    // transport security.
    RequireTransportSecurity() bool
}
Copy the code

The GetRequestMetadata method returns the required information for authentication, and the RequireTransportSecurity method indicates whether secure links are enabled. In production environments, this is usually enabled, but for testing purposes, it is not enabled here.

Implementation interface:

type Authentication struct {
	User     string
	Password string
}

func (a *Authentication) GetRequestMetadata(context.Context, ...string) (
	map[string]string, error,
) {
	return map[string]string{"user": a.User, "password": a.Password}, nil
}

func (a *Authentication) RequireTransportSecurity(a) bool {
	return false
}
Copy the code

Connection:

conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithPerRPCCredentials(&auth))
Copy the code

Well, now our service has Token authentication. If the username or password is incorrect, the client will receive:

2021/10/11 20:39:35 rpc error: code = Unauthenticated desc = invalid token
exit status 1
Copy the code

If the user name and password are correct, it can be returned normally.

One-way certificate authentication

There are two methods of certificate authentication:

One-way authentication
Two-way authentication

Let’s take a look at one-way authentication:

Generate a certificate

The openSSL tool is used to generate a self-signed SSL certificate.

1, generate private key:

openssl genrsa -des3 -out server.pass.key 2048
Copy the code

2, remove the private key password:

openssl rsa -in server.pass.key -out server.key
Copy the code

3. Generate a CSR file:

openssl req -new -key server.key -out server.csr -subj "/C=CN/ST=beijing/L=beijing/O=grpcdev/OU=grpcdev/CN=example.grpcdev.cn"
Copy the code

4. Generate certificate:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Copy the code

One more word about the three files included in x.509 certificates: key, CSR, and CRT.

Key: private key file on the server, used to encrypt data sent to and decrypt data received from the client.
CSR: Certificate signing request file that is submitted to a certificate authority (CA) to sign a certificate.
CRT: a certificate signed by a certificate authority (CA) or a self-signed certificate by the developer. It contains information about the certificate holder, the public key of the holder, and the signature of the signer.

GRPC code

Once the certificate is in place, all that remains is to modify the program, starting with the server-side code.

// Certificate authentication - one-way authentication
creds, err := credentials.NewServerTLSFromFile("keys/server.crt"."keys/server.key")
iferr ! =nil {
	log.Fatal(err)
	return
}

server := grpc.NewServer(grpc.Creds(creds))
Copy the code

There are only a few lines of code that need to be changed, simple enough, and then the client.

Because the authentication mode is one-way, you do not need to generate a certificate for the client. You only need to copy the CRT file of the server to the corresponding directory of the client.

// Certificate authentication - one-way authentication
creds, err := credentials.NewClientTLSFromFile("keys/server.crt"."example.grpcdev.cn")
iferr ! =nil {
	log.Fatal(err)
	return
}
conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(creds))
Copy the code

Ok, now our service supports one-way certificate authentication.

But that’s not all. Here’s a possible problem:

2021/10/11 21:32:37 rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0"
exit status 1
Copy the code

CommonName is deprecated in Go 1.15 and SAN certificates are recommended. If you want to be compatible with the previous mode, you can set the environment variable to support, as follows:

export GODEBUG="x509ignoreCN=0"
Copy the code

Note, however, that as of Go 1.17, environment variables are no longer in effect and must be used via SAN. So, for the upcoming Go version upgrade, it’s better to support it as soon as possible.

Two-way certificate authentication

Finally, look at two-way certificate authentication.

A certificate with SAN is generated

Again, the certificate will be generated, but this time with a difference, we need to generate the certificate with the SAN extension.

What is a SAN?

SAN (Subject Alternative Name) is an extension defined in SSL standard X509. SSL certificates that use SAN fields can extend the domain names supported by the certificate so that one certificate can support the resolution of multiple domain names.

Copy the default OpenSSL configuration file to the current directory.

Linux is available in:

/etc/pki/tls/openssl.cnf
Copy the code

Mac operating system:

/System/Library/OpenSSL/openssl.cnf
Copy the code

Modify the temporary configuration file by finding the [req] paragraph and uncomment the following statement.

req_extensions = v3_req # The extensions to add to a certificate request
Copy the code

Then add the following configuration:

[ v3_req ]
# Extensions to add to a certificate request

basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = www.example.grpcdev.cn
Copy the code

The [alt_names] location can be configured with multiple domain names, for example:

[ alt_names ]
DNS.1 = www.example.grpcdev.cn
DNS.2 = www.test.grpcdev.cn
Copy the code

For testing convenience, only one domain name is configured here.

1. Generate a CA certificate:

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -key ca.key -subj "/CN=example.grpcdev.com" -days 5000 -out ca.pem
Copy the code

Generate server certificate:

Openssl req -new-nodes \ -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \ -config <(cat openssl.cnf \ <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \ -keyout server.key \ -out Csr-ca ca.pem -cakey ca.key -cacreateserial \ -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \ -out server.pemCopy the code

3. Generate client certificate:

Openssl req -new-nodes \ -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \ -config <(cat openssl.cnf \ <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \ -keyout client.key \ -out Csr-ca ca.pem -cakey ca.key -cacreateserial \ -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \ -out client.pemCopy the code

GRPC code

To start modifying the code, look at the server first:

// Certificate authentication - two-way authentication
// Read and parse information from the certificate file to obtain the certificate public key and key pair
cert, _ := tls.LoadX509KeyPair("cert/server.pem"."cert/server.key")
Create a new, empty CertPool
certPool := x509.NewCertPool()
ca, _ := ioutil.ReadFile("cert/ca.pem")
// Try to parse the PEM encoded certificate passed in. If the parse succeeds, it will be added to CertPool for later use
certPool.AppendCertsFromPEM(ca)
// Build the TransportCredentials option based on TLS
creds := credentials.NewTLS(&tls.Config{
	// Set the certificate chain to allow one or more certificates
	Certificates: []tls.Certificate{cert},
	// The client certificate must be verified. You can set the following parameters as required
	ClientAuth: tls.RequireAndVerifyClientCert,
	// Set the root certificate set. The verification mode is set in ClientAuth
	ClientCAs: certPool,
})
Copy the code

Now look at the client:

// Certificate authentication - two-way authentication
// Read and parse information from the certificate file to obtain the certificate public key and key pair
cert, _ := tls.LoadX509KeyPair("cert/client.pem"."cert/client.key")
Create a new, empty CertPool
certPool := x509.NewCertPool()
ca, _ := ioutil.ReadFile("cert/ca.pem")
// Try to parse the PEM encoded certificate passed in. If the parse succeeds, it will be added to CertPool for later use
certPool.AppendCertsFromPEM(ca)
// Build the TransportCredentials option based on TLS
creds := credentials.NewTLS(&tls.Config{
	// Set the certificate chain to allow one or more certificates
	Certificates: []tls.Certificate{cert},
	// The client certificate must be verified. You can set the following parameters as required
	ServerName: "www.example.grpcdev.cn",
	RootCAs:    certPool,
})
Copy the code

And you’re done.

The Python client

Having said that gRPC is cross-language, we end this article by writing a client in Python that requests the Go server.

Use the simplest way to do this:

The proto file uses the original “base template” proto file:

syntax = "proto3"; package proto; // The greeting service definition. service Greeter { // Sends a greeting rpc SayHello (HelloRequest) returns (HelloReply) {} // Sends stream message rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {} } // The  request message containing the user's name. message HelloRequest { string name = 1; } // The response message containing the greetings message HelloReply { string message = 1; }Copy the code

Similarly, the pb.py file needs to be generated from the command line:

python3 -m grpc_tools.protoc -I . --python_out=. --grpc_python_out=. ./*.proto
Copy the code

After successful execution, two files helloWorld_pb2. py and helloWorld_PB2_grpc. py are generated in the directory.

This process may also report an error:

ModuleNotFoundError: No module named 'grpc_tools'
Copy the code

Don’t panic, it is lack of package, just install:

pip3 install grpcio
pip3 install grpcio-tools
Copy the code

A final look at the Python client code:

import grpc

import helloworld_pb2
import helloworld_pb2_grpc


def main() :
    channel = grpc.insecure_channel("127.0.0.1:50051")
    stub = helloworld_pb2_grpc.GreeterStub(channel)
    response = stub.SayHello(helloworld_pb2.HelloRequest(name="zhangsan"))
    print(response.message)


if __name__ == '__main__':
    main()
Copy the code

In this way, you can request the server services started by Go through the Python client.

conclusion

This article through the actual combat point of view, directly with the code to explain some applications of gRPC.

Content includes simple gRPC service, stream processing mode, validator, Token authentication and certificate authentication.

There are other things worth exploring, such as timeout control, REST interfaces, and load balancing. There will be time to work on the rest of this.

The code in this article has been tested and verified, can be directly executed, and has been uploaded to GitHub, partners can look at the source code, a reference to the article to learn.

Source code address:

Github.com/yongxinz/go…
Github.com/yongxinz/go…

Recommended reading:

Run the grpcurl command to access the gRPC service
Three useful Go development tools are recommended
Docker log pit
Cannot assign requested address

Reference article:

Github.com/mwitkow/go-…
Github.com/Bingjian-Zh…
Gaodongfei.com/archives/st…
Liaoph.com/openssl-san…
www.cnblogs.com/jackluo/p/1…

introduce

An introduction to

Flow way

The validator

Token authentication

One-way certificate authentication

Generate a certificate

GRPC code

Two-way certificate authentication

A certificate with SAN is generated

GRPC code

The Python client

conclusion

Related Posts

Spring Boot Starter Swagger

Six cross-domain solutions, all told to you today

Linux Commands