“This is the 26th day of my participation in the First Challenge 2022. For details: First Challenge 2022”

1. Why provide support for other protocols

After completing multiple gRPC services, there is always a need to provide HTTP interfaces or support for multiple protocols for a single RPC method, but why is this the case?

This is mainly due to the following possibilities: first, heartbeat, monitoring interface, etc., and second, business scenarios change. The same RPC method needs to provide its services for multiple protocol business scenarios, but it is impossible to reproduce an identical one, so multi-protocol support is very urgent.

In addition, gRPC protocol is HTTP/2 protocol in essence. If the service wants to accommodate two protocols on the same port, special processing is required. So in the next section, we’ll talk about support for the most frequently touched HTTP/1.1 interface, and the various scenarios and considerations that have been extended to that.

The following will be divided into three major cases for practical operation explanation. Although the codes of each case are relatively independent, they are related to each other in knowledge points.

2 Another port listens for HTTP

So the first, the most basic requirement: To implement gRPC (HTTP/2) and HTTP/1.1 support, allow to be divided into two ports to proceed, we open the project root directory main.go file, modify its startup logic, and respectively implement gRPC and HTTP/1.1 running logic, write the following code:

var grpcPort string var httpPort string func init() { flag.StringVar(&grpcPort, "grpc_port", "8001", "GRPC startup port number ") flag.stringvar (&httpPort, "http_port", "9001", "HTTP startup port number ") flag.parse ()}Copy the code

First, we adjust the original gRPC service startup port to HTTP/1.1 and gRPC port number reading, then we implement the specific service startup logic, continue to write the following code:

func RunHttpServer(port string) error { serveMux := http.NewServeMux() serveMux.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte(`pong`)) }) return http.ListenAndServe(":"+port, serveMux) } func RunGrpcServer(port string) error { s := grpc.NewServer() pb.RegisterTagServiceServer(s, server.NewTagServer()) reflection.Register(s) lis, err := net.Listen("tcp", ":"+port) if err ! = nil { return err } return s.Serve(lis) }Copy the code

In the code above, we have divided the service startup into two methods: the RunHttpServer method for HTTP, which initializes a new HTTP multiplexer and adds a /ping route and its Handler for basic heartbeat detection. In addition, gRPC is the same as before, keeping the relevant logic of gRPC Server, only reencapsulation as RunGrpcServer method.

Next we write the startup logic and continue with the following code:

func main() { errs := make(chan error) go func() { err := RunHttpServer(httpPort) if err ! = nil { errs <- err } }() go func() { err := RunGrpcServer(grpcPort) if err ! = nil { errs <- err } }() select { case err := <-errs: log.Fatalf("Run Server err: %v", err) } }Copy the code

In the above code, we declare a chan to receive err information from the Goroutine. Then we call the RunHttpServer and RunGrpcServer methods respectively in the Goroutine. The reason why you put it in a Goroutine is because actually listening to the HTTP EndPoint and gRPC EndPoint is a blocking behavior.

If the RunHttpServer or RunGrpcServer method starts or runs incorrectly, err is written to chan, so we only need to check it with SELECT.

Next we verify that the output is as expected, using the following command:

$grpcurl - plaintext localhost: 8001 proto. TagService. GetTagList $curl http://127.0.0.1:8002/pingCopy the code

The first command should output the result set for the tag list, and the second command should output the Pong string, completing the function of listening on gRPC Server and HTTP Server on different ports in one application.

3 Listen on the same port number

In the previous section, we completed the requirement for dual ports to listen to different traffic. However, in some use or deployment scenarios, it can be troublesome and two ports need to be taken care of. At this point, there will be a requirement for multiple protocols to be compatible on one port.

3.1 Introduction and Installation

In Go, we can implement multi-protocol support using the third-party open source library CMUx, which multiplexes connections based on payloads (matching the first few bytes of a connection to differentiate the current connection type). GRPC, SSH, HTTPS, HTTP, Go RPC, and almost all other protocols can be provided on the same TCP Listener, which is a relatively generic solution.

Note, however, that a connection can be gRPC or HTTP, but not both. That is, we assume that the client connection is for gRPC or HTTP, but not both on the same connection.

Next we execute the following installation command in the project root directory:

$go get -u github.com/soheilhy/[email protected]Copy the code

3.2 Multi-protocol support

To start coding, open the startup file main.go in the root directory of the project and change it to the following code:

Var port string func init() {flag.stringVar (&port, "port", "8003", "start port") flag.parse ()}Copy the code

First of all, we adjust the default port number of the startup port. Since it is on the same port, we adjust back a port variable. Next, we write the implementation logic of the specific Listener, which is essentially the same as the previous section, but re-split the logic of TCP, gRPC and HTTP. To facilitate the use of the connection multiplexer, change to the following code:

func RunTCPServer(port string) (net.Listener, error) {
return net.Listen("tcp", ":"+port)
}

func RunGrpcServer() *grpc.Server {
s := grpc.NewServer()
pb.RegisterTagServiceServer(s, server.NewTagServer())
reflection.Register(s)

return s
}

func RunHttpServer(port string) *http.Server {
serveMux := http.NewServeMux()
serveMux.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`pong`))
})

return &http.Server{
Addr:    ":" + port,
Handler: serveMux,
}
}
Copy the code

Next we modify the startup logic in main as follows:

func main() { l, err := RunTCPServer(port) if err ! = nil { log.Fatalf("Run TCP Server err: %v", err) } m := cmux.New(l) grpcL := m.MatchWithWriters(cmux.HTTP2MatchHeaderFieldPrefixSendSettings("content-type", "application/grpc")) httpL := m.Match(cmux.HTTP1Fast()) grpcS := RunGrpcServer() httpS := RunHttpServer(port) go grpcS.Serve(grpcL) go httpS.Serve(httpL) err = m.Serve() if err ! = nil { log.Fatalf("Run Serve err: %v", err) } }Copy the code

GRPC (HTTP/2) and HTTP/1.1 are all TCP based on the network layer. The second point is the Application/GRPC identifier of the Content-Type. In section 3.4.4.1.3, we have analyzed that GRPC also has a specific flag bit, namely application/ GRPC. Cmux is also based on this identifier to perform shunt.

You need to restart the service to verify that the grpcurl tool and the HTTP/1.1 interface with curl are responding correctly.

4 The same port provides dual traffic with the same method

Although you have made a lot of attempts, the demander still wants a more direct approach. The demander wants to implement an RPC method for gRPC (HTTP/2) and HTTP/1.1 traffic support in the application, rather than simply creating HTTP handlers as in the previous chapters. After your in-depth communication, in fact, they want to use gRPC as internal API communication but also want to provide external RESTful, but do not want to build a conversion gateway, write two sets of too tedious does not meet… .

Also, some internal developers have reported that they usually just want to call the interface for basic validation during local/development debugging… You don’t want to have to call the grpcurl tool every time, check the list, and fill in the entry parameter, compared to using a tool like Postman (with a Web UI).

Is there any other way to do this? In fact, there is a grPC-Gateway in the open source community that can do this, as shown below (from the official image) :

Grpc-gateway is a plug-in for Protoc that reads the protobuf service definition and generates a reverse proxy server that converts RESTful JSON apis into GRPC, It is generated primarily from Google.api.http in protobuf’s service definition.

To put it simply, GRPC-gateway can convert RESTful requests into GRPC requests, realizing the requirement that the same RPC method provides dual traffic support of GRPC protocol and HTTP/1.1.

4.1 GrPC-Gateway Introduction and Installation

Protoc -gen-grpc-gateway protoc-gen-grpc-gateway protoc-gen-grpc-gateway

$go get -u github.com/grpc-ecosystem/grpc-gateway/[email protected]Copy the code

Move the executable file of the compiled and installed Protoc Plugin from $GOPATH to the appropriate bin directory, for example:

$ mv $GOPATH/bin/protoc-gen-grpc-gateway /usr/local/go/bin/
Copy the code

The command operation here is not absolutely necessary, the main purpose is to move the binary protoc-gen-grpc-gateway to the bin directory so that it can be executed, make sure that under $PATH, only this effect is achieved.

4.2 Proto file processing

4.2.1 Proto file modification and compilation

Therefore, for the use of GRPC-gateway, we need to adjust the tag.proto file under the proto command of the project to the following:

syntax = "proto3"; package proto; import "proto/common.proto"; import "google/api/annotations.proto"; service TagService { rpc GetTagList (GetTagListRequest) returns (GetTagListReply) { option (google.api.http) = { get: "/api/v1/tags" }; }}...Copy the code

We increased the Google in the proto file/API/annotations. The introduction of the proto file, and added in the corresponding RPC method for HTTP routing annotations. Next we recompile the proto file and execute the following command in the project root directory:

$ protoc -I/usr/local/include -I. \
       -I$GOPATH/src \
       -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
       --grpc-gateway_out=logtostderr=true:. \
       ./proto/*.proto
Copy the code

After the command is executed, the tag.pb.gw.go file is generated, which is the. Pb. go and. Pb.gw. go files used in the proto directory.

We use a new protoc command option with the -i parameter of the format: -ipath, –proto_path=PATH, which specifies the search directory for import (that is, the import command in the Proto file). Multiple directories can be specified. If this command is not specified, the current working directory is default.

For example, if the protoc command format is Mfoo/bar.proto=quux/shme, the specified package name will be replaced by the required name when proTO is generated or compiled. Foo /bar.proto is compiled to a package named quux/shme), more options for executing protoc –help to view help documentation.

Annotations. What is Annotations

We just in GRPC – gateway proto files are generated using the Google/API/annotations. The proto file, in fact, it is the product of googleapis, we have described in the previous chapter.

In addition, you can combine the protoc generation command of grpc-gateway to see that it also puts googleapis in the third_party directory of grpc-gateway repository, so annotations. Proto Grpc-gateway is used under this way to ensure compatibility and stability (version control).

Annotations. Proto file (annotations. Proto file)

syntax = "proto3"; package google.api; import "google/api/http.proto"; import "google/protobuf/descriptor.proto"; . extend google.protobuf.MethodOptions { HttpRule http = 72295728; }Copy the code

Take a look at a portion of the http.proto file used by the core as follows:

message HttpRule {
  string selector = 1;
  oneof pattern {
    string get = 2;
    string put = 3;
    string post = 4;
    string delete = 5;
    string patch = 6;
    CustomHttpPattern custom = 8;
  }

  string body = 7;
  string response_body = 12;
  repeated HttpRule additional_bindings = 11;
}
Copy the code

In general, it provides support for HTTP conversions, defines HTTP options extended by Protobuf, and can be used in Proto files to define CONFIGURATIONS related to HTTP for API services. And you can specify that each RPC method maps to one or more HTTP REST API methods.

So if you hadn’t imported the annotations. Proto file and filled in the corresponding HTTP Option in the proto file, you would have executed the build command, and you wouldn’t have gotten an error, but you wouldn’t have generated anything.

4.3 Service logic implementation

Next, we started to implement the grPC-gateway based access support for GRPC (HTTP/2) and HTTP/1.1 dual traffic with the same RPC method under the same port. We opened the startup file main.go under the root directory of the project and modified it into the following code:

Var port string func init() {flag.stringVar (&port, "port", "8004", "start port") flag.parse ()}Copy the code

4.3.1 Diversion of different protocols

We adjust the service startup port number for this case and continue to write the following code in main.go:

func grpcHandlerFunc(grpcServer *grpc.Server, otherHandler http.Handler) http.Handler {
return h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
grpcServer.ServeHTTP(w, r)
} else {
otherHandler.ServeHTTP(w, r)
}
}), &http2.Server{})
}
Copy the code

This is a very core method, the important triage and setup has two parts, as follows:

• Traffic differentiation between gRPC and HTTP/1.1:
  • Determine ProtoMajor. This field represents the version number of the client request. The client always uses HTTP/1.1 or HTTP/2.
  • Header Determination of content-Type: GRPC flag bitapplication/grpcSure.
• Non-encryption mode Settings for gRPC services: focus on the “H2C” identifier in the code, the “H2C” identifier allows HTTP/2 to run over plaintext TCP, this identifier is used for HTTP/1.1 upgrade header fields and identifies HTTP/2 over TCP, and the official standard librarygolang.org/x/net/http2/h2cHTTP/2 implementation of unencrypted mode, we can directly use.

In terms of the overall method logic, we can see that the key is to call the h2C.newhandler method for special processing. The H2C.newhandler method returns an HTTP. handler, which internally intercepts all H2C traffic. Then according to different request traffic types, it is hijacked and redirected to the corresponding Hander for processing, and finally achieves both HTTP/1.1 and HTTP/2 functions on the same port.

4.3.2 Server implementation

After we have completed the traffic distribution and processing of the different protocols, we need to implement the specific logic of its Server and continue to write the following code in the main.go file:

import ( "github.com/grpc-ecosystem/grpc-gateway/runtime" ... ) func RunServer(port string) error { httpMux := runHttpServer() grpcS := runGrpcServer() gatewayMux := runGrpcGatewayServer() httpMux.Handle("/", gatewayMux) return http.ListenAndServe(":"+port, grpcHandlerFunc(grpcS, httpMux)) } func runHttpServer() *http.ServeMux { serveMux := http.NewServeMux() serveMux.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte(`pong`)) }) return serveMux } func runGrpcServer() *grpc.Server { s := grpc.NewServer() pb.RegisterTagServiceServer(s, server.NewTagServer()) reflection.Register(s) return s } func runGrpcGatewayServer() *runtime.ServeMux { endpoint := "0.0.0.0:" + port gwmux := Runtime. NewServeMux() DOPts := [] grpc.dialoption {grpc.withInsecure ()} _ = pb.RegisterTagServiceHandlerFromEndpoint(context.Background(), gwmux, endpoint, dopts) return gwmux }Copy the code

In the code above, the main difference from the previous case is the registration associated with grPC-Gateway in the RunServer method, Core is to call the RegisterTagServiceHandlerFromEndpoint method to register TagServiceHandler event, its internal will automatically transform and dial to gRPC Endpoint, and close the connection at the end of the context.

In addition, when registering the TagServiceHandler event, we specified the Server as unencrypted mode by setting grpc.WithInsecure in grpc.DialOption, otherwise the program will have problems when running. This is because when gRPC Server/Client is started and invoked, it must be clear whether it is encrypted or not.

4.3.3 Operation and Verification

Next we write the main startup method and call the RunServer method as follows:

func main() { err := RunServer(port) if err ! = nil { log.Fatalf("Run Serve err: %v", err) } }Copy the code

After the restart of the service, we verified the RPC method as follows:

$$curl curl http://127.0.0.1:8004/ping http://127.0.0.1:8004/api/v1/tags $grpcurl - plaintext localhost: 8004 proto.TagService.GetTagListCopy the code

In the correct case, response data is returned, corresponding to the heartbeat detection, HTTP/1.1 of the RPC method, and gRPC (HTTP/2) of the RPC method, respectively.

4.3.4 Custom Error

And after we did that, we thought, We can in the gRPC by reference to gRPC google.golang.org/grpc/status method can – status, gRPC – message and gRPC – details detailed customization (our errcode Grpc-gateway, as a proxy, will display error messages as follows:

{" error ":" failed to get the label list ", "code" : 2, the "message" : "failed to get the label list", "details" : [{" @ type ": "Type.googleapis.com/proto.Error", "code" : 20010001, the "message" : "failed to get the label list"}}]Copy the code

As a result, this is really a complete conversion of GRPC error, too direct, this is obviously not conducive to browser side reading, the client will not know what to call the standard.

In fact, grpc-status actually corresponds to our HTTP status code, and the business error code corresponds to the message body required by the client. Therefore, we need to customize the grpc-gateway error and continue to write the following code in the main.go file:

type httpError struct { Code int32 `json:"code,omitempty"` Message string `json:"message,omitempty"` } func grpcGatewayError(ctx context.Context, _ *runtime.ServeMux, marshaler runtime.Marshaler, w http.ResponseWriter, _ *http.Request, err error) { s, ok := status.FromError(err) if ! ok { s = status.New(codes.Unknown, err.Error()) } httpError := httpError{Code: int32(s.Code()), Message: s.Message()} details := s.Details() for _, detail := range details { if v, ok := detail.(*pb.Error); ok { httpError.Code = v.Code httpError.Message = v.Message } } resp, _ := json.Marshal(httpError) w.Header().Set("Content-type", marshaler.ContentType()) w.WriteHeader(runtime.HTTPStatusFromCode(s.Code())) _, _ = w.Write(resp) }Copy the code

In the above code, we processed the returned gRPC error twice, converting it to the corresponding HTTP status code and the corresponding error body to ensure that the client can interact according to the standards of the RESTful API.

The next step is to register the error handling methods customized for grPC-gateway as follows:

Func RunServer(port string) error {httpMux := runHttpServer() grpcS := runGrpcServer() endpoint := "0.0.0.0:" + port runtime.HTTPError = grpcGatewayError gwmux := runtime.NewServeMux() ... }...Copy the code

Restart the service and check the result.

$curl - http://127.0.0.1:8004/api/v1/tags < HTTP / 1.1-500 Internal Server Error "content-type: Application /json {"code":20010001,"message":" failed to get tag list "}Copy the code

You can see that the output HTTP status code and message body are correct.

4.4 How is this implemented

We have already mentioned that gRPC (HTTP/2) and HTTP/1.1 are shunt through the content-Type and ProtoMajor flags in the Header, but what is the processing logic after shunt? GRPC to register (RegisterTagServiceServer), gRPC – gateway to also want to register (RegisterTagServiceHandlerFromEndpoint), what’s the use?

We will explore how grPC-gateway is implemented, that for our developers, the most commonly encountered is. Pb.gw. Go registration method, as follows:

func RegisterTagServiceHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) { conn, err := grpc.Dial(endpoint, opts...) if err ! = nil { return err } defer func() { if err ! = nil { if cerr := conn.Close(); cerr ! = nil { grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) } return } go func() { <-ctx.Done() if cerr := conn.Close(); cerr ! = nil { grpclog.Infof("Failed to close conn to %s: %v", endpoint, cerr) } }() }() return RegisterTagServiceHandler(ctx, mux, conn) }Copy the code

Actually when call such RegisterXXXXHandlerFromEndpoint registration method, mainly to create gRPC connection and controls, it has been called gRPC internally. Dial to dial-up connection gRPC Server, And holds a Conn for subsequent HTTP/1/1 calls to forward. In addition, the processing of closing the connection is relatively robust. We put it into defer for closing, or control the closing time of the connection according to the context.

Next is that the exact RegisterTagServiceHandler internal registration method, the call is actually the following methods:

func RegisterTagServiceHandlerClient(ctx context.Context, mux *runtime.ServeMux, client TagServiceClient) error {
  
mux.Handle("GET", pattern_TagService_GetTagList_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {

ctx, cancel := context.WithCancel(req.Context())
defer cancel()
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, _ := runtime.AnnotateContext(ctx, mux, req)
resp, md, _ := request_TagService_GetTagList_0(rctx, inboundMarshaler, client, req, pathParams)
ctx = runtime.NewServerMetadataContext(ctx, md)

forward_TagService_GetTagList_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})

return nil
}
Copy the code

This method contains the entire HTTP/1.1 conversion to gRPC pre-operation, including at least the following four processing:

• Registration method: Registers the HTTP Endpoint predefined by the current RPC method (based on the information contained in the.pb.gw.go generated by the proto file) with the HTTP multiplexer passed in externally, This is the GMUx returned by the runtime.NewServeMux method in our program.

• Timeout: controlled based on the context passed in externally.

• Request/response data: Default serialization based on the MIME type passed in, for example: Application/jSONPB, Application/JSON. In addition to its implementation is a Marshaler, namely we can by calling the GRPC – the runtime of the gateway. WithMarshalerOption method to register the MIME type we need and its corresponding Marshaler.

• Metadata: gRPC Metadata is converted into context for easy use.

5 Other Schemes

Are there any other external solutions besides implementing application proxies such as GRPC-gateway in applications?

External solutions, or external components, are commonly referred to as gateways, and currently Envoy offers GRPC-JSON transcoder support for RESTful JSON API clients to send requests to an Envoy via HTTP/1.1 and proxy them to the gRPC service. APISIX also offers similar functionality and is currently being incubated by Apache, which is also worth watching.

In fact, there are not many options, and none of them are provided as a single technical solution, but as one of the functions in the gateway, if you are interested, you can learn more about it.