Gateway server

The so-called gateway, in fact, is to maintain the connection of the player client and forward the game request sent by the player to the specific back-end service server, with the following function points:

  • Long-term operation, must have high stability and performance
  • Open to the public, that is, the client needs to know the IP address and port number of the gateway before connecting to it
  • Multi-protocol support
  • Unified portal. There may be many back-end services in the architecture. If there is no unified portal, the client needs to know the IP and port of each back-end service
  • Request forwarding: Because of unified entry, the gateway must be able to forward the client’s request to the exact service, and routes must be provided
  • No sense of update, because the player is connected to the gateway server, as long as the connection is constantly; Updating the backend server is either insensitive or insensitive to the player (depending on how it is implemented)
  • Business agnostic (there may inevitably be a little business for game server gateways)

The Micro framework itself already implements an API gateway for HTTP requests, as you can see in a previous blog post

Card games using microservices reconstruction notes (ii) : Micro framework introduction: Micro Toolkit

But for the game server, is generally the need for long links, we need to achieve their own

Connection protocol

The gateway itself should support multiple protocols. Here I take Websocket as an example to illustrate the idea in the process of reconstruction. Other protocols are similar. Melody is recommended. Based on the WebSocket library, the syntax is very simple and a few lines of code can implement the Websocket server. Our game requires a Websocket gateway because the client does not support HTTP2 and cannot connect directly to the GRPC server

package main

import (
	"github.com/micro/go-web"
	"gopkg.in/olahol/melody.v1"
	"log"
	"net/http"
)

func main() {
	// New web service
	service := web.NewService(web.Name("go.micro.api.gateway"))

	// parse command line
	service.Init()

	// new melody
	m := melody.New()

	// Handle websocket connection
	service.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		_ = m.HandleRequest(w, r)
	})
	
	// handle connection with new session
	m.HandleConnect(func(session *melody.Session) {
		
	})
	
	// handle disconnection
	m.HandleDisconnect(func(session *melody.Session) {
		
	})
	
	// handle message
	m.HandleMessage(func(session *melody.Session, bytes []byte) {
		
	})

	// run service
	iferr := service.Run(); err ! = nil { log.Fatal("Run: ", err)
	}
}
Copy the code

Forward requests

The gateway can receive and send data with a uniform data structure of [] bytes, much like a GRPC stream, so you can use protobuf’s Oneof feature to define requests and responses, see the previous blog post

Note on card game refactoring using microservices (vi) : Protobuf pit crawl

Define gateway.basic.proto to classify messages received/sent by the gateway

message Message { oneof message { Req req = 1; Rsp Rsp = 2; // Notify Notify = 3; // The client push does not require a response. // server push Stream Stream = 5; // Two-way traffic request Ping Ping = 6; // ping Pong pong = 7; // pong } }Copy the code

Req and notify are stateless requests from the client, corresponding to the stateless server at the back end. You only need to implement your own routing rules, for example

message Req { string serviceName = 1; // Service name string method = 2; // Method name bytes args = 3; / / parameter Google. Protobuf. Timestamp Timestamp = 4; // Timestamp... }Copy the code
  • ServiceName serviceName that calls the RPC server
  • Method Method name for calling the RPC server
  • Args call parameter
  • Timestamp request timestamp, used by the client to identify the server response, simulating HTTP requestreq-rsp

The idea is similar to the MICRO Toolkit API gateway (RPC processor), relatively simple, see the previous blog.

Our project uses HTTP for this type of request and does not pass through the gateway, only some basic REQS such as authReq handle session authentication. The main considerations are that HTTP is simple, stateless, and easy to maintain, and that such services do not require high real-time performance.

GRPC stream forward

Game servers are generally stateful, bidirectional, and have high real-time requirements. Req-rsp mode is not suitable, so gateway is needed for forwarding. Each time you add a GRPC back-end server, you only need to add a stream to oneof to extend

message Stream { oneof stream { room.basic.Message roomMessage = 1; // Room server game.basic.Message gameMessage = 2; // Game server mateMessage = 3; // Match server}}Copy the code

In addition, a corresponding routing request needs to be defined to handle the forwarding to which back-end server (different implementations may not be required), which will involve some services, such as

message JoinRoomStreamReq {
    room.basic.RoomType roomType = 1;
    string roomNo = 2;
}
Copy the code

Here, the correct room server is selected based on the room number and room type requested by the client route, and the gateway (maybe even linked to an older version of the room server).

After selecting the correct server, establish the stream bidirectional stream

address := "xxxxxxx"Context m := make(map[string]string) // some metadata streamCtx, CancelFunc := context.withcancel (CTX) cancelFunc := context.withcancel (CTX) Err := xxxClient.stream (metadata.NewContext(streamCtx, m), client.withAddress (address))"stream", stream)
session.Set("cancelFunc"CancelFunc) // Start a goroutine to collect the stream message and forward go func(c context.context, s pb.xxxxxstream) {// Close stream defer on exitfunc() {
	session.Set("stream", nil)
	session.Set("cancelFunc", nil)

	iferr := s.Close(); err ! = nil { //do something with close err
	}
    }()

    for {
	select {
    	case <-c.Done():
    		// do something with ctx cancel
    		return
    
    	default:
    		res, err := s.Recv()
    		iferr ! = nil { //do something with recv err
    			return} // The oneof wrapper can be used to tell the client which stream sent the message... } } }(streamCtx, stream)Copy the code

Forwarding is relatively simple, directly on the code

A request for a stream

message Stream { oneof stream { room.basic.Message roomMessage = 1; // Room server game.basic.Message gameMessage = 2; // Game server mateMessage = 3; // Match server}}Copy the code

Add forward code

s, exits := session.Get("stream")
if! exits {return
}

if stream, ok := s.(pb.xxxxStream); ok {
    err := stream.Send(message)
    iferr ! = nil { log.Println("send err:", err)
    	return}}Copy the code

When you need to close a stream, you simply call cancelFunc

if v, e := session.Get("cancelFunc"); e {
    if c, ok := v.(context.CancelFunc); ok {
    	c()
    }
}
Copy the code

Benefits of using oneOf

With oneof, you can switch cases along the way, adding only one case for each REQ or stream, and the code still looks simple and clean

func HandleMessageBinary(session *melody.Session, bytes []byte) {
    var msg pb.Message

    iferr := proto.Unmarshal(bytes, &msg); err ! = nil { //do something
    	return
    }
    
    defer func() {
	err := recover()
	iferr ! = nil { //do something with panic
	}
    }()
    
    switch x := msg.Message.(type) {
    case *pb.Message_Req:
    	handleReq(session, x.Req)
    
    case *pb.Message_Stream:
    	handleStream(session, x.Stream)
    
    case *pb.Message_Ping:
    	handlePing(session, x.Ping)
    
    default:
    	log.Println("unknown req type")
    }
}

func handleStream(session *melody.Session, message *pb.Stream) {
    switch x := message.Stream.(type) {
    case *pb.Stream_RoomMessage:
        handleRoomStream(session, x.RoomMessage)
    
    case *pb.Stream_GameMessage:
        handleGameStream(session, x.GameMessage)
    
    case *pb.Stream_MateMessage:
        handleMateStream(session, x.MateMessage)
    }
}
Copy the code

Hot update

It is very important to keep updating the game, my ideas will be introduced in the blog after, you can pay attention to a wave of hey hey

The pit!

  • Such a gateway, seems to be no problem, yet run on a period of time to usepprofThe observation is thatgoroutineAnd memory are slowly growing, which is existencegoroutine leak!The reason is that micro source code in the packaging of GRPC, not perfect to close the stream, only receivedio.EOFThe CONN connection to the GRPC is closed only when the error”
func (g *grpcStream) Recv(msg interface{}) (err error) {
    defer g.setError(err)
    iferr = g.stream.RecvMsg(msg); err ! = nil {if err == io.EOF {
            // #202 - inconsistent gRPC stream behavior
            // the only way to tell if the stream is done is when we get a EOF on the Recv
            // here we should close the underlying gRPC ClientConn
            closeErr := g.conn.Close()
            ifcloseErr ! = nil { err = closeErr } } }return
}
Copy the code

And there’s a TODO

// Close the gRPC send stream
// #202 - inconsistent gRPC stream behavior
// The underlying gRPC stream should not be closed here since the
// stream should still be able to receive after this function call
// TODO: should the conn be closed in another way?
func (g *grpcStream) Close() error {
    return g.stream.CloseSend()
}
Copy the code

The solution is also simple: fork a source code and change it to close the stream and shut down conn at the same time (our business is possible because the GRPC Stream client and server both implement to close the stream after receiving an ERR), or wait for the author to update and use a more scientific way to shut down

  • The session in the melodygetandsetMap read/write contention will occur during data, and panic will occurissue, the solution is also relatively simple

Study together

I have learned golang, Micro, K8S, GRPC, Protobuf and other knowledge for a short time, if there is any misunderstanding, welcome criticism and correction, you can add my wechat to discuss learning