Golang Microservices Tutorial (part 1)

Link to the original article: Ewan Valentine. IO. Translation authorized by author Ewan Valentine.

This section dabble in the use of gRPC, more can refer to: gRPC Client and Server data interaction of the four modes

preface

An overview of the series

The Golang Microservices Tutorial is a 10-part series that summarizes the complete process of microservices development, testing, and deployment.

This section introduces the basic concepts and terminology of microservices before creating a concise version of our first microservice, the consignment service. In sections 2 through 10, we will create the following microservices:

Consignment service (consignment service)
Inventory-service
User-service (user service)
Authentication-service authentication service
Role-service (role service)
Vessel -service (vessel-service)

The complete technology stack is as follows:

Golang, gRPC, Go-Micro // Development language and ITS RPC framework Google Cloud, MongoDB // Cloud platform and data storage Docker, Kubernetes, Terrafrom // Containerization and cluster architecture NATS, CircleCI // Messaging system and continuous integrationCopy the code

Code warehouse

By EwanValentine/ Shippy, and by wuYin/ Shippy

Each chapter corresponds to a branch of the repository, such as the code in part1 of this article in feature/part1

The development environment

The author’s development environment for macOS, this article uses the make tool to compile efficiently, Windows users need to install manually

$go env GOARCH="amd64" # macOS env GOOS=" Darwin "# GOROOT="/usr/local/go"Copy the code

To prepare

To master the basic Syntax of Golang, I recommend reading Go Web Programming by Xie Da.

Install gRPC/Protobuf

Go get the -u google.golang.org/grpc # gRPC framework go get - u github.com/golang/protobuf/protoc-gen-go # installation go protobuf version The compilerCopy the code

Micro service

What project are we going to write?

We need to build a port cargo management platform. The development of this project is based on the architecture of microservices, which is simple and universal in concept. Without further ado, let’s begin the journey of microservices.

What is a microservice?

In traditional software development, the entire application’s code is organized into a single code base, which typically takes the form of the following split code:

Split by characteristics: for example, the MVC pattern
Split by function: In a larger project, code may be encapsulated in packages that handle different businesses, and the packages may be split inside

No matter how you split it, the end result is that both code will be developed and managed in one library. See: Google’s single code base management

Microservices are an extension of the second approach, where code is broken up into packages by function, each of which is a single code base that runs independently. The differences are as follows:

What are the advantages of microservices?

Reduce complexity

Breaking the code of an entire application down into small, independent microservice codebase by function is reminiscent of the Unix philosophy: Do One Thing and Do It Well. In the traditional application of a single code base, the modules are tightly coupled and the boundary is blurred. With the continuous iteration of the product, the development and maintenance of the code will become more complex, and the potential bugs and vulnerabilities will also become more and more.

Improve scalability

In project development, there may be part of the code will frequently be used in multiple modules, this kind of reuse modules are often pulled out as a public library use, such as authentication module, when it wants to extend the functionality (add login message authentication code, etc.), the size of the single code base only grow, still need to redeploy the entire application. In a microservices architecture, validation modules can stand alone as a single service that can be run, tested, and deployed independently.

Following the concept of splitting code with microservices greatly reduces the coupling between modules and makes it much easier to scale horizontally, which is well suited to the high performance, high availability, and distributed development environment of cloud computing.

Nginx has a series of articles exploring many of the concepts of microservices that can be read here

The benefits of using Golang?

Microservices are an architectural concept rather than a specific framework project. Many programming languages can be implemented, but some have inherent advantages for microservices development, and Golang is one of them

Golang itself is lightweight, highly efficient, and has native support for concurrent programming to make better use of multi-core processors. The built-in NET standard library supports web development as well. Please refer to Xie Da’s essay: The Advantages of Go language

In addition, the Golang community has a great open source microservice framework, Go-Mirco, which we’ll use in the next section.

Protobuf and gRPC

In a traditional application with a single code base, modules can call functions directly from each other. But in a microservice architecture, communication is a big problem because the code base for each service runs independently and cannot be called directly. There are two solutions:

JSON or XML protocol API

Microservices can communicate with each other using HTTP-based JSON or XML protocols: Before the communication between service A and service B, SERVICE A must encode the data to be passed into JSON/XML format, and then pass the data to B in the form of string. After receiving the data, service B needs to decode before using it in the code:

Advantages: Data is easy to read and easy to use. It is a mandatory protocol for interacting with browsers
Disadvantages: In the case of large amount of data, the overhead of Encode and decode will become large, and the extra field information will lead to higher transmission cost

API of RPC protocol

The following JSON data uses metadata such as description and weight to describe the meaning of the data itself. This metadata is used in the Browser/Server architecture to make it easy for the Browser to parse the data:

{
  "description": "This is a test consignment",
  "weight": 550,
  "containers": [
    {
      "customer_id": "cust001",
      "user_id": "user001",
      "origin": "Manchester, United Kingdom"
    }
  ],
  "vessel_id": "vessel001"
}
Copy the code

However, when two microservices communicate with each other, if they agree on the format of data transfer, they can communicate directly using binary data streams, eliminating the need for bulky and redundant metadata.

GRPC profile

GRPC is Google’s open source lightweight RPC communication framework. The communication protocol is based on binary data stream, which makes gRPC have excellent performance.

GRPC supports THE HTTP 2.0 protocol and uses binary frames for data transmission. It can also establish a continuous two-way data flow between the two parties. Refer to: Introduction to Google HTTP/2

Protobuf serves as the communication protocol

Two microservices communicate via HTTP 2.0-based binary data frames, so how do you agree on the format of the binary data? The answer is to use gRPC’s built-in Protobuf protocol, whose DSL syntax clearly defines data structures for communication between services. GRPC Go: Beyond the basics

Consignment service microservice development

With the necessary conceptual explanation above, let’s now begin to develop our first microservice: consignment Service

The project structure

Assuming this project is called Shippy, you need:

in$GOPATHCreate a new shippy project directory under the SRC directory of
Create a new file in the project directoryconsignment-service/proto/consignment/consignment.proto

For the sake of teaching, I will put the code for all the microservices in the project in a unified shippy directory. This project structure is called “mono-repo,” or the reader can split the microservices into individual projects as “multi-repo.” More on REPO style battles: MONO VS MULTI

Your project structure should now look like this:

$GOPATH/ SRC ├ ─ shippy ├ ── consignment service ├ ── proto ├ ── consignment. ProtoCopy the code

The development process

Defines the protobuf protocol file

// shipper/consignment-service/proto/consignment/consignment.proto syntax = "proto3"; package go.micro.srv.consignment; // ShippingService {// Consignment cargo RPC CreateConsignment (Consignment) returns (Response) {}} // Consignment cargo message Consignment { string id = 1; String description = 2; Int32 weight = 3; // Containers = 4; String vessel_id = 5; Message Container {string id = 1; // Container id string customer_id = 2; String origin = 3; String user_id = 4; Message Response {bool created = 1; // Successful Consignment Consignment = 2; // New consignment of goods}Copy the code

Protobuf doc

Generate protocol code

The Protoc compiler uses the GRPC plug-in to compile.proto files

In order to avoid repeated execution of compilation and running commands on the terminal, this project uses the make tool to create the consignment service/Makefile

build: # protoc: Nothing to be done for build Not four or eight Spaces protoc - i. - go_out = plugins = GRPC: $(GOPATH)/SRC/shippy/consignment - service proto/consignment/consignment. ProtoCopy the code

Executing make build generates consignment.pb.go in the proto/consignment directory

The mapping between consignment. Proto and consignment. Pb. go

Service: defines the function ShippingService exposes to the outside world: CreateConsignment, which is processed by the GRPC plug-in of the Protobuf compiler to generate an interface

Type ShippingServiceClient interface {// Consignment Consignment interface CreateConsignment(CTX context.Context, in *Consignment, opts... grpc.CallOption) (*Response, error) }Copy the code

Message: Defines the data format for communication, which is processed by the Protobuf compiler to generate the struct

type Consignment struct { Id string `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"` Description string `protobuf:"bytes,2,opt,name=description" json:"description,omitempty"` Weight int32 `protobuf:"varint,3,opt,name=weight"  json:"weight,omitempty"` Containers []*Container `protobuf:"bytes,4,rep,name=containers" json:"containers,omitempty"` / /... }Copy the code

Implementing the server

The server needs to implement the ShippingServiceClient interface and create the consignment service/main.go

Package the main import (/ / guide such as protoc automatically generated package pb "shippy/consignment service/proto/consignment" "context" ".net "" log" "google.golang.org/grpc") const (PORT = ":50051") // // type IRepository interface {Create(consignment *pb.Consignment) (*pb.Consignment, error) // Store new cargo} // // our warehouse for multiple cargo, // type Repository struct {consignments []* pd.consignment} func (repo *Repository) Create(consignment *pb.Consignment) (*pb.Consignment, error) { repo.consignments = append(repo.consignments, consignment) return consignment, Nil} func (repo *Repository) GetAll() []*pb.Consignment {return repo. Consignments} // // {repo Repository} // // service Implements the ShippingServiceServer interface in consignment. Pb. go // makes the service the gRPC server // // Consignment new cargo func (s *service) CreateConsignment(CTX context. context, req * pb.consignment) (* pb.response, Error) {// Receive the consignment, err := s.repo.create (req) if err! = nil { return nil, err } resp := &pb.Response{Created: true, Consignment: consignment} return resp, nil } func main() { listener, err := net.Listen("tcp", PORT) if err ! = nil { log.Fatalf("failed to listen: %v", err) } log.Printf("listen on: %s\n", PORT) server := grpc.newserver () repo := Repository{} // register the microservice to the rRPC server // ShippingServiceServer binding pb. RegisterShippingServiceServer (server, & service {'}) if err: = server. Serve (the listener); err ! = nil { log.Fatalf("failed to serve: %v", err) } }Copy the code

The above code implements the methods required for the consignment service microservice and sets up a gRPC server to listen on port 50051. If you run go run main.go at this point, you will successfully start the server:

Implementing the client

The information of the goods we are going to ship is placed in consignment consignment-cli/consignment. Json:

{
  "description": "This is a test consignment",
  "weight": 550,
  "containers": [
    {
      "customer_id": "cust001",
      "user_id": "user001",
      "origin": "Manchester, United Kingdom"
    }
  ],
  "vessel_id": "vessel001"
}
Copy the code

The client reads the JSON file and ships the goods. Create a new file in the project directory: consingment-cli/cli.go

package main import ( pb "shippy/consignment-service/proto/consignment" "io/ioutil" "encoding/json" "errors" "google.golang.org/grpc" "log" "os" "context" ) const ( ADDRESS = "localhost:50051" DEFAULT_INFO_FILE = Func parseFile(fileName string) (*pb. consignment, error) {data, err := ioutil.ReadFile(fileName) if err ! = nil { return nil, err } var consignment *pb.Consignment err = json.Unmarshal(data, &consignment) if err ! = nil { return nil, errors.New("consignment.json file content error") } return consignment, Nil} func main() {// connect to the gRPC server conn, err := grpc.dial (ADDRESS, grpc.withinsecure ()) if err! = nil { log.Fatalf("connect error: %v", Err)} defer conn. Close () / / initializes the gRPC client client: = pb. NewShippingServiceClient (conn) / / specified on the command line new cargo information infoFile json file If len(os.args) > 1 {infoFile = os.args [1]} err := parseFile(infoFile) if err ! = nil { log.Fatalf("parse info file error: %v", err)} // call RPC // store the goods in our own warehouse resp, err := client.createconsignment (context.background (), consignment) if err ! = nil {log.fatalf ("create consignment error: %v", err)} log.printf ("created: %t", resp.created)}Copy the code

Run go run main.go and then run go run cli.go:

We can add a new RPC to view all consignments and add a GetConsignments method, so that we can see all the consignment that exists:

// shipper/consignment-service/proto/consignment/consignment.proto syntax = "proto3"; package go.micro.srv.consignment; // ShippingService {// Consignment cargo RPC CreateConsignment (Consignment) returns (Response) {} // View the Consignment cargo information RPC GetConsignments (GetRequest) returns (Response) {}} message Consignment {string ID = 1; String description = 2; Int32 weight = 3; // Containers = 4; String vessel_id = 5; Message Container {string id = 1; // Container id string customer_id = 2; String origin = 3; String user_id = 4; Message Response {bool created = 1; // Successful Consignment Consignment = 2; // Consignment consignments = 3; // Consignment consignments = 3; // If the client wants to request data from the server, it must have a request format, even if it is empty message GetRequest {}Copy the code

Now run make Build to get the newly compiled microservices interface. If you run go run main.go, you will get an error message like this:

If you’re familiar with Go, you know you’re forgetting the methods needed to implement an interface. Let’s update the consignment service/main.go:

package main import ( pb "shippy/consignment-service/proto/consignment" "context" "net" "log" "google.golang.org/grpc" ) Const (PORT = ":50051") // // // type IRepository interface {Create(consignment *pb. consignment) (*pb.Consignment, error) // Store new cargo GetAll() []*pb.Consignment // obtain all cargo in the warehouse} // // warehouse where we store multiple batches of cargo, // type Repository struct {consignments []* pd.consignment} func (repo *Repository) Create(consignment *pb.Consignment) (*pb.Consignment, error) { repo.consignments = append(repo.consignments, consignment) return consignment, Nil} func (repo *Repository) GetAll() []*pb.Consignment {return repo. Consignments} // // {repo Repository} // // implement the ShippingServiceServer interface in consignment. Pb. go // make the service as the gRPC server // // to ship new cargo func (s *service) CreateConsignment(ctx context.Context, req *pb.Consignment) (*pb.Response, Error) {// Receive the consignment, err := s.repo.create (req) if err! = nil { return nil, err } resp := &pb.Response{Created: true, Consignment: Consignment} return resp, nil} func (s *service) GetConsignments(CTX context.Context, req *pb.GetRequest) (*pb.Response, error) { allConsignments := s.repo.GetAll() resp := &pb.Response{Consignments: allConsignments} return resp, nil } func main() { listener, err := net.Listen("tcp", PORT) if err ! = nil { log.Fatalf("failed to listen: %v", err) } log.Printf("listen on: %s\n", PORT) server := grpc.NewServer() repo := Repository{} pb.RegisterShippingServiceServer(server, &service{repo}) if err := server.Serve(listener); err ! = nil { log.Fatalf("failed to serve: %v", err) } }Copy the code

If you now use go run main.go, everything should be fine:

Finally let’s update the consignment-cli/cli.go to get the consignment information:

func main() { ... Resp, err = client.getConsignments (context.background (), &pb.getrequest {}) if err! = nil { log.Fatalf("failed to list consignments: %v", err) } for _, c := range resp.Consignments { log.Printf("%+v", c) } }Copy the code

At this point, run go run cli.go again and you should see all the consignment created. Multiple runs will see multiple consignments shipped:

So far, we’ve created a microservice and a client using Protobuf and GRPC.

In the next article, we’ll look at using the Go-Micro framework and creating our second microservice. Meanwhile, in the next article, we’ll look at how to enable Docker to containerize our microservices.