Are you still ripping the app? Try go-Zero's automatic microservice generation

Why is microservices so hard to do?

In order to do a good job in micro services, we need to understand and master a lot of knowledge points, from several dimensions:

Basic functional level
1. Concurrency control & limiting traffic to prevent service from being overwhelmed by sudden traffic
2. Service registration and service discovery to ensure that nodes can be dynamically detected
3. Load balancing: Distributes traffic based on the capacity of nodes
4. Time-out control to avoid idle efforts for time-out requests
5. Fuse breaker design, fast failure, guarantee the recovery ability of failed nodes
Higher functional level
1. Request authentication to ensure that each user can access only his or her own data
2. Link tracing, used to understand the entire system and quickly locate problems with specific requests
3. Logs are used for data collection and problem locating
4. Observability, no measurement, no optimization

For each of these points, we need to explain the principle and implementation of a long time, so it is very difficult for us back-end developers to master these points and implement them into the business system, but we can rely on a framework that has been proven by a large amount of traffic. The Go-Zero microservices framework was born for this purpose.

In addition, we always believe in tools over conventions and documentation. We wanted to reduce the mental burden of developers as much as possible, focus on code that produces business value, and reduce duplication of code, so we developed the GoCTL tool.

Here’s a quick tutorial on how to create a microservice with Go-Zero using a bookstore service. Once you’re done, you’ll see how easy it is to write a microservice!

1. Introduction of bookstore service examples

To keep the tutorial simple, we will use the bookstore service as an example and implement only the functions of adding books and checking prices.

The bookstore service was written to demonstrate the whole process of go-Zero building a complete microservice, with implementation details as simple as possible.

2. Micro-service architecture diagram of the bookstore

3. Overview of code generation for each layer of Goctl

All the functional modules with green background are automatically generated and activated on demand, while the red modules need to be written by themselves, that is, to add dependencies and write business-specific logic. The diagram of each layer is as follows:

API Gateway

model

Let’s Go through the process of building microservices quickly. Let’s Go! 🏃 came ️

4. Preparation

Install etCD, mysql, redis

Install the goctl tool

GO111MODULE=on GOPROXY=https://goproxy.cn/,direct go get -u github.com/tal-tech/go-zero/tools/goctl
Copy the code

Create the working directory Bookstore
Initialize go.mod by executing go mod init bookstore in the bookstore directory

5. Write API Gateway code

Bookstore/API generated by goctl in bookstore/ API:

goctl api -o bookstore.api
Copy the code

Edit bookstore. API and remove the info at the beginning of the file for brevity as follows:

type (
    addReq struct {
        book  string `form:"book"`
        price int64  `form:"price"`
    }

    addResp struct {
        ok bool `json:"ok"`})type (
    checkReq struct {
        book string `form:"book"`
    }

    checkResp struct {
        found bool  `json:"found"`
        price int64 `json:"price"`
    }
)

service bookstore-api {
    @server(
        handler: AddHandler
    )
    get /add(addReq) returns(addResp)

    @server(
        handler: CheckHandler
    )
    get /check(checkReq) returns(checkResp)
}
Copy the code

The usage of type is the same as that of go. Service defines API requests such as GET, POST, head, and delete.

service bookstore-api {This line defines the service name
@serverPart defines the attributes used by the server
handlerDefines the server handler name
get /add(addReq) returns(addResp)Defines routes, request parameters, return parameters, and so on for the GET method

Use goctl to generate API Gateway code

goctl api go -api bookstore.api -dir .
Copy the code

The generated file structure is as follows:

API ├ ─ ─ bookstore. API / / API defines ├ ─ ─ bookstore. Go / / the main entry definition ├ ─ ─ etc │ └ ─ ─ bookstore - API. Yaml / / configuration file └ ─ ─ internal ├ ─ ─ ├─ ├─ ├.go config │ ├─ ├.go config │ ├─ ├.go └ ─ ─ routes. Go / / define routing processing ├ ─ ─ logic │ ├ ─ ─ addlogic. Go / / implementation addlogic │ └ ─ ─ checklogic. Go / / implementation checklogic ├ ─ ─ SVC │ └ ─ ─ Conf // conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/confCopy the code

Start the API Gateway service, listening on port 8888 by default
```
go run bookstore.go -f etc/bookstore-api.yaml
Copy the code
```

Test the API Gateway service

curl -i "http://localhost:8888/check? book=go-zero"Copy the code

Returns the following:

HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 06:46:18 GMT
Content-Length: 25

{"found":false."price":0}
Copy the code

As you can see, our API Gateway does nothing but return a null value, and we will implement the business logic in the RPC service

Can modify the internal/SVC/servicecontext. Go to deliver services rely on (if required)
The implementation logic can modify corresponding files in internal/ Logic
You can use Goctl to generate API call code for various client languages
At this point, you can use goctl to generate client code for concurrent development of the client, support multiple languages, see documentation

6. Write the Add RPC service

Write the add.proto file in the RPC /add directory

Proto file templates can be generated by command

goctl rpc template -o add.proto
Copy the code

The modified file content is as follows:

syntax = "proto3";

package add;

message addReq {
    string book = 1;
    int64 price = 2;
}

message addResp {
    bool ok = 1;
}

service adder {
    rpc add(addReq) returns(addResp);
}
Copy the code

Use goctl to generate RPC code and execute commands in the RPC /add directory

goctl rpc proto -src add.proto
Copy the code

The file structure is as follows:

├─ Add.go // RPC/Add.go // RPC/Add.go // RPC/Add.go // RPC/Add.go // RPC/add.go // RPC/add.go // │ ├─ Adder_Mock. Go // Mock methods │ ├── ├─ ├─ ├─ ├─ ├─ ├.go // ├─ ├─ config │ ├.go │ ├─ ├─ ├─ ├─ ├─ ├─ ├.go │ ├─ server │ ├─ ├.go Don't need to modify the │ └ ─ ─ SVC │ └ ─ ─ servicecontext. Go / / define servicecontext, transfer rely on └ ─ ─ pb └ ─ ─ the add. Pb. GoCopy the code

It can run directly as follows:

  $ go run add.go -f etc/add.yaml
  Starting rpc server at 127.0.0.1:8080...
Copy the code

You can modify configurations such as listening ports in the etc/add.yaml file

7. Write the Check RPC service

Write the check.proto file in the RPC /check directory

Proto file templates can be generated by command

goctl rpc template -o check.proto
Copy the code

The modified file content is as follows:

syntax = "proto3";

package check;

message checkReq {
    string book = 1;
}

message checkResp {
    bool found = 1;
    int64 price = 2;
}

service checker {
    rpc check(checkReq) returns(checkResp);
}
Copy the code

Goctl generates RPC code and executes commands in the RPC /check directory

goctl rpc proto -src check.proto
Copy the code

The file structure is as follows:

RPC/Check ├─ Check. Go // RPC Service Main Function ├─ Check. Proto // RPC interface ├─ Check. │ ├─ Checker_Mock. Go // Mock methods ├── ├─ ├─ ├─ config │ ├─ ├.go │ ├── ├.go │ ├── ├.go │ ├─ ├─ ├─ ├─ ├─ ├─ ├.go │ ├─ server │ ├─ ├─ ├.go │ ├ ─ ├ ─ garbage, └─ garbage, │ ├ ─ garbage, │ ├ ─ garbageCopy the code

You can modify configurations such as listening ports in the etc/check.yaml file

Yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml

$ go run check.go -f etc/check.yaml
Starting rpc server at 127.0.0.1:8081...
Copy the code

8. Modify the API Gateway code to invoke the Add/Check RPC service

Modify the bookstore-api.yaml configuration file and add the following content

Add:
  Etcd:
    Hosts:
      - localhost:2379
    Key: add.rpc
Check:
  Etcd:
    Hosts:
      - localhost:2379
    Key: check.rpc
Copy the code

Use etCD to automatically discover available Add/Check services

Modify the internal/config/config. Go as follows, to increase the add/check service dependency

type Config struct {
    rest.RestConf
    Add   rpcx.RpcClientConf     // Manual code
    Check rpcx.RpcClientConf     // Manual code
}
Copy the code

Modify the internal/SVC/servicecontext. Go, is as follows:

type ServiceContext struct {
    Config  config.Config
    Adder   adder.Adder          // Manual code
    Checker checker.Checker      // Manual code
}

func NewServiceContext(c config.Config) *ServiceContext {
    return &ServiceContext{
        Config:  c,
        Adder:   adder.NewAdder(rpcx.MustNewClient(c.Add)),         // Manual code
        Checker: checker.NewChecker(rpcx.MustNewClient(c.Check)),   // Manual code}}Copy the code

Use ServiceContext to pass dependencies between different business logic

Modify the internal logic/addlogic. Go in the Add method, as follows:

func (l *AddLogic) Add(req types.AddReq) (*types.AddResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Adder.Add(l.ctx, &adder.AddReq{
        Book:  req.Book,
        Price: req.Price,
    })
    iferr ! =nil {
        return nil, err
    }

    return &types.AddResp{
        Ok: resp.Ok,
    }, nil
    // End of manual code
}
Copy the code

Add booksto the Bookstore system by calling Adder’s Add method

Modify the internal logic/checklogic. Go Check method, in the following:

func (l *CheckLogic) Check(req types.CheckReq) (*types.CheckResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Checker.Check(l.ctx, &checker.CheckReq{
        Book:  req.Book,
    })
    iferr ! =nil {
        return nil, err
    }

    return &types.CheckResp{
        Found: resp.Found,
        Price: resp.Price,
    }, nil
    // End of manual code
}
Copy the code

The price of a book is queried from the Bookstore system by calling checker’s Check method

9. Define database table structure and generate CRUD+cache code

Create RPC /model directory in bookstore: mkdir -p RPC /model

Create table book.sql in RPC /model directory

CREATE TABLE `book`
(
  `book` varchar(255) NOT NULL COMMENT 'book name',
  `price` int NOT NULL COMMENT 'book price'.PRIMARY KEY(`book`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
Copy the code

Create a DB and a table

create database gozero;
Copy the code

source book.sql;
Copy the code

Run the following command in the RPC /model directory to generate CRUD+cache code. -C indicates redis cache
```
goctl model mysql ddl -c -src book.sql -dir .
Copy the code
```
It is also possible to use the datasource command instead of the DDL to specify that the database link is generated directly from the schema

The generated file structure is as follows:
```
├─ bookstoremodel.go // set parameters and variablesCopy the code
```

10. Modify the ADD/Check RPC code to invoke the CRUd +cache code

Yaml and RPC /check/etc/check.yaml are modified with the following contents:
```
DataSource: root:@tcp(localhost:3306)/gozero
Table: book
Cache:
  - Host: localhost:6379
Copy the code
```
Multiple Redis can be used as the cache, supporting redis single point or Redis cluster

Modify the RPC/add/internal/config. Go and RPC/check/internal/config. Go, is as follows:

type Config struct {
    rpcx.RpcServerConf
    DataSource string             // Manual code
    Table      string             // Manual code
    Cache      cache.CacheConf    // Manual code
}
Copy the code

Added mysql and Redis cache configurations

Modify the RPC/add/internal/SVC/servicecontext. Go and RPC/check/internal/SVC/servicecontext. Go, is as follows:

type ServiceContext struct {
    c     config.Config
    Model *model.BookModel   // Manual code
}

func NewServiceContext(c config.Config) *ServiceContext {
    return &ServiceContext{
        c:             c,
        Model: model.NewBookModel(sqlx.NewMysql(c.DataSource), c.Cache, c.Table), // Manual code}}Copy the code

Modify the RPC/add/internal/logic/addlogic. Go, is as follows:

func (l *AddLogic) Add(in *add.AddReq) (*add.AddResp, error) {
    // Start the manual code
    _, err := l.svcCtx.Model.Insert(model.Book{
        Book:  in.Book,
        Price: in.Price,
    })
    iferr ! =nil {
        return nil, err
    }

    return &add.AddResp{
        Ok: true,},nil
    // End of manual code
}
Copy the code

Modify the RPC/check/internal/logic/checklogic. Go, is as follows:

func (l *CheckLogic) Check(in *check.CheckReq) (*check.CheckResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Model.FindOne(in.Book)
    iferr ! =nil {
        return nil, err
    }

    return &check.CheckResp{
        Found: true,
        Price: resp.Price,
    }, nil
    // End of manual code
}
Copy the code

So far the code modification is complete, all manual modification of the code I added a note

11. Full call demo

The add API call

curl -i "http://localhost:8888/add? book=go-zero&price=10"Copy the code

Returns the following:

HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 09:42:13 GMT
Content-Length: 11

{"ok":true}
Copy the code

Check the API call

curl -i "http://localhost:8888/check? book=go-zero"Copy the code

Returns the following:

HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 09:47:34 GMT
Content-Length: 25

{"found":true."price":10}
Copy the code

12. Benchmark

Because writing depends on mysql’s write speed, it is equivalent to pressing mysql, so the pressure test only tests the check interface, which is equivalent to reading from mysql and using the cache. For convenience, the pressure test directly presses this book, because there is a cache, and many books are the same, it does not affect the pressure test results.

Before pressing, let’s increase the number of open file handles:

ulimit -n 20000
Copy the code

Change the log level to ERROR to prevent too much info from affecting the pressure test results. Add the following to each YAML configuration file:

Log:
	Level: error
Copy the code

It can be seen that I can achieve 30,000 + QPS on my MacBook Pro.

13. Complete code

Github.com/tal-tech/go…

14. To summarize

We always emphasize tools over conventions and documentation.

Go-zero is not only a framework, but also a technical architecture based on framework + tools that simplifies and normalizes the entire microservice construction.

We kept it simple while encapsulating the complexity of microservices governance within the framework, greatly reducing the mental burden of developers and enabling business development to move forward quickly.

The code generated by Go-Zero + GoCTL contains various components of microservice governance, including concurrency control, adaptive fuses, adaptive load reduction, automatic cache control, and so on, and can be easily deployed to accommodate large traffic.

Any good ideas to improve the efficiency of the project, welcome to exchange at any time! 👏

15. Project address

Github.com/tal-tech/go…

16. Wechat communication group

To add my wechat account: Kevwan, please specify Go-Zero, AND I will join the Go-Zero community group 🤝

Good future technology

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Are you still ripping the app? Try go-Zero’s automatic microservice generation

Why is microservices so hard to do?

1. Introduction of bookstore service examples

2. Micro-service architecture diagram of the bookstore

3. Overview of code generation for each layer of Goctl

4. Preparation

5. Write API Gateway code

6. Write the Add RPC service

7. Write the Check RPC service

8. Modify the API Gateway code to invoke the Add/Check RPC service

9. Define database table structure and generate CRUD+cache code

10. Modify the ADD/Check RPC code to invoke the CRUd +cache code

11. Full call demo

12. Benchmark

13. Complete code

14. To summarize

15. Project address

16. Wechat communication group

Are you still ripping the app? Try go-Zero’s automatic microservice generation

Why is microservices so hard to do?

1. Introduction of bookstore service examples

2. Micro-service architecture diagram of the bookstore

3. Overview of code generation for each layer of Goctl

4. Preparation

5. Write API Gateway code

6. Write the Add RPC service

7. Write the Check RPC service

8. Modify the API Gateway code to invoke the Add/Check RPC service

9. Define database table structure and generate CRUD+cache code

10. Modify the ADD/Check RPC code to invoke the CRUd +cache code

11. Full call demo

12. Benchmark

13. Complete code

14. To summarize

15. Project address

16. Wechat communication group

Related Posts

Kafka: Producer

Hive Learning Note 6: HiveQL Basics

Spring source code learning (2) default tag parsing