Why is microservices so hard to do?

In order to do a good job in micro services, we need to understand and master a lot of knowledge points, from several dimensions:

  • Basic functional level

    1. Concurrency control & limiting traffic to prevent service from being overwhelmed by sudden traffic
    2. Service registration and service discovery to ensure that nodes can be dynamically detected
    3. Load balancing: Distributes traffic based on the capacity of nodes
    4. Time-out control to avoid idle efforts for time-out requests
    5. Fuse breaker design, fast failure, guarantee the recovery ability of failed nodes

  • Higher functional level

    1. Request authentication to ensure that each user can access only his or her own data
    2. Link tracing, used to understand the entire system and quickly locate problems with specific requests
    3. Logs are used for data collection and problem locating
    4. Observability, no measurement, no optimization

For each of these points, we need to explain the principle and implementation of a long time, so it is very difficult for us back-end developers to master these points and implement them into the business system, but we can rely on a framework that has been proven by a large amount of traffic. The Go-Zero microservices framework was born for this purpose.

In addition, we always believe in tools over conventions and documentation. We wanted to reduce the mental burden of developers as much as possible, focus on code that produces business value, and reduce duplication of code, so we developed the GoCTL tool.

Here’s a quick tutorial on how to create a microservice with Go-Zero using a bookstore service. Once you’re done, you’ll see how easy it is to write a microservice!

1. Introduction of bookstore service examples

To keep the tutorial simple, we will use the bookstore service as an example and implement only the functions of adding books and checking prices.

The bookstore service was written to demonstrate the whole process of go-Zero building a complete microservice, with implementation details as simple as possible.

2. Micro-service architecture diagram of the bookstore

3. Overview of code generation for each layer of Goctl

All the functional modules with green background are automatically generated and activated on demand, while the red modules need to be written by themselves, that is, to add dependencies and write business-specific logic. The diagram of each layer is as follows:

  • API Gateway

  • RPC

  • model

Let’s Go through the process of building microservices quickly. Let’s Go! 🏃 came ️

4. Preparation

  • Install etCD, mysql, redis

  • Install the goctl tool

    GO111MODULE=on GOPROXY=https://goproxy.cn/,direct go get -u github.com/tal-tech/go-zero/tools/goctl
Copy the code

  • Create the working directory Bookstore

  • Initialize go.mod by executing go mod init bookstore in the bookstore directory

5. Write API Gateway code

  • Bookstore/API generated by goctl in bookstore/ API:

    goctl api -o bookstore.api
Copy the code

    Edit bookstore. API and remove the info at the beginning of the file for brevity as follows:

    type (
    addReq struct {
        book  string `form:"book"`
        price int64  `form:"price"`
    }

    addResp struct {
        ok bool `json:"ok"`})type (
    checkReq struct {
        book string `form:"book"`
    }

    checkResp struct {
        found bool  `json:"found"`
        price int64 `json:"price"`
    }
)

service bookstore-api {
    @server(
        handler: AddHandler
    )
    get /add(addReq) returns(addResp)

    @server(
        handler: CheckHandler
    )
    get /check(checkReq) returns(checkResp)
}
Copy the code

    The usage of type is the same as that of go. Service defines API requests such as GET, POST, head, and delete.

    • service bookstore-api {This line defines the service name
    • @serverPart defines the attributes used by the server
    • handlerDefines the server handler name
    • get /add(addReq) returns(addResp)Defines routes, request parameters, return parameters, and so on for the GET method

  • Use goctl to generate API Gateway code

    goctl api go -api bookstore.api -dir .
Copy the code

    The generated file structure is as follows:

    API ├ ─ ─ bookstore. API / / API defines ├ ─ ─ bookstore. Go / / the main entry definition ├ ─ ─ etc │ └ ─ ─ bookstore - API. Yaml / / configuration file └ ─ ─ internal ├ ─ ─ ├─ ├─ ├.go config │ ├─ ├.go config │ ├─ ├.go └ ─ ─ routes. Go / / define routing processing ├ ─ ─ logic │ ├ ─ ─ addlogic. Go / / implementation addlogic │ └ ─ ─ checklogic. Go / / implementation checklogic ├ ─ ─ SVC │ └ ─ ─ Conf // conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/conf/confCopy the code

  • Start the API Gateway service, listening on port 8888 by default

    go run bookstore.go -f etc/bookstore-api.yaml
Copy the code

  • Test the API Gateway service

    curl -i "http://localhost:8888/check? book=go-zero"Copy the code

    Returns the following:

    HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 06:46:18 GMT
Content-Length: 25

{"found":false."price":0}
Copy the code

    As you can see, our API Gateway does nothing but return a null value, and we will implement the business logic in the RPC service

  • Can modify the internal/SVC/servicecontext. Go to deliver services rely on (if required)

  • The implementation logic can modify corresponding files in internal/ Logic

  • You can use Goctl to generate API call code for various client languages

  • At this point, you can use goctl to generate client code for concurrent development of the client, support multiple languages, see documentation

6. Write the Add RPC service

  • Write the add.proto file in the RPC /add directory

    Proto file templates can be generated by command

    goctl rpc template -o add.proto
Copy the code

    The modified file content is as follows:

    syntax = "proto3";

package add;

message addReq {
    string book = 1;
    int64 price = 2;
}

message addResp {
    bool ok = 1;
}

service adder {
    rpc add(addReq) returns(addResp);
}
Copy the code

  • Use goctl to generate RPC code and execute commands in the RPC /add directory

    goctl rpc proto -src add.proto
Copy the code

    The file structure is as follows:

    ├─ Add.go // RPC/Add.go // RPC/Add.go // RPC/Add.go // RPC/Add.go // RPC/add.go // RPC/add.go // │ ├─ Adder_Mock. Go // Mock methods │ ├── ├─ ├─ ├─ ├─ ├─ ├.go // ├─ ├─ config │ ├.go │ ├─ ├─ ├─ ├─ ├─ ├─ ├.go │ ├─ server │ ├─ ├.go Don't need to modify the │ └ ─ ─ SVC │ └ ─ ─ servicecontext. Go / / define servicecontext, transfer rely on └ ─ ─ pb └ ─ ─ the add. Pb. GoCopy the code

It can run directly as follows:

  $ go run add.go -f etc/add.yaml
  Starting rpc server at 127.0.0.1:8080...
Copy the code

You can modify configurations such as listening ports in the etc/add.yaml file

7. Write the Check RPC service

  • Write the check.proto file in the RPC /check directory

    Proto file templates can be generated by command

    goctl rpc template -o check.proto
Copy the code

    The modified file content is as follows:

    syntax = "proto3";

package check;

message checkReq {
    string book = 1;
}

message checkResp {
    bool found = 1;
    int64 price = 2;
}

service checker {
    rpc check(checkReq) returns(checkResp);
}
Copy the code

  • Goctl generates RPC code and executes commands in the RPC /check directory

    goctl rpc proto -src check.proto
Copy the code

    The file structure is as follows:

    RPC/Check ├─ Check. Go // RPC Service Main Function ├─ Check. Proto // RPC interface ├─ Check. │ ├─ Checker_Mock. Go // Mock methods ├── ├─ ├─ ├─ config │ ├─ ├.go │ ├── ├.go │ ├── ├.go │ ├─ ├─ ├─ ├─ ├─ ├─ ├.go │ ├─ server │ ├─ ├─ ├.go │ ├ ─ ├ ─ garbage, └─ garbage, │ ├ ─ garbage, │ ├ ─ garbageCopy the code

    You can modify configurations such as listening ports in the etc/check.yaml file

    Yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml/etc/check.yaml

    $ go run check.go -f etc/check.yaml
Starting rpc server at 127.0.0.1:8081...
Copy the code

8. Modify the API Gateway code to invoke the Add/Check RPC service

  • Modify the bookstore-api.yaml configuration file and add the following content

    Add:
  Etcd:
    Hosts:
      - localhost:2379
    Key: add.rpc
Check:
  Etcd:
    Hosts:
      - localhost:2379
    Key: check.rpc
Copy the code

    Use etCD to automatically discover available Add/Check services

  • Modify the internal/config/config. Go as follows, to increase the add/check service dependency

    type Config struct {
    rest.RestConf
    Add   rpcx.RpcClientConf     // Manual code
    Check rpcx.RpcClientConf     // Manual code
}
Copy the code

  • Modify the internal/SVC/servicecontext. Go, is as follows:

    type ServiceContext struct {
    Config  config.Config
    Adder   adder.Adder          // Manual code
    Checker checker.Checker      // Manual code
}

func NewServiceContext(c config.Config) *ServiceContext {
    return &ServiceContext{
        Config:  c,
        Adder:   adder.NewAdder(rpcx.MustNewClient(c.Add)),         // Manual code
        Checker: checker.NewChecker(rpcx.MustNewClient(c.Check)),   // Manual code}}Copy the code

    Use ServiceContext to pass dependencies between different business logic

  • Modify the internal logic/addlogic. Go in the Add method, as follows:

    func (l *AddLogic) Add(req types.AddReq) (*types.AddResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Adder.Add(l.ctx, &adder.AddReq{
        Book:  req.Book,
        Price: req.Price,
    })
    iferr ! =nil {
        return nil, err
    }

    return &types.AddResp{
        Ok: resp.Ok,
    }, nil
    // End of manual code
}
Copy the code

    Add booksto the Bookstore system by calling Adder’s Add method

  • Modify the internal logic/checklogic. Go Check method, in the following:

    func (l *CheckLogic) Check(req types.CheckReq) (*types.CheckResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Checker.Check(l.ctx, &checker.CheckReq{
        Book:  req.Book,
    })
    iferr ! =nil {
        return nil, err
    }

    return &types.CheckResp{
        Found: resp.Found,
        Price: resp.Price,
    }, nil
    // End of manual code
}
Copy the code

    The price of a book is queried from the Bookstore system by calling checker’s Check method

9. Define database table structure and generate CRUD+cache code

  • Create RPC /model directory in bookstore: mkdir -p RPC /model

  • Create table book.sql in RPC /model directory

    CREATE TABLE `book`
(
  `book` varchar(255) NOT NULL COMMENT 'book name',
  `price` int NOT NULL COMMENT 'book price'.PRIMARY KEY(`book`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
Copy the code

  • Create a DB and a table

    create database gozero;
Copy the code
    source book.sql;
Copy the code

  • Run the following command in the RPC /model directory to generate CRUD+cache code. -C indicates redis cache

    goctl model mysql ddl -c -src book.sql -dir .
Copy the code

    It is also possible to use the datasource command instead of the DDL to specify that the database link is generated directly from the schema

    The generated file structure is as follows:

    ├─ bookstoremodel.go // set parameters and variablesCopy the code

10. Modify the ADD/Check RPC code to invoke the CRUd +cache code

  • Yaml and RPC /check/etc/check.yaml are modified with the following contents:

    DataSource: root:@tcp(localhost:3306)/gozero
Table: book
Cache:
  - Host: localhost:6379
Copy the code

    Multiple Redis can be used as the cache, supporting redis single point or Redis cluster

  • Modify the RPC/add/internal/config. Go and RPC/check/internal/config. Go, is as follows:

    type Config struct {
    rpcx.RpcServerConf
    DataSource string             // Manual code
    Table      string             // Manual code
    Cache      cache.CacheConf    // Manual code
}
Copy the code

    Added mysql and Redis cache configurations

  • Modify the RPC/add/internal/SVC/servicecontext. Go and RPC/check/internal/SVC/servicecontext. Go, is as follows:

    type ServiceContext struct {
    c     config.Config
    Model *model.BookModel   // Manual code
}

func NewServiceContext(c config.Config) *ServiceContext {
    return &ServiceContext{
        c:             c,
        Model: model.NewBookModel(sqlx.NewMysql(c.DataSource), c.Cache, c.Table), // Manual code}}Copy the code

  • Modify the RPC/add/internal/logic/addlogic. Go, is as follows:

    func (l *AddLogic) Add(in *add.AddReq) (*add.AddResp, error) {
    // Start the manual code
    _, err := l.svcCtx.Model.Insert(model.Book{
        Book:  in.Book,
        Price: in.Price,
    })
    iferr ! =nil {
        return nil, err
    }

    return &add.AddResp{
        Ok: true,},nil
    // End of manual code
}
Copy the code

  • Modify the RPC/check/internal/logic/checklogic. Go, is as follows:

    func (l *CheckLogic) Check(in *check.CheckReq) (*check.CheckResp, error) {
    // Start the manual code
    resp, err := l.svcCtx.Model.FindOne(in.Book)
    iferr ! =nil {
        return nil, err
    }

    return &check.CheckResp{
        Found: true,
        Price: resp.Price,
    }, nil
    // End of manual code
}
Copy the code

    So far the code modification is complete, all manual modification of the code I added a note

11. Full call demo

  • The add API call

    curl -i "http://localhost:8888/add? book=go-zero&price=10"Copy the code

    Returns the following:

    HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 09:42:13 GMT
Content-Length: 11

{"ok":true}
Copy the code

  • Check the API call

    curl -i "http://localhost:8888/check? book=go-zero"Copy the code

    Returns the following:

    HTTP / 1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Sep 2020 09:47:34 GMT
Content-Length: 25

{"found":true."price":10}
Copy the code

12. Benchmark

Because writing depends on mysql’s write speed, it is equivalent to pressing mysql, so the pressure test only tests the check interface, which is equivalent to reading from mysql and using the cache. For convenience, the pressure test directly presses this book, because there is a cache, and many books are the same, it does not affect the pressure test results.

Before pressing, let’s increase the number of open file handles:

ulimit -n 20000
Copy the code

Change the log level to ERROR to prevent too much info from affecting the pressure test results. Add the following to each YAML configuration file:

Log:
	Level: error
Copy the code

It can be seen that I can achieve 30,000 + QPS on my MacBook Pro.

13. Complete code

Github.com/tal-tech/go…

14. To summarize

We always emphasize tools over conventions and documentation.

Go-zero is not only a framework, but also a technical architecture based on framework + tools that simplifies and normalizes the entire microservice construction.

We kept it simple while encapsulating the complexity of microservices governance within the framework, greatly reducing the mental burden of developers and enabling business development to move forward quickly.

The code generated by Go-Zero + GoCTL contains various components of microservice governance, including concurrency control, adaptive fuses, adaptive load reduction, automatic cache control, and so on, and can be easily deployed to accommodate large traffic.

Any good ideas to improve the efficiency of the project, welcome to exchange at any time! 👏

15. Project address

Github.com/tal-tech/go…

16. Wechat communication group

To add my wechat account: Kevwan, please specify Go-Zero, AND I will join the Go-Zero community group 🤝

Good future technology