advice

• Video to see books or to see to systematic learning
• Systematic summary and thinking
• Learning methodology
• Margin Note recommendation software

Engineering project structure

• The directory organization

• named

• layered

• Resource initialization

• Dependency injection

• Inversion of control

  MAO watched more than 20 refences and four YouTube videos

Standard Go Project Layout

Github.com/golang-stan…

The necessity of engineering

• Get your own main.go
  • Start with something very simple (a main.go file is more than enough)
  • If you are trying to learn Go, or if you are building a PoC or a toy project for yourself, this project layout is not necessary
• More and more people are involved in projects that need to be structured and engineered together
• You will need more structure, including a Toolkit to easily generate templates for projects, and a unified project directory layout as far as possible.

/cmd

• Store the main file of the project
• It’s not a good idea to put too much code in main to do some initialization
• The directory name for each application should match the name of the executable you want
  • (for example, / CMD /myapp).
  • Don’t put too much code in this directory.
• If you think the code can be imported and used in other projects, it should be in the/PKG directory.
• If your code is not reusable, or you don’t want others to reuse it, put it in the /internal directory.

/internal

• It can only be imported into this project
• For example, internal/demo
• Private application and library code. This is where you don’t want someone else importing code into their application or library.
• This layout pattern is performed by the Go compiler itself.
  • See Go 1.4 Release Notes for more details.

Pay attention to

You are not limited to the top-level internal directory. There can be multiple internal directories at any level of the project tree. You can choose to add some additional structure to the internal package to separate shared and non-shared internal code. This is not required (especially for smaller projects), but it is good to have visual clues to the intended purpose of the package.

• The actual application code can be placed in the /internal/app directory (e.g. /internal/app/myapp),
• Application shared code can be placed in /internal/ PKG (for example, /internal/ PKG /myprivlib).

Because we are used to integrating related services, such as account services, with internal RPC, Job, admin, etc., app needs to be differentiated. For a single service, drop /internal/myapp.

You can also create a PKG under each app and this is only for minor reuse by that app

/pkg

• Reusable code utils, Common, etc

• Library code that can be used by external applications (e.g. / PKG/myPubliclib). Other items import into these libraries, so think twice before putting anything there :-),

  • The internal directory is a better way to ensure that private packages cannot be imported, as it is enforced by Go.

• The/PKG directory is still a good way to explicitly indicate that the code in that directory is a good way for others to use safely.

• / PKG directory, you can refer to the go standard library organization mode, according to the function.

  • string net http

• /internla/ PKG is used for common shared code across multiple applications within a project, but only within a single project.

• Written by Travis Jeffery I ‘l l take PKG over internal blog posts (travisjeffery.com/b/2019/11/i… A good overview of the PKG and internal directories and when it makes sense to use them.

This is also a way to group the Go code into one location when the root directory contains a large number of non-GO components and directories, making it easier to organize the various Go tools running.

Kit Project Layout

• Every company should have a uniform kit kit project (base library/framework) and APP project for different microservices.
  • The common code is all put together
• The basic library Kit is an independent project, and it is recommended to have only one at the company level. It should be dismantled according to the functional directory
  • The branch brings a lot of management work, so it is suggested to merge.
  • Do it administratively and don’t do it on your own

Refer to the article Package Oriented Design (www.ardanlabs.com/blog/2017/0.)

“To this end, the Kit project is not allowed to have a vendor folder. If any of packages are dependent on 3rd party packages, They must always build against the latest version of those dependences.”

• Third party packages are not allowed because of dependencies
• Kit packages relying on third parties can cause problems as dependencies change

Kratos library

Features required for kit projects:

• Unify normative things as soon as possible
• Standard library layout
  • The naming of package is described in Effectice Go
• Highly abstract
• Support plugins
• Go – based library kit

Service Application Project Layout

/api

Store the API protocol definition directory, xxapi.proto protobuf file, and the generated GO file. We usually describe API documentation directly in proto files.

• Put the DEFINITION of the API in your project so that people can find it when they call your code
  • Everyone can’t see it
• The old way of doing it was somebody else wrote the API documentation and somebody else read the documentation

/configs

Configuration file template or default configuration.

• Suggest YAML

/test

Additional external test applications and test data. You can always construct /test directories on demand. For larger projects, it makes sense to have a data subdirectory. For example, you can use /test/data or /test/testdata (if you need to ignore the contents in the directory). Note that Go also ignores the word “. Or directories or files beginning with “_”, so you have more flexibility in how you name your test data directories.

The emphasis should not include :/ SRC

• Some Go projects do have a SRC folder, but this usually happens when the developer has a Java background, where it is a common pattern.
• Do not use the project-level SRC directory with Go for the SRC directory of their workspace.
• Go path (package) first-class citizen, does not require SRC
• Distinguish between GOPATH (Workspace)

Service Application Project

A GitLab project can place multiple micro-service apps (similar to Monorepo). Multiple projects can also be established according to the Group of GitLab, and each project corresponds to an APP.

• In the multi-app mode, each micro-service in the APP directory is created according to its own global unique name, such as “account.service. VIP”, for example: Account/VIP /*.
• The app level directory PKG holds common libraries (not infrastructure libraries) related to the business. If your application does not want to export these directories, you can place them in myapp/internal/ PKG.

The app service types in microservices are divided into four types: Interface, Service, Job, and admin.

interface:

• External BFF services that accept requests from users, such as exposing HTTP/gRPC interfaces.
• It’s the BFF layer and the top layer is the APIGateway
• xxxinterface

service:

• Internal microservices only accept requests from other internal services or gateways, such as exposing the gRPC interface for internal services only.
• xxxservice

admin:

• Unlike Services, they are more business-oriented services, usually with higher data permissions, and isolation provides better code-level security.

job:

• Biased resident execution
• Services for streaming task processing typically rely on Message Broker upstream.
• For example, binlog subscription

task:

• Biased toward scheduled execution
• Scheduled tasks, such as cronjobs, are deployed on the task hosting platform.
• Hosting to the scheduled task platform
  • gocron
  • airflow
  • Cobra CMD library

The CMD application directory is responsible for starting, closing, configuration initialization, and so on. Listening logs for resource initialization

Evolve Service Application project-v1

On our old layout, the app directory had API, CMD, configs, internal, and README, CHANGELOG, OWNERS.

• API: Place API definition (Protobuf), and corresponding generated client code, based on pB-generated swagger.json.
• Yaml, redis. Yaml, and application.yaml.
• Internal: to avoid the possibility that someone in the same business references the internal directory
  • The model structure maps the MYSQL structure table
  • Dao and other internal structs. (data acess object)
    • Access DB, table-oriented, one method for each table
  • The Service business logic depends on the DAO
  • server
• Server: The routing code to place HTTP/gRPC, and the DTO conversion code.
• CMD initialized

The model sample

• The dao depend on the model
• Service relying on the dao
• Server depends on the service
• restful —> server
  • How to write the service and listener bindings

  

Pain points

Json: “-” – josn: int64 not supported – mysql layer data is thrown to the presentation layer – complex presentation layer data needs to be coupled to the data layer code

To solve

Data Transfer Object (DTO): Data Transfer Object, this concept is derived from the J2EE design mode. But here, the general term is used to refer to data transfer objects between the presentation layer /API layer and the service layer (business logic layer).Copy the code

Depend on the cause

The dependency path of the project is: Model -> DAO -> service -> API, model struct is connected to each layer, until the API needs to do DTO object conversion.

• Model: Put the structure corresponding to the “storage layer”, is the one – by – one allusion to storage.
• Dao: Data read and write layer, database and cache are all processed in this layer.
  • This includes cache miss handling.
  • Processing started at the business logic layer and evolved to the DAO layer
  • The business layer focuses on the business
• Service: Combines various data access to build business logic.
• Server: Relies on the service defined by Proto as an input parameter to provide a quick global method to start the service.
• API: Defines the APIproto file and generates stub code that generates the interface implementer in the service.

Service method signature because the implementation of the API interface definition, DTO directly in the business logic layer directly used, more directly used DAO, the most simplified code.

Anemia model

DO(Domain Object): Domain Object

• Business entities, tangible or intangible, abstracted from the real world. Object conversion lacking DTO -> DO.
• business-oriented

Evolve Service Application project-v2

The app directory contains the API, CMD, configs, and internal directories. README, CHANGELOG, and OWNERS are the common directories.

internal:

• In order to avoid the internal structs such as biz, data and service referenced across directories in the same business.

biz:

• The assembly layer of business logic, the domain layer like DDD, data like THE REPO of DDD, the REPO interface is defined here, using the principle of dependency inversion.
• Persistent interfaces are defined at the business logic layer
• Business logic layer dependency persistence once (dependency inversion)

data:

• Business data access, including cache, DB encapsulation, biz repO interface. We might confuse Data with DAO, data is heavy on business meaning, all it does is take domain objects back out, we remove the infra layer of DDD.

service:

• It implements the apI-defined service layer, the ddD-like application layer, which handles the DTO -> DO transformation of biz domain entities, and coordinates various biz interactions, but should not deal with complex logic.
• Pay attention to the realization of GRPC interface layer

Persistent Object (PO): It forms a one-to-one mapping relationship with the data structure of the persistence layer (usually a relational database). If the persistence layer is a relational database, each field (or several fields) in the data table corresponds to one (or several) attributes of THE PO. Github.com/facebook/en… (Not recommended by GORM)

Lifecycle

Lifecycle needs to consider the object initialization and Lifecycle management of the service application, the initialization of all HTTP/gRPC dependent pre-resources including data, biz, service, and then start the listening service.

We use github.com/google/wire to manage dependency injection for all resources.

Why dependency injection?

The core is to:

• 1, convenient test;
• 2. Single initialization and reuse;

Inversion of control

Instead of initializing internally you have to create multiple links if multiple people want to do it and that’s not an easy thing to do internally but hopefully the tests are initialized externally and then distributed to different people who need them so they can reuse them and it’s also easy for unit testing

Wire

The initialization and shutdown of blog.golang.org/wire manual resources are tedious and error-prone. As mentioned above, we used DI, combined with Google Wire, static go generate to generate static code, which can be easily diagnosed and viewed, instead of using Reflection at run time.

API design

• The focus will be gRPC
• Protobuff define message
• The transport layer (TPC/HTTP/RPC) is isolated from the service implementation

gRPC

What is gRPC can be summed up with a sentence on the official website

“A High-performance, Open-source Universal RPC Framework”

• Multilingual: Language neutral, supports multiple languages.
• Lightweight and high-performance: Serialization supports Protocol Buffer (PB) and JSON. PB is a language-independent high-performance serialization framework.
• pluggable
• IDL: Defines services based on files. Proto3 tools are used to generate data structures, server interfaces, and client stubs in specified languages.
• Design concept
• Mobile: Based on the standard HTTP2 design, support bidirectional flow, header compression, single TCP multiplexing, server push and other features, these features make gRPC on mobile devices more energy saving and save network traffic.
• Services rather than objects, messages rather than references: coarse-grained message interaction design concepts between systems that facilitate microservices.
• Load-independent: Different services require different message types and encodings, such as Protocol Buffers, JSON, XML, and Thrift.
• Streaming: Streaming API.
• Blocking and non-blocking: Supports asynchronous and synchronous processing of message sequences that interact between client and server.
• Metadata exchange: Common crosscutting concerns, such as authentication or tracing, rely on data exchange.
• Standardized status codes: Clients typically respond in limited ways to errors returned by API calls.

Take a look at the GPRC documentation and use it

Don’t focus on performance too early, standardize first.

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helloworld/helloworld.proto
Copy the code

API Project

Github.com/googleapis/… Github.com/envoyproxy/… github.com/istio/api

Automatically updates a project’s API directory and then automatically updates it to a dedicated external directory corresponding to another project

• Hidden source code can only see the API code
• It refers to Gogole’s ideas
• Have a good directory structure
• Easy to do API version management
  • Code review, version control

How to avoid deleting others’ files by mistake

To unify the retrieval and specification of apis, we built a unified BAPIS repository internally, integrating all internal and external apis.

• API repository for cross-departmental collaboration.
• Version management, based on Git control.
• Normalization check, APIlint.
• APIdesignreview, change diff.
• Permission management, directory OWNERS.

  • Each directory defines the OWNERS file tag authors

API Project Layout

Proto is defined in the project and API is used as the root directory of the package name. Proto is managed in the unified repository and warehouse is used as the root directory of the package name.

API Compatibility

Backward compatible (non-destructive) changes

• Adding an API interface to an API service definition is always safe from a protocol perspective.
• Adding a field to a request message is compatible as long as the client does not treat the field the same in the new and old versions.
• Add fields to the response message

Response messages that are not resources (for example, ListBooksResponse) can be extended without necessarily breaking client compatibility without changing the behavior of other response fields. Any fields previously populated in the response should continue to be populated with the same semantics, even if redundancy is introduced.

Backward incompatible (destructive) modifications

• Deletes or renames a service, field, method, or enumeration value
  • Basically, if the client code can reference something, then deleting or renaming it is an incompatible change, and the major version number must be changed.
• Changing the type of the field Even if the new type is transport-format compatible, this may cause a change in the code generated by the client library and therefore must be added
  • Major Version number. For compiled static languages, compilation errors are easy to introduce. • Modify the visible behavior of existing requests
  • Clients often rely on API behavior and semantics, even if such behavior is not explicitly supported or documented. Therefore, in most cases, the behavior or semantics of modifying API data will be viewed by consumers as disruptive. If the behavior is not cryptographically hidden, you should assume that the user has discovered it and will rely on it.
• Add read/write fields to resource messages

API Naming Conventions

The package name is the application identifier (APP_ID) and is used to generate gRPC request paths or to reference messages between PROTos. Package names declared in the file should be consistent with product and service names. The package name with the API version must end with that version.

• My.package. v1, for the API directory, defines the service related interface to provide business use.

// RequestURL: /<package_name>.. <service_name>/{method} package <package_name>.;

Refer to the googleAPI design specification

The input parameter is designed as an object for later extension

API Primitive Fields

GRPC uses the Protobuf V3 format by default because the required and optional keywords are removed and all optional fields are default. If there is no assigned field, the default value for the underlying type field is 0 or “”.

For Protobuf V3, you are advised to use github.com/protocolbuf… Fields of type Warpper, which wrap a message, use time-varying Pointers. Protobuf is a strong schema description file that can be easily extended. Can it be used for configuration file definitions?

API Errors

GRPC also uses standard status codes according to Google’s specifications

Use a small group of standard errors and a large number of resources

For example, instead of defining different types of “not found” errors, the server uses a standard Google.rpc.code. NOT_FOUND error Code and tells the client which particular resource was not found. The smaller state space reduces the complexity of the document, provides better idiomatic mapping in the client library, and reduces the logical complexity of the client without limiting the inclusion of actionable information (/ Google/RPC /error_details).

Error propagation

If your API services depend on other services, you should not blindly propagate errors from those services to your clients. In case of translation errors, we recommend the following: Hide implementation details and confidential information. Adjust the party responsible for the error. For example, a server that receives an INVALID_ARGUMENT error from another service should propagate INTERNAL to its own callers.

Global error code

Global error codes are loose, contract-breaking, and based on our discussion above, a translation for each service that propagates an error ensures that each service + error enumeration should be unique and documented in the PROTO definition.

API Design

FieldMask partial update:, the client can perform need to update the field information: paths: “author” paths: “submessage. Submessage. Field” empty FieldMask default to “all of the fields”

Configuration management

• The completed solution
• How do I create and manage configuration files
• Initialize the

Environment Variables (Configuration)

Region, Zone, Cluster, Environment, Color, Discovery, AppID, Host, and other Environment information are imported into containers or physical machines through the online runtime platform for the kit library to read.

Static configuration

For resources that need to be initialized, such as HTTP /gRPC Server, Redis, and mysql, it is very risky to change the configurations of these resources online. I usually do not encourage on-the-fly changes, which may lead to unexpected accidents. Changing the static configuration is no different from releasing the Bianry app. It should be an iterative release.

Dynamic configuration

The application may require some on-line switches to control some simple policies of the business, which are frequently adjusted and used. We use these configurations as base types (int, bool) to dynamically change the business flow together. At the same time, it can be combined with the official standard library like pkg.go.dev/expvar.

Global configuration

Usually, all kinds of components and middleware we rely on have a large number of default configurations or specified configurations, which are copied in each project, which is prone to accidents. Therefore, we use global configuration templates to customize commonly used components, and then carry out partial replacement in specialized applications.

Configuration Best Pratice

Code changing system functionality is a lengthy and complex process that often involves Review, testing, etc., but changing a single configuration option can have a significant impact on functionality, and often the configuration is untested. Configuration objectives:

• Avoid complex
• Multiple configurations
• Simplification effort
• Infrastructure -> User oriented transformation
  • User oriented don’t be complicated
  • Middleware can be complex
• Mandatory and optional configurations
• Configure defense programming
• Permissions and change tracking
• The configured version is aligned with the application
• Security configuration changes: progressive deployment, rollback changes, automatic rollback

Package management

• gomod

Github.com/gomods/athe… Goproxy. Cn blog.golang.org/modules2019 blog.golang.org/using-go-mo… Blog.golang.org/migrating-t… Blog.golang.org/module-mirr… Blog.golang.org/publishing-… Blog.golang.org/v2-go-modul… Blog.golang.org/module-comp…

test

• Small tests lead to good code quality, good exception handling, and elegant error reporting; Medium – to large-size testing leads to overall product quality and data validation.
• Different types of projects have different requirements for testing, and there is a general rule of thumb, called the 70/20/10 rule: 70% small tests, 20% medium tests and 10% large tests.
• If a project is user-oriented, has a high degree of integration, or a complex user interface, they should have more medium and large tests; If it is a basic platform or a data-oriented project, such as an index or web crawler, it is better to have a large number of small tests, with a much smaller number of medium and large tests required.

“Code implemented automatically to verify that a single function or module of independent functionality works as expected, focusing on typical functionality problems, data corruption, error conditions, and size difference errors. Off-by-one errors are a common type of programming error. “- The Google Way of Testing Software

Basic requirements for unit testing:

• fast
• The environment is consistent
• Any order
• parallel

Docker-compose based on the implementation of cross-platform cross-language container dependency management solution, to solve the problem of (mysql, Redis, MC) container dependency in the scenario of running unittest:

• Install Docker locally.
• Non-invasive environment initialization.
• Quickly reset the environment.
• Run anytime, anywhere (independent of external services).
• A semantic API declares resources.
• Real external dependencies, not in-process emulation.

Properly check the health of the services in the container so that resources are not ready when UNITTest starts. App should initialize data by itself, such as DB scheme and initial SQL data. In order to meet the consistency of the test, the container will be destroyed at the end of each test.

Before unit testing begins, import the packaged Testing library to make it easy to start and destroy the container. For unit tests of services, use libraries such as Gomock to mock out daOs, so you should program for abstraction when designing packages. To execute Unittest in CI environment, you need to consider the Docker network in physical machine, or start a Docker in Docker again.

Complete the entire unit test using Subtests + Gomock provided by GO. / API is suitable for integration testing, directly testing API, using API testing framework (such as YAPI), and maintaining a large number of business test cases. / Data Docker compose simulates the underlying infrastructure realistically, thus removing the abstraction layer of infra. /biz relies on THE REPO and RPC client, and uses Gomock to simulate the implementation of interface to conduct business unit tests. /service relies on the biz implementation and builds the biz implementation class passed in for unit testing. Conduct feature development based on git branch, conduct unittest locally, then submit gitlab merge request for unittest of CI, build based on feature branch, and complete functional test. Master is then merged for integration testing and regression testing once live.

• The API test
• The DAO test
• Test server layer
• The project structure is well structured and the tests are efficient and convenient
• Unit testing
  • The technical adjustment is good
  • Technical debt

References