A distributed unique ID generation system developed by Go

Id-maker is an open source project designed to generate unique ids in distributed environments. I stopped last week to develop and test the code for the project.

Meituan has an open source project called Leaf, which is developed in Java. This project is based on this idea, using Go development to achieve.

The overall amount of code in the project is not large, so I think it is a good choice whether you want to use it in the actual production environment or find a project to practice.

Project background

Globally unique ids are a strong requirement on most systems. For example, express delivery, takeout, movies, etc., all need to generate a unique ID to ensure that the single number is unique.

What are the requirements of business systems for ID numbers?

  1. Global uniqueness: No duplicate ID numbers, which is a basic requirement since they are unique.
  2. Increasing trend: The MySQL InnoDB engine uses clustered indexes. Since most RDBMS use b-tree data structure to store index data, we should try to use ordered primary keys to ensure write performance.
  3. Monotonically increasing: Ensure that the next ID is greater than the previous ID. For example, the transaction version number, IM incremental messages, and sorting are required.
  4. Information security: if the ID is continuous, malicious user pickpocket work is very easy to do, directly in order to download the specified URL; If it is the order number, it is even more dangerous, the competitor can directly know our daily order. Therefore, in some application scenarios, irregular and irregular ids are required.

In this context, it is important to have a highly available unique ID generation system.

Projects using

There are two ways to generate ids:

  1. Generate ids based on the database.
  2. Generate ID according to snowflake algorithm.

There are two ways to invoke an interface:

  1. The HTTP method
  2. GRPC way

The HTTP method

1. Health check:

The curl http://127.0.0.1:8080/pingCopy the code

2, obtain ID:

Get ID of test where tag is test:

The curl http://127.0.0.1:8080/v1/id/testCopy the code

3. Get snowflake ID:

The curl http://127.0.0.1:8080/v1/snowidCopy the code

GRPC way

1, obtain ID:

grpcurl -plaintext -d '{"tag":"test"}' -import-path $HOME/src/id-maker/internal/controller/rpc/proto -proto segment.proto localhost:50051 proto.Gid/GetId
Copy the code

2. Get snowflake ID:

grpcurl -plaintext -import-path $HOME/src/id-maker/internal/controller/rpc/proto -proto segment.proto localhost:50051 proto.Gid/GetSnowId
Copy the code

Local development

# Run MySQL
$ make compose-up

# Run app with migrations
$ make run
Copy the code

The project architecture

The project was developed using the Go-clean-template architecture template with the following directory structure:

Here is a brief description of each directory:

  • CMD: program entry
  • Config: indicates the configuration file
  • Docs: Generated project documents
  • Integration-test: indicates the integration test
  • Internal: business code
  • PKG: Packages of calls

To borrow two official images:

The overall hierarchy looks like this, with the models at the bottom, defining our table structure, and then the business logic layer in the middle, which provides the interface to the outermost API to call, and the outermost API is the tools and call portals.

The biggest benefit of this is decoupling. No matter how the outermost layer changes, the core code may not need to change at all as long as the corresponding interface is implemented in the business logic layer.

So, the calling relationship between them looks like this:

HTTP > usecase
       usecase > repository (Postgres)
       usecase < repository (Postgres)
HTTP < usecase
Copy the code

The above is all the content of this project. If you are interested, please leave a message for me. It would be great if you could give me a STAR.

Project Address: : ID-maker

Previous articles:

  • I’ve heard that 99% of Go programmers have been cheated by Defer
  • Test little sister asked me how to use gRPC, I directly threw this article to her
  • GRPC, critical praise
  • Run the grpcurl command to access the gRPC service
  • Three useful Go development tools are recommended

Recommended reading:

  • go-clean-template
  • hwholiday/gid
  • Leaf — Meituan-Dianping distributed ID generation system