Golang deep application in twelve zan

We are twelve Likes, a small team dedicated to helping e-commerce sellers get into mini apps. Our home page is www.12zan.cn/. In practice, we use a large number of small tools written by Golang. Almost every tool has a very short code, usually about 200 lines to complete an independent function and play a very important role. For example, the proxy server, which is a little more than 500 lines of code, is our core pillar, and QPS is almost as good as NGINx in pressure testing.

Docker-based infrastructure

As an infrastructure, LET me introduce our machine architecture.

Our entire business is built on Ali Cloud, with 5 servers, each pair of which has an independent external IP address and is also in the same Intranet. On each machine runs a daemon we wrote ourselves in Golang, which listens for and executes commands for business restarts, new domain names, and the like (all of which are passed to Docker). At the same time, there is a Consul process on each machine, and these Consuls are joined together. In addition, we ran an Nginx with Docker for port 80 on each machine and an HTTP proxy written by Golang itself. Nginx does basic logging and then dumps the request to the HTTP proxy, which goes into Consul to find out which port on which IP address it should forward the request to.

400 lines of Golang code to write HTTP Proxy

On the first day of the architecture selection, we decided that we would be servizing, using a lot of HTTP interfaces to provide services, using our own HTTP proxy to distribute requests, and adding our own business logic such as API permission verification logic. We limit all businesses to *.app.12zan.net, so when we go to a chat service, for example, the interface requested is chat.app.12zan.net, and when we go to a comment service, the interface requested is comment.app.12zan.net.

The first thing I did after deciding on the domain name was to write a very simple HTTP proxy server based on Consul with Golang. Thanks to Golang’s excellent HTTP library, we did it all in a few hundred lines of code. Every time an HTTP request comes in, the proxy server will go to Consul based on the HTTP_HOST field in the HTTP request to check which backend has registered the service with this domain name. Then, according to the specified algorithm, the proxy server will fetch a backend and proxy the HTTP request. Each specific business may run on any one of our five docker machines, or it may run on multiple Docker instances. So there is a mechanism for choosing which actual Docker instance to service the request. We now support random selection, hash selection by client IP address, hash selection by URL, and load selection.

Service registration for WEB services

We made our own Docker image based on PHP + Laralel and NodeJS + Koajs scenarios. In addition to running PHP + Nginx and NodeJs-built Web services, this image also contains a Golang-written Consul client. In the Docker container, the client starts up with either PHP + Nginx or nodejs web service, and then registers its own service with the host’s Consul process, notifying it that the application is being serviced on the consul IP port. If there is a request to **.app.12zan.net before, you can forward it to me; It also reports the number of processes, CPU usage, and memory usage every second. It’s just as simple, just a few hundred lines of Golang code, to hack out consul client. Why golang? Consul is naturally in golang’s camp. Secondly, we have a wide variety of Docker containers, so consul is directly cross-platform compiled on Mac and runs on Linux64. Regardless of whether the Docker container is python-based, Ruby-based, or NodeJS, simply copying the binary into it will work correctly, unlike other languages that need to resolve dependencies.

We also developed a Web Console interface where we can register our app and add new instances to it. To register an app, we need to specify the address of the repository (by the way, our code management is gogs written with Golang), specify the domain name of the external service, and specify whether it is a NodeJS app or a PHP +laravel app. After adding an application, you can create an instance under the application and have the system run the instance on the specified IP address. The process of instance running is actually sending a notification to a machine to perform the process of starting a Docker instance. Docker starts with some environment variables, such as the current internal IP, the port docker listens to, and the domain name used to provide services externally.

Logging and storage

One of the problems with this architecture is that the backend may be running on any machine. Today it may be A, tomorrow it may be B, so if I save the file on A, will I fail when I ask B to provide the service? So we came up with this solution (also because of poverty…). , we moved all the files to ali Cloud OSS service. In order to save files uploaded by users or generated by the system to oss, whether Nodejs applications, PHP applications or python applications, we wrote an ossUploader, compiled executable file to publish, just need to execute it. Pass in the local path and the target path on OSS to ensure that you upload to OSS is complete, no need to mull over the OSS SDK in nodeJS, PHP, Python, Ruby, Java, etc.

Logs are handled the same way. The contents of all log files are monitored by gtail, a golang-written tool (just like the tail -f command in Linux), and any new content is moved to oss for storage by Gtail. Of course, executables are also published.

With this implementation, our actual business can really move between five machines.

News broadcast