Microservices practices based on Docker and Kubernetes

I. Microservitization

Microservices Architecture

Microservice is to split a single application into several small services, which are loosely coupled and highly cohesive. Each small service can be developed independently, independent of specific programming languages, and can also use different data storage technologies. Each service can be deployed independently with its own process. They communicate with each other through lightweight mechanisms (such as HTTP-based API interfaces), and all services collectively implement specific business functions.

There are two ways for the client to communicate with the server. The first way is for the client to communicate directly with each micro-service. This architecture has four disadvantages:

(1) Low efficiency due to multiple service requests;

(2) Exposed service interface;

(3) Interface protocols cannot be unified;

(4) The client code is complex and the server is difficult to upgrade.

The second way is to unify the proxy of each service by the API gateway and provide a unified interface protocol externally. This architecture has three advantages:

(1) Encapsulate service interface details to reduce communication times;

(2) Unified communication protocol, reduce client code coupling;

(3) Unified authentication, flow control and anti-attack;

Under this architecture, gateways can also become system bottlenecks.

Accordingly, these two architectures also bring about two ways of service registration discovery. The first is for clients to communicate with microservices by querying their addresses in the service registry, and the second is to add a unified API gateway to query. The former will increase the complexity of the client and cost a lot of development, while the second operation will be more concise. Therefore, we choose the second architecture in practice.

With the increase of the number of microservices, the invocation relationship between services is prone to coupling, or even cyclic invocation. The best solution is to layer the services, that is, asynchronously decouple the interdependent services through message queue and other technologies to reduce the dependency between services.

– Service layering –

Specific practice of microservices

1. Technical selection

In practice, our API Gateway uses OpenResty, which is based on Nginx and extends Lua support to build highly concurrent Web services. We use HTTP interface to realize client communication, data is basically encapsulated in JSON format, communication interface between services is also based on HTTP, and use message queue for asynchronous decoupling; For service registration discovery, we are using Consul; We chose Lua (extending the capabilities of the API Gateway), Node.js (for developing back-end services), and Java (for scenarios of intensive computing and communicating with big data) as the main development languages.

2. Specific implementation process

In practice, we use Lua to develop our own microservices framework — WebLua, which encapsulates the communication protocol between services and the methods and dependencies to access external resources (such as Mysql, Redis, etc.), while providing application slots. We can regard each APP as a function module, and each APP needs to be plugged into WebLua to run. WebLua can combine modules conveniently, so that one APP can run a micro-service, or multiple apps can provide services together. In this way, developers only need to focus on business APP development, which greatly improves the development efficiency. The right side of the figure is the specific code directory structure. Each APP can be divided into Action, Page and Data layers. The Action layer intercepts the request before and after processing, and can do some special processing, such as permission verification before the request. Page layer mainly parses and verifies the parameters of the request. The Data layer is responsible for specific business processing and provides Shell scripts to realize APP packaging, deployment and installation.

API gateway

An important role in the architecture is the API gateway, which is introduced below.

As can be seen from the above comparison figure, there is no API Gateway on the left, and many modules, such as Auth and Logging, need to be implemented by themselves, resulting in repeated construction of modules and intrusion into services, making it difficult to expand functions. The diagram on the right is the architecture diagram after using API Gateway. All common modules are implemented in API Gateway, which is easy to maintain, build in one place and benefit from everywhere. In this case, the API Gateway must be more demanding — its functionality must be easily extensible.

To implement such an API gateway, we extended the API gateway functionality through plug-ins, based on OpenResty, borrowing from Kong and Orange’s plug-in mechanism.

As you can see from the API Gateway architecture diagram above, the Gateway installs a number of plug-ins, each of which plays a role in one or more phases of the request. Plug-in configurations are updated on Consul and take effect in real time. Plug-in rules can be flexibly configured. In doing so, we give plugin developers more freedom to define their own formats.

Containerization

In the practice of micro-service landing, we chose Docker. The practice based on Docker will be introduced in detail below.

First, Calico was selected for the network component and Consul was selected for service registration discovery and configuration management. Consul-template monitors Consul configuration and service changes in real time.

The system is based on CentOS system image, install OpenResty, Nodejs, JDK, and obtain the environment image, and then install the micro-service framework on this basis, obtain Gorp image. Then on this basis, the specific application service is installed to obtain the application service image.

– Service registration discovery and configuration update process –

In API gateway, service registration is implemented through Ony-Agent and configuration updates are implemented through Ony-template. Consul Template updates three configurations, including: Services: specifies the service address of all microservices of the agent. Products: In brief, it is the mapping table of requests to microservices, as shown in the upper left. All requests have a unified specification. Prod can be obtained from Host, APP can be obtained from URI, and these two information can dynamically route requests to specific services. Naging-conf: Nginx configuration of the product.

The application service container is registered in the same way as the API gateway. First, the service registers with Consul through the Consul Agent running inside the container. Second, the service monitors configuration changes on Consul using Consul-Template and updates configuration files. An update to the OpenResty or WebNode configuration overwrites the corresponding configuration file and then restarts the corresponding service.

The figure above is a docker-based cluster architecture. It can be seen that Docker cluster consists of three nodes and the whole microservice is divided into three layers. The top layer is API Gateway, the middle layer is business layer, and the bottom layer is some basic microservices common to multiple products.

4. Kubernetes practice

Although microservices have many benefits, they also bring many problems, one of which is the complexity of operation and maintenance. In the past, o&M only needed to face one single application, but now it may face dozens or even hundreds of micro-services. In this case, we need Kubernetes to solve the problem. Kubernetes is an open source container choreography tool from Google that can be used to help manage containers.

Initially, we migrated containers to the Kubernetes cluster without making any changes, just using Pod to run all the services in the Kubernetes cluster. But as we went deeper into Kubernetes, we made some changes to microservices.

1. First of all, we will deploy the service in Deployment mode, which will ensure the existence of a certain copy of the service at all times and improve the stability of the service.

2. Second, we use Service, which can implement load balancing on behalf of Pod.

1. Kube-dns can resolve the Service name into a ClusterIP, and when the Service is not deleted and rebuilt, its ClusterIP remains unchanged, so that the cache of DNS resolution does not have invalid problem. Based on the characteristics of Kube-DNS and Service, we subsequently reformed the Service registration discovery system.

The figure above shows our current Service Deployment mode, Pod is created in Deployment mode, and Service is used for proxy.

Another problem we encountered in practice was configuration management.

(1) After microservitization, there are many and scattered configuration files;

(2) There are many unnecessary differences between different environments, such as database names;

(3) In many different environments, the same configuration items are exposed to test and operation;

(4) Rollback is troublesome without version control;

(5) Invalid input cannot be verified by Consul Web UI.

We made the following adjustments to address these problems:

(1) Unify the unnecessary differences between different environments;

(2) Template configuration files to expose only the difference part, and realize centralized configuration of different configuration files at the same time;

(3) Centralized management of product configuration based on Consul development configuration center; Verify the validity of input; Added version control to facilitate rollback.

– Configuration center flowchart –

As for log service, we integrated Fluent-bit into the application container and configured two input sources, TCP and tail. There are also two outputs, one is Elasticsearch, and all logs are uploaded to ES to display and query through Kibana. The other is log audit service. Some operation logs that need to be audited are sent to the log audit service for further analysis.

As the number of microservices increases, request links may be extended, making it very inconvenient for developers to track problems and troubleshoot performance bottlenecks. Therefore, we introduced Zipkin, which is mainly used for distributed link tracking, implemented a plug-in in the API Gateway for Span collection, and back-end services are implemented through open source middleware.

The diagram above shows the current overall architecture, with the K8S cluster at the bottom, where Kube-DNS is deployed, Consul for service registration discovery and configuration management, and our layered microservice architecture, with some auxiliary management systems on the right.

Five, the summary

The above is a promotion of the whole micro-service practice process based on Docker and Kubernetes. We have done 9 important things during the practice of micro-service, which has simplified the operation process and improved the work efficiency. Individual push design implements its own microservice framework, completes containerized deployment of microservices, develops API gateway, uses Kubernetes to arrange containers based on Consul’s Service registration and configuration management, revamps Service registration and discovery system based on Service and Kube-DNS, and builds its own configuration center. Optimized log service and realized Zipkin link tracking.

The original source: tuicool – > www.tuicool.com/articles/nQ…