With the progress of society and the development of technology, people have a more urgent demand for the efficient use of resources. In recent years, with the rapid development and maturity of the Internet and mobile Internet, the micro-servitization of large applications has also attracted the enthusiastic attention of enterprises, and the container cloud solution based on Kubernetes+Docker has also entered the public’s vision. Kepler Cloud is a microservice governance solution based on Kubernetes+Docker+Istio.

A, Microservices

1.1 Solve the problems after microservitization of large applications

Now all the major enterprises are talking about micro-services. Under the trend of micro-services, everyone in the technical circle will talk about micro-services and various solutions after micro-services.

1.2 What are we talking about when we talk about microservices?

There are many good reasons to use microservices architectures, but there is no free lunch, and while microservices offer many advantages, they also add complexity. Teams should proactively deal with this complexity if the application can benefit from microservices.

1.2.1 How to microservize

  • How to split microserver
  • Business API Rules
  • Data consistency assurance
  • Later scalability considerations

Of course, this is not the main discussion of this article, I will not talk about how to split microservices, each enterprise each application is not quite the same, suitable for their own solution is the best solution to split. We mainly solve some problems caused by microservitization.

1.2.2 Problems after microservitization

  • Environmental consistency
  • How to quickly allocate resources
  • How to deploy quickly
  • How do you do basic monitoring
  • Service registration and discovery
  • How to do load balancing

The above are the basic problems that need to be solved for microservitization of large applications. If virtual machines are used in the traditional way, the resource expenditure will be very large. So how can these problems be solved? Such as:

  • Traffic management
  • Service degradation
  • Authentication and authorization

Of course our ape friends have a solution to all of these problems.

1.3 Service governance

1.3.1 Java system

Assuming that we are Java architecture applications, it is easy to solve, for example, we can consider using the SpringCloud family bucket series. It can also be used separately:

  • Eureka
  • Hystrix
  • Zuul
  • Spring-cloud
  • Spring-boot
  • ZipKin

Java system can be very convenient to do the basic part of our micro-service, but it is still not very comfortable to solve the environment consistency, and if there are services of other languages will be difficult to integrate into.

Let’s look at how basic programming languages can be combined to solve basic problems.

1.3.2 Other systems

  • Consul
  • Kong
  • Go-kit
  • Jaeger/Zipkin

So let’s say we’re using Golang, let’s hold Golang again. Go is a language born for microservices. It’s not too convenient. Efficient development speed and quite good performance, simple and concise.

We can use these tools to build a decent microservices architecture.

  • Consul: Serves as a service discovery and configuration center
  • Kong: Serves as a service gateway
  • Jaeger: As link tracing
  • Go-kit: Develop components

The problem with this solution, however, is that it is too intrusive for the services, and each service requires a lot of embedded code, which is still a headache.

Docker & Kubernetes

Docker+ K8S based on the practice of building a platform.

2.1 Docker

Docker is a very powerful container.

  • Improved resource utilization
  • Environment consistency and portability
  • Fast expansion and scaling
  • Version control

With Docker, we found that there were more things to play with and more flexibility. Not only is resource utilization improved and environment consistency ensured, version control is also more convenient.

We used to use Jenkins to Build, and when we needed to roll back, we had to go through the Jenkins Build process again, which was very troublesome. If it’s a Java application, the build time will be very long.

With Docker, it’s easy to just pull down a version of the image and start it (if you have a local cache and start the version directly), which is very efficient.

(Photo credit network)

Since we use the Docker container as the basis of the service, we definitely need to choreograph the container, which would be terrible if we didn’t. For Docker container choreography, we have many choices: Docker Swarm, Apache Mesos, Kubernetes. Among these choreography tools, we choose Kubernetes, the king of service choreography.

2.1.1 Docker VS the VM

  • VM: It takes 1 minute to create a VM, 3 minutes to deploy the environment, and 2 minutes to deploy the code.
  • Docker: Start the container within 30 seconds.

2.2 according to choose Kubernetes

Let’s compare these three container choreography tools.

2.2.1 Apache Mesos

The goal of Mesos is to build an efficient and extensible system that can support a wide variety of frameworks, both current and future. This is one of the bigger issues today: frameworks like Hadoop and MPI are independent, which makes fine-grained sharing between frameworks impossible.

However, its base language is not Golang, it is not in our stack, and our maintenance costs for it will be high, so we ruled it out first.

2.2.2 Docker Swarm

Docker Swarm is a scheduling framework developed by Docker. One of the benefits of being developed by Docker itself is the use of the standard Docker API. Swarm’s architecture consists of two parts:

(Photo credit network)

Its use is not covered in detail here.

2.2.3 Kubernetes

Kubernetes is a Docker container marshaling system that uses the concepts of label and POD to divide containers into logical units. Pods are a collection of co-located containers that are deployed and scheduled together to form a service, which is a major difference between Kubernetes and the other two frameworks. This approach simplifies cluster management compared to similarity-based container scheduling, such as Swarm and Mesos.

Not only that, but it also provides a very rich API that allows us to manipulate it and do more things with it. In fact, another key point is to conform to our Golang technology stack, and supported by large factories.

The specific use of Kubernetes is not too much described here, there are a lot of information on the website for reference.

2.3 Kubernetes Kubernetes in

Kubernetes (K8S) is an open source platform for automating container operations such as deployment, scheduling, and scaling between node clusters.

  • Automates container deployment and replication
  • Expand or shrink the container at any time
  • Organize containers into groups and provide load balancing between containers
  • Easily upgrade to a new version of the application container
  • Provide container elasticity, replace containers if they fail, etc…

2.4 Kubernetes is not enough either

So far we have solved the following problems:

  • Docker: Environment consistency, fast deployment.
  • Kubernetes: service registration and discovery, load balancing, quick resource allocation.

And surveillance, of course. We’ll talk about that later. Let’s start with what do we do to solve some higher level problems?

What about service authentication, link tracing, log management, circuit breakers, traffic management, error injection, and so on, without intrusive code changes to the service?

One solution that has become very popular in recent years is Service Mesh.

Third, the Service Mesh

An infrastructure layer that handles communication between services for reliable request delivery in the complex service topology of cloud native applications.

  • A dedicated infrastructure layer that handles communication between services makes the process of request delivery more reliable through a complex topology.
  • As a set of lightweight high-performance network agents, deployed with applications, applications do not need to be aware of their existence.

Delivering requests reliably in cloud native applications can be complex, and this complexity is managed through a number of powerful techniques: link fuses, latency awareness, load balancing, service discovery, service renewal, and offline and cull.

There are many ServiceMesh frameworks on the market, and we chose Istio to stand at the forefront.

3.1 Istio

An open platform to connect, manage, and secure microservices.

  • Platform support: Kubernetes, Mesos, Cloud Foundry.
  • Observables :Metrics, logs, traces, dependency. Visualisation.
  • Service Identity & Security: Provides a verifiable Identity for Service and service-to-service authentication.
  • Traffic management: dynamically controls the communication between services, inbound/outbound routing, and fault injection.
  • Policy Execution: Check prerequisites and manage quotas between services.

3.2 Why do we choose Istio?

Because of the big factory support ~ in fact, the main or its concept is quite good.

Although it was only version 1.0, we started with version 0.6, testing the environment run, then 0.7.1 came out, we upgraded to version 0.7.1, then 0.8.0LTS came out, we started using version 0.8.0 and made an upgrade plan.

The latest version is currently up to 1.0.4, but we’re not going to upgrade. I want to wait until it gets to 1.2 before we start mass adoption. 0.8.0LTS is still ok in small scale at present.

3.3 Istio architecture

Let’s take a look at the architecture of Istio.

Istio control panel is mainly divided into three blocks: Pilot, Mixer and ISTIO-Auth.

  • Pilot: Serves primarily as a service discovery and routing rule and manages all envoys, which can be very resource-intensive.
  • Mixer: Mainly responsible for policy requests and quota management, as well as Tracing, all requests are reported to the Mixer.
  • Istio-auth: Upgrade traffic, authentication and other functions. Currently, we have not enabled this function. The need is not too great, because the cluster itself is isolated from the outside.

Each Pod is injected into a Sidecar, and all the traffic in the container is transferred to the Envoy via Iptables for processing.

Kubernetes & Istio

Istio can be deployed independently, but it is clearly a better choice when combined with Kuberntes. Small scale architecture based on Kubernetes. Some people worry about its performance, in fact, after the production test, tens of thousands of QPS is completely no problem.

4.1 Kubernetes Cluster

What does our K8S cluster look like under resource constraints?

4.4.1 Master cluster

  • Master Cluster:

    • ETCD, Kube-Apiserver, Kubelet, Docker, Kube-Proxy, Kube-Scheduler, kube-Controller – Manager, Calico, Keepalived, IPVS

4.1.2 Node Node

  • Node:

    • Kubelet, Kube-Proxy, Docker, Calico, IPVS.

(Photo credit network)

We call the Master API is managed by Keepalived, a failure of a Master, can ensure smooth floating to other Master API, does not affect the whole cluster operation.

Of course, we also configured two edge nodes.

4.1.3 Edge Node

  • Edge node
  • Traffic entrance

The main function of edge nodes is to provide the cluster with exposed service capabilities, so it does not need to be stable. Our IngressGateway is deployed on these two edge nodes and managed via Keeplived.

4.2 External Service Request Process

The outermost layer is DNS, which is parsed to Nginx, where Nginx forwards traffic to VIP of the cluster, and VIP to HAproxy of the cluster, which sends external traffic to our edge node Gateway.

Each VirtualService is bound to the Gateway for service load, traffic limiting, troubleshooting, routing rules, and canary deployment. The Service eventually goes to the Pods where the Service is located.

This was done without the detection of Mixer and strategy, and only the IStio-ingressGateway was used. This will change if you use all Istio components, but the main flow remains the same.

4.3 Logging

Log collection we adopt a low coupling, strong scalability, easy maintenance and upgrade solution.

  • The Filebeat node collects host logs.
  • Each Pods is injected into the Filebeat container to collect business logs.

Filebeat is deployed with the application container and the application does not need to know it exists, just specify the directory for logging input. The configuration used by Filebeat is read from ConfigMap. You only need to maintain the log collection rules.

Above, we can see the logs collected from Kibana.

4.4 Prometheus + Kubernetes

  • Monitoring system based on time series.
  • Seamlessly integrate infrastructure and application levels with Kubernetes.
  • A powerful key-value data model.
  • Big factory support.

4.4.1 Grafana

4.4.2 Alarm

At present, we support Wechat, KplCloud, Email and IM. All alarms can be configured on the platform to be sent to various places.

4.4.3 Overall architecture

The architecture is composed of peripheral services and basic services within the cluster, including:

  • Consul is used as a configuration center.
  • Prometheus+Grafana is used to monitor the K8s cluster.
  • Zipkin provides its own definition of link tracking.
  • ELK log collection, analysis, all logs in our cluster will be pushed here.
  • Gitlab code repository.
  • Jenkins is used to build code and package it into a Docker image and upload it to the repository.
  • Repository Image Repository.

Clusters are:

  • HAProxy+ KeeprLived forwards traffic.
  • The network is Calico, which has beta support for kube-Proxy’s IPVS proxy mode. If Calico detects that Kube-Proxy is running in this mode, it automatically activates Calico ipvS support, so we enabled ipvs.
  • The DNS inside the cluster is CoreDNS.
  • We deployed two gateways, using Istio’s IngressGateway and TraefikIngress as backup. Once IngressGateway is down we can quickly switch to TraefikIngress.
  • Above are the related components of Istio.
  • Finally, our APP service.
  • The cluster collects logs using Filebeat and sends them to external ES.
  • Internal cluster monitoring is as follows:
    • State-metrics are used to automate scaling of monitoring components
    • Mail&Wechat self-developed alarm service
    • Prometheus+Grafana+AlertManager Monitors services and related basic components in a cluster
    • The InfluxDB+Heapster flow database stores monitoring information of all services

4.5 How do YOU deploy applications with Kubernetes?

4.5.1 Research and development package into mirror, warehouse and management version

  • Learning Docker.
  • Learn to configure the warehouse, manual packaging upload trouble.
  • Learn k8S related knowledge.

4.5.2 Jenkins is responsible for packaging, mirroring and version update

  • Operation and maintenance work has increased a lot. Applications need to be configured and services need to be changed.
  • You need to manage a bunch of YAML files.

Is there a foolproof solution that doesn’t require much technology to learn and is easy to use?

Fifth, Kplcloud platform

5.1 Kepler Cloud Platform

Kepler cloud Platform is a lightweight PaaS platform.

  • Provide a controllable management platform for microservitization projects.
  • Each service is independently deployed, maintained, and extended.
  • Simplify the process, no longer need cumbersome application process, maximum automatic processing.
  • Realize fast release, independent monitoring and configuration of microservices.
  • Realize zero-intrusive service discovery, service gateway, link tracking and other functions of micro-service projects.
  • Provides a configuration center to centrally manage configurations.
  • R&d, production, testing, operations, and even the boss can publish their own apps.

5.2 Service deployment on the Kepler platform

To reduce learning costs and deployment difficulties, it is easy to deploy applications on the Kepler platform by adding a Dockerfile.

Dockerfile reference:

The above is the normal mode, Jenkins code Build and Docker Build.

This is a relatively free way to deploy and can be customized to your needs, with a learning cost of course.

5.2.1 Why not automatically generate Dockerfile?

It is possible to generate dockerfiles automatically, but the requirements of each service may be different. Some need to add files, some need to add parameters during Build, and so on. We can’t expect all projects to be the same, that would hinder the development of technology. So the next best thing is we give a template, and the developers adapt it to their own needs.

5.3 Tool Integration

  • The Kepler cloud platform integrates apis such as GitLab, Jenkins, Repo, K8S, Istio, Promtheus, Email, and WeChat.
  • Manage the entire life cycle of the service.
  • Provides service management, creation, publishing, versioning, monitoring, alarm, logging and some peripheral additional functions, message center, configuration center, also can log into the container, service offline and so on.
  • You can adjust the service mode and service type, expand and scale the service by one click, roll back service API management, and storage management.

5.4 Release Process

Users submit their Dockerfiles and codes to Gitlab, and then fill in some parameters to create their own applications on the Kepler cloud platform.

After the application is created, Jenkins will create a Job, pull down the code and execute the Docker build (if multi-stage build is not selected, go Build or MVN will be executed first), and then push the packaged Docker image to the image warehouse. Finally, call the platform API or call the K8S notification to pull the latest version.

Users only need to manage their applications on the Kepler cloud platform, and everything else is automated.

5.5 Start by creating a service

We introduce the platform by creating a service.

Platform main interface:

Click Create Service to enter the create page.

Fill in basic information:

Fill in the details:

Basic information Golang is used as an example. The parameters you need to fill in are slightly different for other languages.

If you choose to provide external services, you will enter the third step. The third step is to fill in routing rules. If there is no special requirement, you can directly submit the route by default.

5.5.1 Service Details

Build Upgrade application version:

Invoke the service pattern, which can be adjusted between normal and service grid.

Service Ability to provide external services:

Capacity Expansion CPU and memory adjustment:

Adjust the number of pods started:

Web version of the terminal:

5.5.2 Scheduled Task

5.5.3 Persistent Storage

The administrator creates StorageClass and PersistentVolumeClaim, and the user only needs to select the relevant PVC to bind to his service.

The storage uses NFS.

5.5.4 Tracing

5.5.5 Consul

Consul is used as a configuration hub, and we provide Golang’s client.

$ go get github.com/lattecake/consul-kv-client
Copy the code

It automatically synchronizes consul’s directory configuration to memory, so you just need to get the configuration directly from memory.

5.5.6 Repository

  • Making: github.com/kplcloud/kp…
  • Document: docs.nsini.com
  • Demo: kplcloud.nsini.com

Author: Wang Cong

Starter: Long of skill