This article is compiled from the content shared by Song Lei, a lecturer at KubeSphere 2020 Meetup.
Hello, everyone. My name is Song Lei. I work in a technology subsidiary of the carrier, mainly doing big data business. I was mainly responsible for the construction and operation of IaaS layer and PaaS layer of the company, which involved two levels. Because Kubernetes is a very comprehensive technical system, it’s not that we deployed a cluster can put the business go up and then out of the box, involves many aspects, such as servers, network, storage, and a series of support tool chain, we can really go into production, so our team is suit to do it.
Business types and practice architecture
At present, we have three types of business: 1. Interface service, which accounts for a large proportion of capacity 2. The external application system mainly deals with smart government, smart ecology, smart city, smart tourism and other businesses
For all three types of business, the overall TPS peaked at around 2500 and averaged around 1500.
Our overall cluster size: All of our clusters are deployed on physical servers. The production cluster has 50 physical nodes, the test cluster has 20-30 nodes, and the overall Kubernetes cluster size is less than 100 physical nodes.
The picture above is our Kubernetes practice.
IaaS: The network at the physical layer of the data center is based on the SDN and VXLAN architecture. Therefore, the selection of network plug-ins is considered.
We have a relatively large Ceph cluster with about 50 physical nodes. The interconnection layer not only runs these services of KubeSphere, but also runs some OpenStack virtual machines. We did some data layering on Ceph, flash drives (for cluster metadata) and SATA disks (for real data), some data heat layering, and then a lot of docking tool chains around the container cluster centered on KubeSphere. Some of the toolchains are not containerized, but outside the chain, such as CMDB configuration management, Prometheus monitoring, Skywalking mainly do microservice full link monitoring, and some log collection analysis, mainly ELK toolchains. Also outside of the KubeSphere cluster, the DevOps layer is based on Jenkins pipeline.
As for traffic entry, we have an overall Nginx cluster in the Internet area because all of our business types are Internet oriented, which is mainly responsible for routing distribution of business and centralized control of traffic.
Selection and practice of storage and network
network
As mentioned above, our physical network is a tenant of the large layer 2 of SDN and VXLAN. Therefore, there are mainly two types of network plug-ins for KubeSphere — Calico and Flannel.
Flannel itself is based on VXLAN. If Flannel is selected, it is equivalent to the VXLAN of our two layers — physical network and Kubernetes network. This involves packet unpacking at two layers, which has a certain impact on performance. Therefore, we finally chose Calico, a pure three-tier BGP network, and then made a plug-in for the network.
storage
At present, we mainly connect with Ceph block storage and serve some stateful services. For example, we will do some helm mirroring, mainly Zookeeper, Redis and MySQL. But these stateful services are primarily used in the test cluster for development testers. Production environments are primarily stateless services, such as Java microservice applications in a distributed framework, as well as Python and GO. Go is mainly used for blockchain, because now the combination of blockchain and K8s is a very necessary business type.
However, RBD block storage has limitations. Many of our businesses require multiple PODS or multiple containers to jointly read and write a certain piece of storage, but block storage cannot be implemented. In the future, we will also have object storage and network storage (NFS) interconnection.
DevOps and log collection practices
CI/CD, with Jenkins at the bottom, is not integrated into KubeSphere, because we previously had a Jenkins Master and Slave architecture platform, pipeline based, mirroring directly to Kubernetes cluster, Make automated CDS.
Log collection is relatively troublesome. The current ELK tool chain mainly collects three types of logs: host log, Pod service log and Kubernetes component related log. Both host and Kubernetes component logs are collected based on the host.
There are two methods to collect Pod service logs:
- Add a Sidecar container to the Pod
- Create two services in a business container. The foreground service is a Java microservice, and the background service is a collection of Filebeat agent, and then the collection agent directly into the image to run
ELK’s tool chain is a mature tool chain, as shown in the figure above.
Gray released
We do grayscale publishing in two forms.
- Iterate for small releases
Based on the weight, the grayscale publishing is done by the copy characteristic of Kubernetes controller. There are multiple copies in a business, first gray release one or two, there is no problem to continue gray release, if there is a problem, back. This is the norm.
- Iterate for large releases
Use the service gray scale. The header of the HTTP request for the user and the user ID in the body are distributed to the different versions through the Nginx and Lua scripts.
Service governance
We may have some future needs in this area of service governance, and there is not a particularly good practice right now.
Currently, our microservice governance is based on auxiliary means, such as full link monitoring, log indicators, to do microservice traffic control and monopoly. In the future, we want to explore on the service grid, and put the monitoring and control of the traffic on the platform layer. The development only needs to focus on the business logic, and there is no better landing scheme at present.
This article is published by OpenWrite!