The author | ping Ali server-side development technology experts

Takeaway: container service Kubernetes is currently hot cloud native infrastructure, the author in the past year launched a number of users rapid growth of application: the application within one month of live users from zero to forty million, the number of users from zero to one hundred million fission growth, expansion and fully enjoy the container service quick and easy operation and high availability features. The author uses the container service Kubernetes cluster to fully cloud the company’s systems for over a year. This article documents the pitting and optimization records.

Pay attention to the “Alibaba Cloud original” public account, reply to the keyword “information”, you can get the PPT collection of meetup activities in 2019 and the most comprehensive KNOWLEDGE map of K8s.

Create the cluster

When creating a cluster, make a good plan and select optimized cluster configurations, which can greatly reduce o&M work in the later period. Some configurations of clusters cannot be modified after being created or are extremely troublesome to modify.

Cluster planning

• Network planning:
  • Network type: Flannel or Terway

Terway is a self-developed network plug-in of Ali Cloud container service. It is fully compatible with Flannel in functions. If Flannel is conservative, it is still used

• Pod CIDR network

The default large network segment is 16. The valid network segment or its subnet is 10.0.0.0/8, 172.16-31.0.0/12-16, 192.168.0.0/16

• Service CIDR

  • The default network segment is 20. You can set this parameter to 10.0.0.0/16-24, 172.16-31.0.0/16-24, and 192.168.0.0/16-24
  • Network segments must be unique and cannot be modified after being created.
  • Multiple switches in multiple areas.

• Access ApiServer from the public network

  • For clusters with high security requirements such as online, you can choose not to expose Apiserver and only have private NETWORK SLB, but in this way, cloud effect publishing is not possible.
  • After a cluster is established, access control is established for the SLB to restrict the ACCESS of the SLB to the cloud effect only.

Note: Every security vulnerability of K8s is almost related to ApiServer. For online K8s cluster, patch should be updated in time, or ApiServer on public network should not be opened, and strict security group and access control should be used.

• The security group
  • Set security group limited access range for master and worker machines.

• Master Machine Planning

For high availability, three nodes are generally used. The Master selection rules are as follows:

It is recommended that the storage of the master machine be 50-100 GB SSD with high performance. Because ETCD is running, the operating system takes up no more than 8 GB.

• Worker machine planning
  • Ali Cloud is the first to promote Shenlong machine. In areas where there is no Shenlong machine, ECS with high configuration is selected. The configuration specifications are multiplied by a certain number according to the POD specifications deployed. Set a fixed request/limit for pod at deployment time.
  • The machine configuration we choose:
    • 32C64G ECS
    • Storage. System disk: 100 GB SSD; data disk: 400 GB efficient cloud disk
    • Operating system: centos 7.4 64-bit

Cluster establishment and configuration

Set when setting up the cluster:

• Establish a cluster through the console, Ali Cloud container service provides a very simple one-click cluster deployment function, through the wizard to complete the establishment of K8S cluster;
• Mount /var/lib/docker to the data disk.
• Set reasonable Pod network CIDR, Service CIDR IP network segment;
• Set reasonable security policies and determine whether to expose Apiserver (for direct cloud release, public network exposure and strict access control are required);
• Ingress select security, can use the Intranet, if the need for public network, can be easily established in the console, while doing a good job of access control;
• Kube-proxy mode, because iptables mode locks iptables when updating a rule, it is recommended to use IPVS mode.
• The number of PODS on a node is 128 by default, which is too large for one node. You are advised to change it to 64.
• Node service port access (NodePort,SLB), can be expanded, and the default is generally sufficient.

Cluster configuration modification:

• Add existing nodes for cluster expansion (refer to the previous node configuration, use /var/lib/docker to mount data disks)
• Master machine upgrade:
• Change or remove worker node:
  • kubectl drain –ignore-daemonsets {node.name}
  • kubectl delete node {node.name}
  • ECS rising and changing
  • Add an existing node to the cluster
• Namespace:
  • You can create namespaces based on application groups. You can set resource quotas and limits for namespaces of application groups that need to be restricted.
• Authorization:
  • How can a sub-account grant RBAC authorization to another sub-account
  • Set permissions by application personnel through fortress machine

Deployment Settings

Stateless deployment

Using stateless Deployment, refer to this article for batch Deployment. Optimization Settings template:

apiVersion: apps/v1beta2 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '34' # label, mapping service labels: app: {app_name}-aone name: {app_name}-aone-1 Namespace: {app_name} spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: {app_name}-aone # Batch restart update strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% Type: rollingUpdate template: Metadata: labels: app: {app_name}-aone spec: containers: # Environment variables add time zone - env: - name: TZ value: Asia/ shanghai-image: >- registry-vpc.cn-north-2-gov-1.aliyuncs.com/{namespace}/{app_name}:20190820190005 imagePullPolicy: Lifecycle: preStop: exec: command: -sudo - '-u' -admin - /home/{user_name}/{app_name}/bin/appctl.sh - {app_name} -stop # livenessProbe is strongly recommended: failureThreshold: 10 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 5900 timeoutSeconds: 1 name: {app_name}-aone # readinessProbe: failureThreshold: 10 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 5900 timeoutSeconds: 1 # periodSeconds cpu: '4' memory: 8Gi requests: cpu: '4' memory: 8Gi terminationMessagePath: /dev/termination-log terminationMessagePolicy: /home/{user_name}/logs/volumeMounts: - mountPath: /home/{user_name}/logs/volumeMounts: - mountPath: /home/{user_name}/logs Name: volume-1553755418538 dnsPolicy: ClusterFirst ## - name: {app_name}-987 restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 # deposit log directory, mapped to the nodes of the/var/lib/docker/logs data disc volumes: - hostPath: path: /var/lib/docker/logs/{app_name} type: '' name: volume-1553755418538Copy the code

Service setting

Because the Cloud Controller Manager of the container service will delete the SLB associated with the service at the same time, in order to prevent the SLB fault from being deleted by mistake when the service configuration is modified, which will lead to the fault that the domain name and security configurations need to be modified, It is strongly recommended that service and SLB be decouple. Service uses NodePort and SLB establishes a back-end server to point to cluster nodes. If real IP addresses need to be transparently transmitted and load balancing is considered, certain configuration rules and methods need to be observed.

NodePort:

apiVersion: v1 kind: Service metadata: name: {app_name} namespace: {namespaces} spec: clusterIP: ExternalTrafficPolicy: Cluster ports: - name: {app_name}-80-7001 nodePort: 32653 port: 80 protocol: TCP targetPort: 7001 - name: {app_name}-5908-5908 nodePort: 30835 port: 5108 protocol: TCP targetPort: 5108 selector: app: {app_name} sessionAffinity: None type: NodePort status: loadBalancer: {}Copy the code

Then, on the load balancing management page, select the backend server pointing to the worker machine of the cluster and set port to the port of the above service: 32653 to complete the configuration. In this way, when the cluster service is modified or deleted and rebuilt, SLB will not be deleted by the CCM of the cluster and will not involve the configuration modification of domain name and security. At the same time, you can set some policies, such as batch cutting when you need to upgrade and modify the service configuration.

conclusion

Although ali cloud container service console is a new product on the cloud, it provides extremely simple one-click deployment function, as well as simple console management. In the past year, THE author has witnessed the transformation process of Ali Cloud container service console from simple to powerful. Despite repeated stumps, the seriousness, responsibility and excellent service attitude of ali Cloud container service students are admirable.

The container service management console also needs to consider more actual operation and maintenance requirements, and closely integrate existing cloud products, such as cloud effect, EDAS, cloud monitoring, log services, etc., to provide better services on an application basis.