The author | ping Ali server-side development technology experts

takeaway: container service Kubernetes is currently hot cloud native infrastructure, the author in the past year launched a number of users speed growth of application: the application within one month of live users from zero to forty million, the number of users from zero to one hundred million fission growth, expansion and fully enjoy the container service quick and easy operation and high availability features. The author uses the container service Kubernetes cluster to fully cloud the company’s systems for over a year. This article documents the pitting and optimization records.

Create the cluster

When creating a cluster, make a good plan and select optimized cluster configurations, which can greatly reduce o&M work in the later period. Some configurations of clusters cannot be modified after being created or are extremely troublesome to modify.

Cluster planning

  • Network planning:

    • Network type: Flannel or Terway

Terway is a self-developed network plug-in of Ali Cloud container service. It is fully compatible with Flannel in functions. If Flannel is conservative, it is still used

  • Pod CIDR network

The default large network segment is 16. The valid network segment or its subnet is 10.0.0.0/8, 172.16-31.0.0/12-16, 192.168.0.0/16

  • Service CIDR

    • The default network segment is 20. You can set this parameter to 10.0.0.0/16-24, 172.16-31.0.0/16-24, and 192.168.0.0/16-24
    • Network segments must be unique and cannot be modified after being created.
    • Multiple switches in multiple areas.
  • Access ApiServer from the public network

    • For clusters with high security requirements such as online, you can choose not to expose Apiserver and only have private NETWORK SLB, but in this way, cloud effect publishing is not possible.
    • After a cluster is established, access control is established for the SLB to restrict the ACCESS of the SLB to the cloud effect only.

Note: Every security vulnerability of K8s is almost related to ApiServer. For online K8s cluster, patch should be updated in time, or ApiServer on public network should not be opened, and strict security group and access control should be used.

  • The security group

    • Set security group limited access range for master and worker machines.
  • Master Machine Planning

For high availability, three nodes are generally used. The Master selection rules are as follows:

Number of nodes master 1 to 5 4C8G6 to 20 4C16G21 to 100 8C32G100 to 200 16C64G

It is recommended that the storage of the master machine be 50-100 GB SSD with high performance. Because ETCD is running, the operating system takes up no more than 8 GB.

  • Worker machine planning

    • Ali Cloud is the first to promote Shenlong machine. In areas where there is no Shenlong machine, ECS with high configuration is selected. The configuration specifications are multiplied by a certain number according to the POD specifications deployed. Set a fixed request/limit for pod at deployment time.
    • The machine configuration we choose:
      • 32C64G ECS
      • Storage. System disk: 100 GB SSD; data disk: 400 GB efficient cloud disk
      • Operating system: centos 7.4 64-bit

Cluster establishment and configuration

Set when setting up the cluster:

  • Establish a cluster through the console, Ali Cloud container service provides a very simple one-click cluster deployment function, through the wizard to complete the establishment of K8S cluster;
  • Mount /var/lib/docker to the data disk.
  • Set reasonable Pod network CIDR, Service CIDR IP network segment;
  • Set reasonable security policies and determine whether to expose Apiserver (for direct cloud release, public network exposure and strict access control are required);
  • Ingress select security, can use the Intranet, if the need for public network, can be easily established in the console, while doing a good job of access control;
  • Kube-proxy mode, because iptables mode locks iptables when updating a rule, it is recommended to use IPVS mode.
  • The number of PODS on a node is 128 by default, which is too large for one node. You are advised to change it to 64.
  • Node service port access (NodePort,SLB), can be expanded, and the default is generally sufficient.

Cluster configuration modification:

  • Add existing nodes for cluster expansion (refer to the previous node configuration, use /var/lib/docker to mount data disks)
  • Master machine upgrade:
  • Change or remove worker node:
    • kubectl drain –ignore-daemonsets {node.name}
    • kubectl delete node {node.name}
    • ECS rising and changing
    • Add an existing node to the cluster
  • Namespace:
    • You can create namespaces based on application groups. You can set resource quotas and limits for namespaces of application groups that need to be restricted.
  • Authorization:
    • How can a sub-account grant RBAC authorization to another sub-account
    • Set permissions by application personnel through fortress machine

Deployment Settings

Stateless deployment

Using stateless Deployment, refer to this article for batch Deployment. Optimization Settings template:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '34'
# tag, map service
  labels:
    app: {app_name}-aone
  name: {app_name}-aone-1
  namespace: {app_name}
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: {app_name}-aone
# Batch restart update policy
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: {app_name}-aone
    spec:
      containers:
       The environment variable increases the time zone
        - env:
            - name: TZ
              value: Asia/Shanghai
        - image: >-
            registry-vpc.cn-north-2-gov-1.aliyuncs.com/{namespace}/{app_name}:20190820190005
          imagePullPolicy: Always
          Perform graceful offline removal service registration before startup
          lifecycle:
            preStop:
              exec:
                command:
                  - sudo
                  - '-u'
                  - admin
                  - /home/{user_name}/{app_name}/bin/appctl.sh
                  - {app_name}
                  - stop
          # Survival check, highly recommended
          livenessProbe:
            failureThreshold: 10
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 5900
            timeoutSeconds: 1
          name: {app_name}-aone
          # Readiness check, strongly recommended
          readinessProbe:
            failureThreshold: 10
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 5900
            timeoutSeconds: 1
          # Resource limit, this must be set properly
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              cpu: '4'
              memory: 8Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          /home/{user_name}/logs = /home/{user_name}/logs = /home/{user_name}/logs
          volumeMounts:
            - mountPath: /home/{user_name}/logs
              name: volume-1553755418538
      dnsPolicy: ClusterFirst
      The private mirror repository key is obtained from the private field
      imagePullSecrets:
        - name: {app_name}-987
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      /var/lib/docker/logs/directory
      volumes:
        - hostPath:
            path: /var/lib/docker/logs/{app_name}
            type: ' '
          name: volume-1553755418538Copy the code

Service setting

Because the Cloud Controller Manager of the container service will delete the SLB associated with the service at the same time, in order to prevent the SLB fault from being deleted by mistake when the service configuration is modified, which will lead to the fault that the domain name and security configurations need to be modified, It is strongly recommended that service and SLB be decouple. Service uses NodePort and SLB establishes a back-end server to point to cluster nodes. If real IP addresses need to be transparently transmitted and load balancing is considered, certain configuration rules and methods need to be observed.

NodePort:

ApiVersion: v1 kind: Service metadata: name: {app_name} namespace: {namespaces} spec: clusterIP: 10.1.50.65The ## policy is related to whether real IP addresses are passed through
  externalTrafficPolicy: Cluster
  ports:
    - name:  {app_name}-80-7001
      nodePort: 32653
      port: 80
      protocol: TCP
      targetPort: 7001
    - name:  {app_name}-5908-5908
      nodePort: 30835
      port: 5108
      protocol: TCP
      targetPort: 5108
  selector:
    app:  {app_name}
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}Copy the code

Then, on the load balancing management page, select the backend server pointing to the worker machine of the cluster and set port to the port of the above service: 32653 to complete the configuration. In this way, when the cluster service is modified or deleted and rebuilt, SLB will not be deleted by the CCM of the cluster and will not involve the configuration modification of domain name and security. At the same time, you can set some policies, such as batch cutting when you need to upgrade and modify the service configuration.

conclusion

Although ali cloud container service console is a new product on the cloud, it provides extremely simple one-click deployment function, as well as simple console management. In the past year, THE author has witnessed the transformation process of Ali Cloud container service console from simple to powerful. Despite repeated stumps, the seriousness, responsibility and excellent service attitude of ali Cloud container service students are admirable.

The container service management console also needs to consider more actual operation and maintenance requirements, and closely integrate existing cloud products, such as cloud effect, EDAS, cloud monitoring, log services, etc., to provide better services on an application basis.

The original link

This article is the original content of the cloud habitat community, shall not be reproduced without permission.