Kubernetes Pod Vertical Automatic Scaling (VPA)

The VPA profile

VPA stands for Vertical Pod Autoscaler. It automatically sets CPU and memory requests based on container resource utilization, allowing appropriate scheduling on nodes to provide appropriate resources for each Pod. It can either shrink containers that overrequest resources, or increase the capacity of under-resourced containers at any time based on their usage. PS: VPA does not change the resource limits value of Pod.

Without further ado, let’s look at the VPA workflowLet’s get into the game.

The deployment of the metrics – server

1. Download the deployment manifest file

Wget HTTP: / / https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yamlCopy the code

2. Modify the components. Yaml file

• Changed the mirror address to gcr. IO for my own warehouse
• Changed the metrics-server startup parameter args, otherwise an error will be reportedunable to fully scrape metrics from source kubelet_summary...

- name: metrics-server
        image: scofield/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - /metrics-server
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP
Copy the code

3. Perform the deployment

kubectl  apply -f components.yaml
Copy the code

4, validation,

[root@k8s-node001 metrics-server]# kubectl get po -n kube-systemNAME READY STATUS RESTARTS AGE metrics- server-7947CB98b6-xw6b8 1/1 Running 0 10m RESTARTS for top information are regarded as successful [root@k8s-node001 metrics-server]# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-node001   618m         7%       4796Mi           15%
k8s-node003   551m         6%       5522Mi           17%
k8s-node004   308m         3%       5830Mi           18%
k8s-node005   526m         6%       5997Mi           38%
k8s-node002   591m         7%       5306Mi           33%
Copy the code

The deployment of vertical – pod – autoscaler

Clone the AutoScaler project

git clone https://github.com/kubernetes/autoscaler.git
Copy the code

2. Modify the deployment file to make the GCR warehouse my own

Admission - controller - deployment. Yaml us. GCR. IO/k8s - artifacts - prod/autoscaling/vpa - admission - controller: 0.8.0 instead Scofield/vpa - admission - controller: 0.8.0 recommender - deployment. Yaml Us. GCR. IO/k8s - artifacts - prod/autoscaling/vpa - recommender: 0.8.0 to image: Scofield/VPA-recommender :0.8.0 Updater-deployment. Yaml US.gcr. IO/k8S -artifacts- Prod/Autoscaling/VPA-artifacts :0.8.0 Scofield/vpa - updater: 0.8.0Copy the code

3, deployment,

[root@k8s-node001 vertical-pod-autoscaler]# cd autoscaler/vertical-pod-autoscaler
[root@k8s-node001 vertical-pod-autoscaler]# ./hack/vpa-up.sh
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated inV1.16 +, unavailableinV1.22 +; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created clusterrole.rbac.authorization.k8s.io/system:metrics-reader created clusterrole.rbac.authorization.k8s.io/system:vpa-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:evictioner created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created serviceaccount/vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created serviceaccount/vpa-updater created  deployment.apps/vpa-updater created serviceaccount/vpa-recommender created deployment.apps/vpa-recommender created Generating certsfor the VPA Admission Controller in/tmp/vpa-certs. Generating RSA private key, 2048 bit long modulus (2 primes) ............................................................................ +++++ .+++++ e is 65537 (0x010001) Generating RSA private key, 2048 bit long modulus (2 primes) ............ + + + + +... +++++ e is 65537 (0x010001) Signature ok subject=CN = vpa-webhook.kube-system.svc Getting CA Private Key Uploading certs  to the cluster. secret/vpa-tls-certs created Deleting /tmp/vpa-certs. deployment.apps/vpa-admission-controller created service/vpa-webhook createdCopy the code

4. Viewing the results, you can see that both metrics- Server and VPA are running properly

[root@k8s-node001 autoscaler-master]# kubectl get po -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
metrics-server-7947cb98b6-xw6b8             1/1     Running   0          46m
vpa-admission-controller-7d87559549-g77h9   1/1     Running   0          10m
vpa-recommender-84bf7fb9db-65669            1/1     Running   0          10m
vpa-updater-79cc46c7bb-5p889                1/1     Running   0          10m
Copy the code

Example 1 updateMode: “Off”

1. First we deploy an nginx service to Namespace: VPA

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: vpa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources:
          requests:
            cpu: 100m
            memory: 250Mi
Copy the code

Look at the result. 2 pods are running correctly

[root@k8s-node001 examples]# kubectl get po -n vpa
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6884b849f7-fswx5   1/1     Running   0          5m54s
pod/nginx-6884b849f7-wz6b8   1/1     Running   0          5m54s
Copy the code

2. For cheap pressure, we create a NodePort service

[root@k8s-node001 examples]# cat nginx-vpa-ingress.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: vpa
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

[root@k8s-node001 examples]# kubectl get svc -n vpaNAME TYPE cluster-ip external-ip PORT(S) AGE nginx NodePort 10.97.250.131 < None > 80:32621/TCP 55s [root@k8s-node001 examples]# curl -i 192.168.100.185:32621HTTP / 1.1 200 OKCopy the code

UpdateMode: “Off” mode is used to obtain the resource, but Pod is not updated

[root@k8s-node001 examples]# cat nginx-vpa-demo.yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa
  namespace: vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: nginx
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: "nginx"
      minAllowed:
        cpu: "250m"
        memory: "100Mi"
      maxAllowed:
        cpu: "2000m"
        memory: "2048Mi"
Copy the code

4. View the deployment result

[root@k8s-node001 examples]# kubectl get vpa -n vpa
NAME        AGE
nginx-vpa   2m34s
Copy the code

5. Use describe to check VPA details, focusing on Container Recommendations

[root@k8s-node001 examples]# kubectl describe vpa nginx-vpa -n vpaName: nginx-vpa Namespace: vpa .... Omit 10000 words haha...... Update Policy: Update Mode: Off Status: Conditions: Last Transition Time: 2020-09-28T04:04:25Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 250m Memory: 262144k Uncapped Target: Cpu: 25m Memory: 262144k Upper Bound: Cpu: 803m Memory: 840190575 Events: <none>Copy the code

Among them

Upper Bound: Uncapped Target: If no minimum or maximum bounds are provided for the VPA, it indicates target utilization. The above results indicate that the recommended CPU request for the Pod is 25M, and the recommended memory request is 26,2144K bytes.Copy the code

Now let’s test nginx and run the test command

[root@k8s-node001 examples]# ab - 100 - c n 10000000 http://192.168.100.185:32621/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.100.185 (be patient)
Completed 1000000 requests
Completed 2000000 requests
Completed 3000000 requests
Copy the code

7. Wait a few minutes for VPA Recommendation changes

[root@k8s-node001 ~]# kubectl describe vpa nginx-vpa -n vpa |tail -n 20
  Conditions:
    Last Transition Time:  2020-09-28T04:04:25Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:     250m
        Memory:  262144k
      Target:
        Cpu:     476m
        Memory:  262144k
      Uncapped Target:
        Cpu:     476m
        Memory:  262144k
      Upper Bound:
        Cpu:     2
        Memory:  387578728
Events:          <none>
Copy the code

Cpu: 476m. Because we set updateMode: “Off”, Pod will not be updated

Example 2 updateMode: “Auto”

1, Now I’m going to updateMode: “Auto”, and see what happens to VPA

[root@k8s-node001 examples]# kubectl apply -f nginx-vpa.yaml
deployment.apps/nginx created
[root@k8s-node001 examples]# cat nginx-vpa.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: vpa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources:
          requests:
            cpu: 100m
            memory: 50Mi

[root@k8s-node001 examples]# kubectl get po -n vpa
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7ff65f974c-f4vgl   1/1     Running   0          114s
nginx-7ff65f974c-v9ccx   1/1     Running   0          114s
Copy the code

Yaml: nginx-vpa-demo.yaml: updateMode: “Auto” and name: nginx-vPA-2

[root@k8s-node001 examples]# cat nginx-vpa-demo.yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa-2
  namespace: vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: nginx
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "nginx"
      minAllowed:
        cpu: "250m"
        memory: "100Mi"
      maxAllowed:
        cpu: "2000m"
        memory: "2048Mi"

[root@k8s-node001 examples]# kubectl apply -f nginx-vpa-demo.yaml
verticalpodautoscaler.autoscaling.k8s.io/nginx-vpa created

[root@k8s-node001 examples]# kubectl get vpa -n vpa
NAME        AGE
nginx-vpa-2   9s
Copy the code

3. Pressure test again

Ab - 1000 - c n 100000000 http://192.168.100.185:32621/Copy the code

4. After a few minutes, use Describe to check vPA details, again focusing only on Container Recommendations

[root@k8s-node001 ~]# kubectl describe vpa nginx-vpa-2 -n vpa |tail -n 30
      Min Allowed:
        Cpu:     250m
        Memory:  100Mi
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         nginx
  Update Policy:
    Update Mode:  Auto
Status:
  Conditions:
    Last Transition Time:  2020-09-28T04:48:25Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:     250m
        Memory:  262144k
      Target:
        Cpu:     476m
        Memory:  262144k
      Uncapped Target:
        Cpu:     476m
        Memory:  262144k
      Upper Bound:
        Cpu:     2
        Memory:  262144k
Events:          <none>
Copy the code

Target changed to Cpu: 587m, Memory: 262144K

5. Take a look at events

[root@k8s-node001 ~]# kubectl get event -n vpa
LAST SEEN   TYPE      REASON              OBJECT                        MESSAGE
33m         Normal    Pulling             pod/nginx-7ff65f974c-f4vgl    Pulling image "nginx"
33m         Normal    Pulled              pod/nginx-7ff65f974c-f4vgl    Successfully pulled image "nginx" in15.880996269s 33M Normal Created Pod/nginx-7FF65f974C-f4vgl Created Container Nginx 33m Normal Started pod/nginx-7ff65f974c-f4vgl Started container nginx 26m Normal EvictedByVPA pod/nginx-7ff65f974c-f4vgl Pod was evicted by  VPA Updater to apply resource recommendation. 26m Normal Killing pod/nginx-7ff65f974c-f4vgl Stopping container nginx 35m Normal Scheduled pod/nginx-7ff65f974c-hnzr5 Successfully assigned vpa/nginx-7ff65f974c-hnzr5 to k8s-node005 35m Normal Pulling pod/nginx-7ff65f974c-hnzr5 Pulling image"nginx"
34m         Normal    Pulled              pod/nginx-7ff65f974c-hnzr5    Successfully pulled image "nginx" in 40.750855715s
34m         Normal    Scheduled           pod/nginx-7ff65f974c-v9ccx    Successfully assigned vpa/nginx-7ff65f974c-v9ccx to k8s-node004
33m         Normal    Pulling             pod/nginx-7ff65f974c-v9ccx    Pulling image "nginx"
33m         Normal    Pulled              pod/nginx-7ff65f974c-v9ccx    Successfully pulled image "nginx" in15.495315629 S 33M Normal Created Pod/Nginx-7FF65f974C-v9CCX Created Container Nginx 33m Normal Started pod/nginx-7ff65f974c-v9ccx Started container nginxCopy the code

Vpa performs EvictedByVPA, automatically stops nginx, and then starts a new Nginx using the resource recommended by VPA. We can check the POD of nginx to confirm

[root@k8s-node001 ~]# kubectl describe po nginx-7ff65f974c-2m9zl -n vpaName: nginx-7FF65f974C-2M9zL Namespace: vPA Priority: 0 Node: k8S-node004/192.168.100.184 Start Time: Mon, 28 Sep 2020 00:46:19 -0400 Labels: app=nginx pod-template-hash=7ff65f974c Annotations: Cni.projectcalico.org/podIP: 100.67.191.53/32 vpaObservedContainers: nginx vpaUpdates: Pod Resources updated by nginx-VPA: container 0: CPU Request, Memory Request Status: Running IP: 100.67.191.53 IPs: IP: Controlled By: ReplicaSet/ nginX-7FF65F974c Containers: nginx: Container ID: docker://c96bcd07f35409d47232a0bf862a76a56352bd84ef10a95de8b2e3f6681df43d Image: nginx Image ID: docker-pullable://nginx@sha256:c628b67d21744fce822d22fdcc0389f6bd763daac23a6b77147d0712ea7102d0 Port: <none> Host Port: <none> State: Running Started: Mon, 28 Sep 2020 00:46:38 -0400 Ready: True Restart Count: 0 Requests: cpu: 476m memory: 262144kCopy the code

Look at key Requests: CPU: 476M, Memory: 262144K and go back to the deployment files

          requests:
            cpu: 100m
            memory: 50Mi
Copy the code

Now you can see what the VPA does. Of course, VPA recommendations change as the load on the service changes. When the resources of a currently running POD do not meet the VPA recommendations, a POD expulsion is performed to redeploy a new service with sufficient resources.

VPA usage restrictions

• Cannot be used with the Horizontal Pod Autoscaler (HPA)
• A Pod, for example, uses a replica controller, such as one belonging to Deployment or StatefulSet

What are the benefits of VPA

• Pod resources are used as needed, so cluster nodes are used efficiently.
• Pods are scheduled to nodes with appropriate available resources.
• You don’t have to run a benchmark task to determine appropriate values for CPU and memory requests.
• VPA can adjust CPU and memory requests on the fly without human intervention, thus reducing maintenance time.

Finally, VPA is a relatively new feature of Kubernetes and has not been implemented on a large scale in production environments. It is not recommended to use automatic update mode in online environments, but you can better understand the resource usage of your service by using recommended mode. For more information, visit our website