Background: I feel that K8S does not have many resources, but the total request ratio of K8S is very high, so I conduct investigation
The screening process
- From the above figure, the total CPU limit of the cluster is extremely high and the cluster usage is extremely low. Note There is room for reducing the request value.
- From the perspective of namespace, production space is obviously abnormal. The average request value of a POD is 1, and the limit value is 2500
-
There are 1000 pods. No wonder the request and Limit values are so large
-
More than 1000 pods are generated by cronjobs, because the namespace is configured with isTIO automatic injection. When the container in cronJobs finishes running, the ISTIo-proxy container still runs normally, so the pod does not exit. Github.com/istio/istio… Found the solution in this issue, it is in a cronjob (cronjob. Spec. JobTemplate. Spec. Metadata) metadata annotation sidecars. Add istio. IO/inject: “False”, and write a script to exit the existing IStio-proxy one by one. The curl – Max – time 2 – s – f – XPOST http://127.0.0.1:15000/quitquitquit
kubectl exec -n production -t pod_name -c istio-proxy -- curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit
Copy the code
The results of
- Now the cluster limit and request look much more normal, but the problem of low utilization still exists, and we will optimize it later