This is the 9th day of my participation in Gwen Challenge
1 Kubernetes restrictions on resources
In Kubernetes, the limits on resources (CPU, memory, etc.) need to be defined in YAML, using Deployment as an example:
apiVersion: v1
kind: Pod
metadata:
name: cpu-overload
namespace: test
spec:
containers:
- name: cpu-overload
image: stress/stress:latest
resources:
limits:
cpu: "2"
memory: 1000Mi
requests:
cpu: "1"
memory: 500Mi
command: ["stress"]
args: ["-c"."2"]
Copy the codeThe CPU has two limitations:
- Requests are the minimum request resources for a container. The requests are relative and cannot be strictly restricted.
- -Serena: It’s an absolute. You can’t go beyond it.
In this example, the CPU limit for the container CPU-overload is that two cores can be used at most when one core is applied for computing resources.
It is important to note that the so-called maximum of 2 cores is actually evenly distributed. If the container really triggers the computing bottleneck, the CPU usage in docker is 200%, but in the host machine, it is not full of 2 cores, but the pressure is spread across multiple CPU cores.
To calculate the resource limit of a POD, add up the resources of each pod container.
2 Passing of resource limits
Kubernetes can be thought of as a large tool wrapped in a series of components. In terms of resource limits, Kubernetes can’t do this by itself. Instead, it passes the resource limits, defined in YAML, to the Docker container. For example, the CPU limit of the container in Deployment is 2 cores at most, and Kubernetes will pass this limit to Docker. Therefore, in essence, the limit of Kubernetes resources comes from Docker, and to what extent can Docker limit resources? It also depends on cgroups of Linux, so Docker was not supported to run on Windows platform long ago. In the final analysis, it is because Cgroups is the product supported by Linux kernel.
Having said that, we can illustrate this transitivity with an example. Before we begin, here are the steps:
- Start a single POD in Kubernetes with a resource limit of up to 4 CPU cores.
- 2. Find the container for this POD and check the container configuration to see if it is limited to 4 cores.
- Find the Cgroups configuration for the container and see if it limits the container to 4 cores. 3, the experimental
2.1 Create a POD with a limit of 1 core
apiVersion: v1
kind: Pod
metadata:
labels:
system: centos7
name: centos7
namespace: test
spec:
containers:
- image: centos:7
imagePullPolicy: IfNotPresent
name: centos7
command: [ "/bin/bash"."-c"."--" ]
args: [ "while true; do sleep 30; done;" ]
ports:
- containerPort: 30008
hostPort: 30008
name: http
protocol: TCP
resources:
limits:
cpu: "1"
memory: 2000Mi
requests:
cpu: "0.1"
memory: 100Mi
hostNetwork: true
restartPolicy: Always
Copy the codeIn this YAML, we limit centos containers to 1 core 2 gb memory. We’re through
kubectl apply -f centos.yaml
Copy the codeRun the POD.
2.2 View the runtime limits of the container
After running as a container, look at the node where the POD is located, go to the node, find the container, and view the runtime configuration of the container with the following instructions
[root@kubernetes-master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
centos7 1/1 Running 0 2m10s 172.20. 511. kubernetes-node1 <none> <none>
docker inspect 7e5f03672fb6
Copy the codeThen, from the pile of output, find the following highlights:
"Isolation": ""."CpuShares": 102."Memory": 2097152000."NanoCpus": 0."CgroupParent": "kubepods-burstable-podb4283fa0_6576_4bd2_8c52_2183a5e2a566.slice"."BlkioWeight": 0."BlkioWeightDevice": null."BlkioDeviceReadBps": null."BlkioDeviceWriteBps": null."BlkioDeviceReadIOps": null."BlkioDeviceWriteIOps": null."CpuPeriod": 100000."CpuQuota": 100000."CpuRealtimePeriod": 0."CpuRealtimeRuntime": 0."CpusetCpus": ""."CpusetMems": ""."Devices": []."DeviceCgroupRules": null."DeviceRequests": null."KernelMemory": 0."KernelMemoryTCP": 0."MemoryReservation": 0."MemorySwap": 2097152000."MemorySwappiness": null."OomKillDisable": false."PidsLimit": null."Ulimits": null."CpuCount": 0."CpuPercent": 0.Copy the codeAmong them:
- Memory: Indicates the limit of Memory resources, expressed in byte. 2097152000 = 2 GB
- CpuShares: Relative weight of CPU usage, one core is 1024, we set request CPU to 0.1, so it is 102
- CpuPeriod: a CPU is 100000, that is, 100 milicPu. This usually does not need to be changed.
- CpuQuota: The absolute limit of CPU resources, usually combined with CpuPeriod. CpuQuota/CpuPeriod is the number of cores that can be used. 100000/100000=1, indicating that one CPU core can be used at most.
- CpusetCpus: specifies the number of cpus to which the container is bound when it is running. Note that this value is not the number of cpus, but the number of cpus to which the container is bound, separated by commas.
From the docker runtime constraints above, this is exactly what Kubernetes defines as a Pod. Now look at the Cgroups restrictions, which are the core.
2.3 Check the restricted content of Cgroups by container
First, let’s look at the name of pod:
[root@kubernetes-master ~]# kubectl get po -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
centos7 1/1 Running 0 2m10s 172.20. 511. kubernetes-node1 <none> <none>
Copy the codeThen, on the host where the pod resides, find the container Id for this pod
docker ps | grep centos7
[root@kubernetes-node1 ~]# docker ps | grep centos7
7e5f03672fb6 7e6257c9f8d8 "/ bin/bash - c - 'wh..." 6 minutes ago Up 6 minutes k8s_centos7_centos7_test_b4283fa0-6576-4bd2-8c52-2183a5e2a566_0
67d02f7cf04d k8s.gcr.io/pause:3.2 "/pause" 6 minutes ago Up 6 minutes k8s_POD_centos7_test_b4283fa0-6576-4bd2-8c52-2183a5e2a566_0
Copy the codePause: Kubernetes: Kubernetes: Kubernetes: Kubernetes: Kubernetes: Kubernetes: Kubernetes: Kubernetes: Kubernetes We just need Centos7:7E5F03672FB6 to get the Cgroup information of this container
[root@kubernetes-node1 ~]# docker inspect 7e5f03672fb6 | grep Cgroup
"Cgroup": ""."CgroupParent": "kubepods-burstable-podb4283fa0_6576_4bd2_8c52_2183a5e2a566.slice"."DeviceCgroupRules": null.Copy the codeOk, that’s it, let’s go straight to the Cgroup configuration directory:
cd /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4283fa0_6576_4bd2_8c52_2183a5e2a566.sl iceCopy the codeIn this directory, there are many files:
[root@kubernetes-node1 kubepods-burstable-podb4283fa0_6576_4bd2_8c52_2183a5e2a566.slice]# ll
total 0
-rw-r--r-- 1 root root 0 Nov 17 14:50 cgroup.clone_children
--w--w--w- 1 root root 0 Nov 17 14:50 cgroup.event_control
-rw-r--r-- 1 root root 0 Nov 17 14:50 cgroup.procs
-r--r--r-- 1 root root 0 Nov 17 14:50 cpuacct.stat
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpuacct.usage
-r--r--r-- 1 root root 0 Nov 17 14:50 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.cfs_period_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.cfs_quota_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.rt_period_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.shares
-r--r--r-- 1 root root 0 Nov 17 14:50 cpu.stat
drwxr-xr-x 2 root root 0 Nov 17 14:50 docker-67d02f7cf04d6eb6ed637e40e1ccebd18b09eb323ad262c1fe3e8aa75ea46edf.scope
drwxr-xr-x 2 root root 0 Nov 17 14:50 docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope
-rw-r--r-- 1 root root 0 Nov 17 14:50 notify_on_release
-rw-r--r-- 1 root root 0 Nov 17 14:50 tasks
Copy the codeWhere, the current directory has a lot of Cgroup content, and there are two subdirectories:
docker-67d02f7cf04d6eb6ed637e40e1ccebd18b09eb323ad262c1fe3e8aa75ea46edf.scope
docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope
Copy the codeThese two directories are actually the two containers in pod (pause, Centos7), and we are entering the centos7 directory
cd docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope
Copy the codeCheck the contents in the directory
-rw-r--r-- 1 root root 0 Nov 17 14:50 cgroup.clone_children
--w--w--w- 1 root root 0 Nov 17 14:50 cgroup.event_control
-rw-r--r-- 1 root root 0 Nov 17 14:50 cgroup.procs
-r--r--r-- 1 root root 0 Nov 17 14:50 cpuacct.stat
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpuacct.usage
-r--r--r-- 1 root root 0 Nov 17 14:50 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.cfs_period_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.cfs_quota_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.rt_period_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 Nov 17 14:50 cpu.shares
-r--r--r-- 1 root root 0 Nov 17 14:50 cpu.stat
-rw-r--r-- 1 root root 0 Nov 17 14:50 notify_on_release
-rw-r--r-- 1 root root 0 Nov 17 14:50 tasks
Copy the codeYou can see several familiar words, such as cpu.cfs_quota_us, which limit CPU resources. Let’s take a look at the contents:
[root@kubernetes-node1 ...scope]# cat cpu.cfs_quota_us
100000
Copy the codeYes, the value is 100000, which is 1 CPU.
2.4 How is the Linux Cgroup associated with the Dock?
In the above way, we have found out how the restrictions on CPU, memory and so on are traced to Cgroup step by step through the Deployment of Kubernets. So how does a Linux Cgroup relate to a container?
Let’s look at tasks in a cgroup directory
[root@kubernetes-node1 docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope]# cat tasks
22552
Copy the codeThis value is the process ID, so Cgroup limits the resource by the process ID. Let’s look at the process ID
[root@kubernetes-node1 docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope]# ps -ef | grep 22552
root 22552 22534 0 14:50 ? 00:00:00 /bin/bash -c -- while true; do sleep 30; done;
root 30414 22552 0 15:08 ? 00:00:00 sleep 30
root 30443 23532 0 15:08 pts/0 00:00:00 grep --color=auto 22552
Copy the codeThe process ID is the cAdvisor process ID, and the parent process must be a container process:
[root@kubernetes-node1 docker-7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2.scope]# ps -ef | grep 22534
root 22534 906 0 14:50 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/7e5f03672fb63500910b092ecfcc793a8f21450528bb32c9d078d31dc4846cb2 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc -systemd-cgroup
root 22552 22534 0 14:50 ? 00:00:00 /bin/bash -c -- while true; do sleep 30; done;
root 30875 23532 0 15:09 pts/0 00:00:00 grep --color=auto 22534
Copy the code3 summary
- Kubernete limit on resources, by Docker, Docker limit on resources, by Linux Cgroup.
- Linux Cgroup limits resources and processes. You only need to add the process ID to the Tasks file in the Cgroup configuration directory. The restriction takes effect immediately.
- Linux Cgroup can limit not only CPU, memory, but also disk IO, etc.