Keep POD healthyP84

Whenever a POD is scheduled to a node, the Kubelet on that node will run the pod container and keep running for as long as the POD exists. If the container’s main process crashes, Kubelet automatically restarts the container. If the application crashes, Kubelet will automatically restart the application. P84

Applications can also stop responding due to conditions such as infinite loops or deadlocks. To ensure that the application can be restarted in this case, the health of the application must be checked externally, rather than relying on the application’s internal checks. P84

Introduction to survival detectorsP84

Kubernetes can use liVENESS probes to check if the container is still running. Survival probes can be specified individually for each container in a POD. Kubernetes will execute probes periodically and automatically restart the container if probes fail. P84

Note: Kubernetes also supports readiness probes, both of which are suitable for two different scenarios. P84

Kubernetes has three mechanisms for probing containers: P84

  • HTTP GETProbe: executes on the container’s IP address (specified port and path)HTTP GETThe request. If the probe receives a response and the response status code does not represent an error (the status code is 2xx or 3XX), the probe is considered successful. If the server returns an error response status code or does not respond, the probe is considered a failure and the container is restarted.
  • TCP SocketProbe: Attempts to establish a TCP connection with the specified port of the container. If the connection is successfully established, the probe succeeds. Otherwise, the container will be restarted.
  • ExecProbe: Executes any command in the container and checks the exit status code of the command. If the status code is 0, the probe succeeds. All other status codes are considered to have failed and the container will be restarted.
Create an HTTP-based survival detectorP85

To make the HTTP GET probe fail, we need to modify the kubia source code so that it returns 500 Internal Server errors from the fifth access. P85

We can then create a POD containing an HTTP GET survival probe with the following description file kubia-liP-probe. yaml. P85

Follow the Kubernetes API v1
apiVersion: v1
Resource type: Pod
kind: Pod
metadata:
  # the name of pod
  name: kubia-liveness
spec:
  containers:
    Create an image for the container
    - image: idealism/kubia-unhealthy
      The name of the container
      name: kubia
      ports:
        The application listens on the port
        - containerPort: 8080
          protocol: TCP
      # Activate a survival probe
      livenessProbe:
        The survival probe type is HTTP GET
        httpGet:
          The network port to which the probe is connected
          port: 8080
          The path requested by the probe
          path: /
Copy the code
Use survival detectorP86

After the pod is created using kubectl create-f kubia-liP-probe. yaml, wait for some time before the container restarts. We can see from kubectl get pod Kubia-liVENESS that the container restarts and continues indefinitely: 86

NAME             READY   STATUS    RESTARTS   AGE
kubia-liveness   1/1     Running   2          4m9s
Copy the code

Kubectl logs Kubia-liVENESS — Previous: View the logs of the previous container to learn why the previous container stopped. P86

Kubectl describe POD Kubia-liVENESS: View POD details. It can be found that termination information is contained in Containers and Events. P86

. Containers: kubia: ... Sun, 07 Jun 2020 17:59:35 +0800 Last State: Terminated # Reason: Error Exit Code: 137 Started: Sun, 07 Jun 2020 17:57:44 +0800 Finished: Sun, 07 Jun 2020 17:59:27 +0800 Ready: True Restart Count: 2 # The container has been restarted twice http-get http://:8080/ delay=0s timeout=1s period=10s #success=1 #failure=3 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/kubia-liveness to minikube-m02 Warning Unhealthy 48s (x6 over 2m58s) kubelet, minikube-m02 Liveness probe failed: HTTP probe failed with statuscode: Normal Killing 48s (x2 over 2M38s) Kubelet, MiniKube-M02 Container Kubia failed liVENESS probe Lua lua/lua lua /Copy the code

Error code 137 is the sum of two digits: 128 + x, where x is the signal number to terminate the process. P86

  • X = 9 is saidSIGKILLIndicates that the process is forcibly terminated, that the signal cannot be captured or ignored, and that no cleanup can be performed during the receiving process until the signal is received
  • X = 15 is saidSIGTERMSignal number, indicating that the process has been terminated, prior to asking the process to terminate, allowing it to clean up files and close, can be captured and interpreted or ignored

The events listed at the bottom show that Kubernetes found the container unhealthy, so terminated and recreated it. P86

Note: When a container is forcibly terminated, a new container is created instead of restarting the original container. P86

Configures the additional properties of the survival probeP87

Can use kubectl explain pod. Spec. Containers. LivenessProbe view live detector can use custom additional parameters.

Create a new description file based on kubia-liP-probe. yaml kubia-liP-probe-initial-delay. yaml, And add the pod. Spec. Containers. LivenessProbe. InitialDelaySeconds attribute, the value of 15, said first detector for 15 seconds.

. Spec: containers: # Imagealize/kubia-Unhealthy... # Activate a livenessProbe:... Wait 15 seconds before first probe initialDelaySeconds: 15Copy the code

This allows probing to be done after the application is ready, so that the application does not take too long to start, causing endless restarts as probes fail.

Create a viable survival detectorP88
  • What should a survival probe check for: A simple survival probe can just check if the server is responding, but to do a better survival check, the probe needs to be configured to request a specific URL path (such as/health), and have the application perform a state check on all the important components running internally to ensure that none of them have stopped or stopped responding.P88
    • Make sure that/healthAuthentication is not required, otherwise probes will continue to fail, causing the container to restart indefinitely
    • Check the inside of the application, and the results should not be influenced by any external factors. For example, the survival probe should not return failure if the database is not connected. If the problem is in the database, restarting the container will not solve the problem.
  • Keep the detector lightP89
    • You should not consume too many computing resources
    • It should not take too long to run. By default, probes execute relatively frequently and must be executed in less than a second
  • Retry is not required in the probe: the failure threshold for the probe is configurable, and the probe must typically fail multiple times before the container is terminatedP89
  • Survival probe summary: Survival probe is performed by Kubelet on POD. The Kubernetes control panel does not participate. So when a node crashes, the control panel creates a replacement for all pods that stop running with the node, but not for pods that are created directly, because these pods are only managed by Kubelet on the node. To avoid this, we should use a controller or similar mechanism to manage the POD.P89

To understandDeployment P89

Note: Pod is managed by Deployment, as mentioned in this section. In fact, pod is managed and created by ReplicaSet, which is managed by Deployment.

Deployment is a Kubernetes resource that ensures that its PODS are always running. If the POD disappears for any reason (for example, the node disappears from the cluster or because the POD has been expelled from the node), the Deployment will notice the absence of the POD and create a replacement. P89

Node 1 in the figure above has two nodes, Pod A is created directly and Pod B is managed by Deployment. After the node exits unexpectedly, the Deployment creates A new Pod B2 to replace the reduced Pod B, and Pod A is completely lost as there is nothing to rebuild. P89

DeploymentThe operation of theP90

Deployment continuously monitors the list of running pods and ensures that the number of pods matching the label selectors (03.pod: The label selectors running on containers in Kubernetes and how to use them) is as expected. P90

This section describes the controller coordination process P91

It is the Deployment’s job to ensure that the number of pods always matches its label selector. P91

Learn about the three parts of Deployment P91

  • Label selector: Used to determineDeploymentWhich pods are in scope
  • Replica Count: specifies the number of pods to be run
  • Pod Template: Used to create a new POD copy

The number of copies in Deployment, the label selector, and the POD template can all be changed at any time, but only the number of copies but changes affect existing pods. P92

Change the effect P92 of the controller’s label selector or POD template

Changing the label selector and POD template has no effect on existing PODS. Changing the label selector takes existing pods out of the scope of Deployment, so the controller stops caring about them. Changing the template only affects new pods created from this Deployment. P92

Benefits of using Deployment P92

  • Ensure pod runs continuously: Start a new pod if an existing one is lost
  • When a cluster node is faulty, it is an object running on the faulty nodeDeploymentCreate alternate copies of all managed pods
  • Easily achieve POD horizontal scaling — manual and automatic

Note: POD instances are never relocated to another node. Deployment creates a brand new POD instance, independent of the instance being replaced. P92

To create aDeployment P92

We can create a Deployment with the following description file kubia-deployment.yaml, which ensures that there are always three pod instances conforming to the label selector app=kubia.

Follow the Kubernetes API v1
apiVersion: apps/v1
The resource type is Deployment
kind: Deployment
metadata:
  The name of the Deployment
  name: kubia
spec:
  # set the number of pods matching the tag selector to 3
  replicas: 3
  Specify the Deployment action object
  selector:
    Need to match the tag specified below
    matchLabels:
      app: kubia
  Yaml (kubia-manual.yaml)
  template:
    metadata:
      # specify the tag app=kubia
      labels:
        app: kubia
    spec:
      containers:
        The name of the container
        - name: kubia
          Create an image for the container
          image: idealism/kubia
          The application listens on the port
          ports:
            - containerPort: 8080
              protocol: TCP
Copy the code

The POD tag in the template must match the label selector for Deployment, and the API server verifies the Deployment definition and does not accept incorrect configuration. P93

If you do not specify a selector, it is automatically configured according to the label in the POD template, which makes the description file more concise. P93

useDeployment P94

Kubectl create -f kubia-deployment.yaml creates a deployment named kubia, which starts three new pods based on the POD template. P94

Kubectl get Pods can view all pods currently created:

NAME                    READY   STATUS             RESTARTS   AGE
kubia-9495d9bf5-5dwj7   1/1     Running            0          3m53s
kubia-9495d9bf5-5j6zr   1/1     Running            0          3m53s
kubia-9495d9bf5-w98f6   1/1     Running            0          3m53s
Copy the code

See Deployment’s response to the deleted POD, P94

Kubectl delete pod kubia-9495d9bf5-5dwj7 delete pod kubia-9495d9bf5-5dwj7 delete pod kubia-9495d9bf5-5dwj7 delete pod kubia-9495d9bf5-5dwj7

NAME                    READY   STATUS              RESTARTS   AGE
kubia-9495d9bf5-5dwj7   1/1     Terminating         0          24m
kubia-9495d9bf5-5j6zr   1/1     Running             0          24m
kubia-9495d9bf5-kxcw5   0/1     ContainerCreating   0          9s
kubia-9495d9bf5-w98f6   1/1     Running             0          24m
Copy the code

How does the controller create a new POD P95

The controller responds to pod deletion by creating a new replacement POD. But it does not react to the deletion itself, but to the resulting state of insufficient PODS. P95

This section describes how to deal with node fault P96

Next we will shut down the network interface of one node to simulate a node failure. P96

  1. minikube ssh --node='m02': Enters the node
  2. sudo ifconfig eth0 down: Disables the network interface of the node
  3. kubectl get nodes: Discovering a nodeminikube-m02Is not ready (NotReady)
  4. kubectl get pods: You might still see the same three pods as before, because Kubernetes waits some time before rescheduling the pod (if the node is inaccessible due to a temporary network failure or Kubelet restart). If the node becomes inaccessible within a few minutes,DeploymentA new pod will start immediately.

Move the POD into/out of the Deployment scope P97

A POD created by Deployment is not bound to Deployment. At any time, Deployment manages the POD that matches the label selector. A POD can be added or removed from the scope of Deployment by changing its label. P97

Although a POD is not bound to a Deploymed-owned ReplicaSet, the pod stores which ReplicaSet it belongs to in metadata.ownerReferences. P98

Label the Pod managed by Deployment P98

Kubectl Label Pod kubiA-9495D9BF5-5MMHB type=special: Adding additional labels to a pod does not affect the administrative scope of Deployment, it only cares if the pod has all the labels referenced in the label selector. P98

Change the label P98 for a managed POD

kubectl label pod kubia-9495d9bf5-5mmhb app=foo –overwrite: Changing the label of one of the pods will no longer match the Label selector of Deployment and will no longer be managed by Deployment, leaving only two matching pods. Therefore, Deployment starts a new POD and restores the number to three. P98

Change the label selector P100 for Deployment

Changing the label selector for Deployment takes all the pods out of Deployment’s administration, causing it to create three new pods. You will never change the controller’s label selector, but will change its POD template from time to time. P100

Modify the POD templateP100

The POD template for Deployment can be modified at any time, and Deployment can be edited using Kubectl Edit Deployment kubia. This will create a new ReplocaSet and change the number of copies of the original ReplocaSet to zero. Thus, using Kubectl Get Pods you will find six pods prefixed by the name of the corresponding ReplocaSet.

NAME                    READY   STATUS        RESTARTS   AGE
kubia-9495d9bf5-kxcw5   1/1     Terminating   0          78m
kubia-9495d9bf5-w98f6   1/1     Terminating   0          102m
kubia-9495d9bf5-xn67d   1/1     Terminating   0          29m
kubia-bc974964b-bp4l2   1/1     Running       0          22s
kubia-bc974964b-r29j2   1/1     Running       0          39s
kubia-bc974964b-xl677   1/1     Running       0          14s
Copy the code

The Replicaset instance owned by Deployment can be modified directly by using kubectl Edit Replicaset kubia-BC974964b. This is similar to changing Deployment directly, creating a new ReplicaSet and changing the number of ReplocaSet copies to 0. This change will not synchronize the new POD template back to the original Deployment, but deleting the Deployment will still remove all associated REPLocasets and the managed PODS.

Horizontal zoom PODP101

Kubectl scale deployment kubia –replicas=10: Can change the number of POD instances that deployment needs to keep (02. Scaling with this command is described in Kubernetes and Docker. P101

You can also change the number of spec.replicas by modifying kubectl Edit Deployment kubia to change the number of POD instances that need to be kept. P102

To delete aDeployment

When deployment is deleted by kubectl delete Deployment kubia, both ReplicaSet and POD are deleted.

When replicaset is deleted by kubectl delete Replicaset kubia-bc974964b, the pod will be deleted, but since Deployment will create a replicaset again, So the corresponding number of pods will be automatically created.

When deployment is deleted by kubectl delete Deployment kubia –cascade=false, ReplicaSet and POD are retained so that ReplicaSet is no longer managed. But PODS are still managed by ReplicaSet. When a matching Deployment is recreated, ReplicaSet is managed again.

Similarly, when a REPLICaset is deleted by kubectl delete Replicaset kubia-bc974964b –cascade=false, the pod is preserved. This way the POD is no longer managed. When a ReplicaSet is created that meets the requirements, these pods are managed.

useReplicaSet P104

Note: The previous section of the book was originally ReplicationController, but I implemented it directly in Deployment and modified it according to the current results. Deployment is currently recommended, and ReplicaSet is managed by Deployment, so I won’t go through this section in detail.

Use more expressive label selectorsP106

Create a new description file kubia-deployment.yaml based on kubia-deployment.yaml, And the spec. The selector. MatchLabels attribute is replaced by the spec. The selector. MatchExpressions: P107

.
spec:
  .
  Specify the Deployment action object
  selector:
    Tags that meet the following requirements need to be matched
    matchExpressions:
      The value of # tag named app is in ["kubia"]
      - app: app
        operator: In
        values:
          - kubia
  .
Copy the code

The matchExpressions run adds additional expressions to the selector. Each expression must contain a key, an operator, and possibly a list of values (depending on the operator). There are four valid operators: P107

  • In: The value of the tag must match one of the values specified
  • NotIn: The value of the tag does not match any specified values
  • Exists: POD must contain a tag with the specified name (regardless of the value). When using this operator, the VALUES field should not be specified
  • DoesNotExist: POD must not contain a label with the specified name. When using this operator, the VALUES field should not be specified

If multiple expressions are specified, all of them must be true for the selector to match pod. If both matchLabels and matchExpressions are specified, then all labels must match, and all expressions must be true for the selector to match pod. P107

useDaemonSetRun a POD on each nodeP107

DaemonSet allows pods to run on every node in the cluster, with exactly one running POD instance on each node. P107

useDaemonSetRun a POD on each nodeP108

DaemonSet, which has no concept of copy numbers, ensures that enough PODS are created and one pod is deployed on each node. If the node goes offline, DaemonSet does not recreate the POD. But when a new node is added to the cluster, it will immediately deploy a new POD instance to that node. P108

useDaemonSetRun pod only on specific nodesP109

DaemonSet deploates pods to all nodes in the cluster, unless these pods are specified to run only on some nodes through the spec.nodeselector property in the POD template. P109

Note: Nodes can be set to non-schedulable, preventing pods from being deployed to nodes. But DaemonSet will deploy pods to these nodes because it cannot be scheduled but the properties are only used by the scheduler, and the purpose of DaemonSet is to run system services, which usually need to run even on unscheduled nodes. P109

Use an example to explain DaemonSet P109

Suppose you have a daemon called SSD-monitor that needs to run on all nodes that contain SSDS. The nodes that contain SSDS have the disk= SSD tag added, so we need to create a DaemonSet that runs the daemon only on nodes that have that tag. P109

Create a DaemonSet description file P110

To simulate ssD-monitor’s monitor, we will use the following Dockerfile to create a mirror that prints SSD OK every 5 seconds.

FROM busybox
ENTRYPOINT while true; do echo 'SSD OK'; sleep 5; done
Copy the code

In order to deploy SSD-Monitor to each node that meets the requirements, we also need to use the following SSD-monitoring-daemtasont.yamL description file for deployment.

Follow the Apps/V1 version of the Kubernetes API
apiVersion: apps/v1
Resource type: DaemonSet
kind: DaemonSet
metadata:
  # DaemonSet
  name: ssd-monitor
spec:
  Specify DaemonSet operation objects
  selector:
    Need to match the tag specified below
    matchLabels:
      app: ssd-monitor
  Start the template used by pod
  template:
    metadata:
      # specify the tag as app= ssD-monitor
      labels:
        app: ssd-monitor
    spec:
      # select the node with the disk= SSD label for deployment
      nodeSelector:
        disk: ssd
      containers:
        The name of the container
        - name: main
          Create an image for the container
          image: idealism/ssd-monitor
Copy the code

Practice P110

  1. kubectl create -f ssd-monitor-daemonset.yaml: Creates a file based on the specified description fileDaemonSet
  2. kubectl get daemonsets: You can see that all values are 0, because no node currently owns themdisk=ssdThe label
  3. kubectl get pods: You can see that there is no POD yet
  4. kubectl label node minikube-m03 disk=ssd: to the nodeminikube-m03taggeddisk=ssd
  5. kubectl get pods: You can see that a POD has just been started 
    NAME                    READY   STATUS              RESTARTS   AGE
ssd-monitor-bbqbp       0/1     ContainerCreating   0          2s
Copy the code
  6. kubectl label node minikube-m03 disk=hdd --overwrite: the nodeminikube-m03The label of thedisk=ssdModified todisk=hdd
  7. kubectl get pods: You can see that the newly started POD is terminating 
    NAME                    READY   STATUS        RESTARTS   AGE
ssd-monitor-bbqbp       1/1     Terminating   0          2m37s
Copy the code

Run pods that perform a single taskP112

introduceJobresourcesP112

Kubernetes runs a POD with Job resource support that does not restart the container when the internal process terminates successfully. Once the task is complete, the POD is considered to be in the completed state. P112

When a node fails, the POD managed by the Job on the node is relocated to another node. If the process exits unexpectedly (if the process returns an error exit code), you can configure the Job to restart the container. P112

defineJobresourcesP113

To simulate time-consuming tasks, we will use the following Dockerfile to create a mirror that calls the sleep 120 command.

FROM busybox
ENTRYPOINT echo "$(date) Batch job starting"; sleep 120; echo "$(date) Finished succesfully"
Copy the code

To manage the deployment of batch-jobs, we also need to use the following batch-job. Yaml description file for deployment.

Follow batch/ V1 version of the Kubernetes API
apiVersion: batch/v1
The resource type is Job
kind: Job
metadata:
  # Job name
  name: batch-job
spec:
  Start the template used by pod
  template:
    metadata:
      # batch job = app=batch-job
      labels:
        app: batch-job
    spec:
      # Job Cannot use Always as the default restart policy
      restartPolicy: OnFailure
      containers:
        The name of the container
        - name: main
          Create an image for the container
          image: idealism/batch-job
Copy the code

Setting the Job restart policy to OnFailure or Never prevents the container from restarting when the task is complete. P114

JobRun a PODP114
  1. kubectl create -f batch-job.yaml: creates the specified file according to the description fileJob
  2. kubectl get jobs: View the job. You can find the newly created jobJob 
    NAME        COMPLETIONS   DURATION   AGE
batch-job   0/1           5s         5s
Copy the code
  3. kubectl get pods: Check the PODJobThe pod created is running 
    NAME                    READY   STATUS        RESTARTS   AGE
batch-job-d59js         1/1     Running       0          10s
Copy the code
  4. kubectl get pods: Wait two minutes to check the POD, you can findJobThe status of the created POD has changedCompletedThe task has been completed. The pod is not deleted, so we can view the pod logs 
    NAME                    READY   STATUS        RESTARTS   AGE
batch-job-d59js         0/1     Completed     0          2m56s
Copy the code
  5. kubectl logs pod batch-job-d59js: View the POD logs 
    Sun Jun  7 22:36:04 UTC 2020 Batch job starting
Sun Jun  7 22:38:04 UTC 2020 Finished succesfully
Copy the code
  6. kubectl get jobsIf you look at the job again, you can see that the one POD you need to run is complete 
    NAME        COMPLETIONS   DURATION   AGE
batch-job   1/1           2m45s      6m25s
Copy the code
inJobRun multiple POD instances inP114

Setting spec.pletions and spec.parallelism in the Job configuration lets the Job create multiple POD instances and allow them to run in parallel. P114

Create a new description file based on batch-job. Yaml multi-completion-parallel- Batch-job. Yaml and add spec.pletions and spec.parallelism attributes. Specifies that 5 pods need to be successfully run and up to 2 pods run in parallel: P115

. spec:  #You must ensure that all five pods are running
  completions: 5
  #Up to two pods can run in parallel
  parallelism: 2
  ...
Copy the code

  1. Kubectl create -f multi-completion-parallel-batch-job. Yaml: creates the specified job according to the description file

  2. Kubectl get Pods: Check the running pods and see that there are two running pods. As soon as one pod completes, the Job will run the next pod until all five have successfully completed

    NAME                                        READY   STATUS        RESTARTS   AGE
multi-completion-parallel-batch-job-fpwv5   1/1     Running       0          37s
multi-completion-parallel-batch-job-m4cqw   1/1     Running       0          37s
Copy the code
limitJobThe time that pod completes the taskP116
  • Pod.spec.activeDeadlineSeconds: You can specify a maximum pod lifetime, terminate the POD and mark it when timeout occursJobFailure can be used to limit the time the POD takes to complete the task
  • Job.spec.backoffLimit: You can configure oneJobMaximum number of attempts before being marked as failure. Default is 6

arrangeJobRun periodically or once in the futureP116

To create aCronJob P116

To run the previous task every 15 minutes, we need to create the following cronjob.yaml description file:

Follow batch/ v1Beta1 version of the Kubernetes API
apiVersion: batch/v1beta1
The resource type is CronJob
kind: CronJob
metadata:
  # Job name
  name: batch-job-every-fifteen-minutes
spec:
  The # Cron expression indicates that the current task is running at 0, 15, 30, 45 minutes every hour of the day
  schedule: "0,15,30,45 * * * *"
  # specify that the Job must start within 15 seconds of the scheduled time, otherwise it will be marked as a failed 'Job'
  startingDeadlineSeconds: 15
  [Batch-job. Yaml] [batch-job. Yaml]
  jobTemplate:
    spec:
      Start the template used by pod
      template:
        metadata:
          # app=periodic-batch-job
          labels:
            app: periodic-batch-job
        spec:
          # Job Cannot use Always as the default restart policy
          restartPolicy: OnFailure
          containers:
            The name of the container
            - name: main
              Create an image for the container
              image: idealism/batch-job
Copy the code

Kubectl get cronjobs: you can view all cronjobs

NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE Batch-job-every-fifteen -minutes 0,15,30,45 * * * * False 0 < None > 8sCopy the code

CronJob always creates a Job for each execution configured in the plan, but there are two possible problems:

  • Create bothJob: Keep tasks idempotent
  • Do not createJob: Make sure the next task runs to complete any missed work

This article is published on GitHub: Reading-notes/kubernetes-in-Action