This is the fourth day of my participation in Gwen Challenge

An overview,

Are introduced in the spring in the boot 2.3 container probe, which is added/physical/health/liveness and/physical/health/readiness both health inspection path, for the deployment of the application in k8s, The Spring-boot-ACTUATOR automatically performs health checks through these two paths. Based on the practice described in official documents and the usage process recorded, this paper introduces it from the following aspects:

  1. Health check in K8S
  2. K8s probes in Spring-boot-ACTUATOR
  3. Practice of Spring Boot health check in K8S

Ii. Health check of K8s containers

1. Probe in K8S

Kubernetes provides three probes (supporting exec, TCP, and HTTP modes) to probe the state of the container:

  • LivenessProbe: Container viability test, used to judge whether the container is healthy, tellkubeletWhen a container is in an unhealthy state. ifLivenessProbeIf the probe detects that the container is unhealthy, thenkubeletThe container will be deleted and processed according to the container restart policy. If a container does not containLivenessProbeProbe, thenkubeletThink of the containerLivenessProbeThe value returned by the probe is alwaysSuccess;
  • ReadinessProbe: Container readiness check to determine whether the container is started and ready to receive requests. ifReadinessProbeProbe detects failure,Endpoint ControllerWill be taken fromService 的 EndpointTo delete the IP address of the Pod where the container residesEndpointEntries. If the container does not provide a ready probe, the default state isSuccess.
  • startupProbe: Container startup check to indicate whether the application in the container has been started. If a startup probe is provided, all other probes are disabled until this probe succeeds. If starting probes fails,kubeletWill kill the container and the container restarts according to its restart policy. If the container does not provide startup probes, the default isSuccess.

StartupProbe has been added to the alpha version of K8S V1.16. Indicates whether the application within the Container is started. All other probes are disabled if a startup probe is provided, until it succeeds. If the startup probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy. If a Container does not provide a startup probe, the default state is Success

2. K8s probe handler

Probe checks are periodic diagnostics performed by Kubelet on containers. Kubelet calls handlers implemented by containers. Each Probe can be configured with one of three types of handlers:

  • ExecAction: Execute the specified command in the container. If the return code is 0 when the command exits, the diagnosis is considered successful.

For example, run the cat command to check whether an important configuration file in pod exists. If it exists, the POD is healthy. Otherwise, the service is abnormal. Exec probe yamL file syntax is as follows:

spec:  
  containers:  
  - name: liveness  
    image: k8s.gcr.io/busybox  
    args:  
    - /bin/sh  
    - -c  
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600  
    livenessProbe:               # Select livenessProbe's probe mechanism
      exec:                      If the file exists, the probe succeeds
        command:  
        - cat  
        - /tmp/healthy  
      initialDelaySeconds: 5          Start probing five seconds after the container runs
      periodSeconds: 5                Check every 5 seconds

Copy the code
  • TCPSocketAction: Performs a TCP check on the specified port on the container’s IP address. If the port is open, the diagnosis is considered successful.

Tcpsocket health check applies to TCP services. The syntax of the YAML file for tcpSocket detection is as follows:

spec:  
  containers:  
  - name: goproxy  
    image: K8s. GCR. IO/goproxy: 0.1  
    ports:  
      - containerPort: 8080  
    Both detection mechanisms are used to establish a TCP connection to port 8080 of the container
    readinessProbe:  
      tcpSocket:  
        port: 8080  
      initialDelaySeconds: 5  Start the first readiness probe after 5s of container startup
      periodSeconds: 10       # Probe every 10s
    livenessProbe:  
      tcpSocket:  
        port: 8080  
      initialDelaySeconds: 15  Start the first activity detection after 15s of container startup
      periodSeconds: 20  # Probe every 20 seconds
Copy the code

In the yamL configuration file above, both types of probes are used. Five seconds after the container is started, Kubelet will send the first readinessProbe probe, which will connect to port 8080 of the container. If the probe is successful, the POD is healthy.

In addition to the readinessProbe probe, Kubelet will send the first livenessProbe 15 seconds after the container is started, still trying to connect to port 8080 of the container, and restarting the container if the connection fails.

  • HTTPGetAction: An HTTPGet request is executed on the specified port and path on the container’s IP address. If the status code of the response is greater than or equal to 200 and less than 400, the diagnosis is considered successful.

The syntax of yamL files for Httpget probes is as follows:

spec:  
  containers:  
  - name: liveness  
    image: k8s.gcr.io/liveness  
    livenessProbe:              # livenessProbe mechanism was adopted
      httpGet:                  Use httpget
        scheme: HTTP         # specify protocol, HTTPS is also supported
        path: /healthz          Check whether the healthZ web file in the root directory of the web page can be accessed
        port: 8080              The listening port is 8080
      initialDelaySeconds: 3     Start probe after container runs for 3 seconds
      periodSeconds: 3                # Detection frequency is 3 seconds

Copy the code

In the configuration file above, the detection method is that the item container sends an HTTP GET request to the HealthZ file under port 8080, and any status code greater than or equal to 200 and less than 400 is returned indicating success, and any other code indicates exception.

Each probe yields one of three results:

  • Success: The container passed the diagnosis.
  • Failure: Container fails diagnosis.
  • Unknown: Diagnosis failed, so no action is taken.

3. Configuration of K8S probe

Probe has the following configuration fields that you can use to precisely control the behavior of survival and readiness detection:

  • InitialDelaySeconds: The number of seconds the container must wait after starting before the survival and readiness probes are initialized. The default is 0 seconds and the minimum is 0.
  • PeriodSeconds: Interval of detection (in seconds). The default is 10 seconds. The minimum is 1.
  • TimeoutSeconds: The number of seconds to wait after the timeout of the probe. The default value is 1 second. The minimum is 1.
  • SuccessThreshold: The minimum number of consecutive successes to be considered successful after a probe has failed. The default value is 1. This value must be 1 for live and start probes. The minimum is 1.
  • FailureThreshold: Number of Kubernetes retries when a probe fails. Abandoning in the case of survival probes means restarting the container. Abandoned pods in ready detection cases are labeled as not ready. The default value is 3. The minimum is 1.

Note: Prior to Kubernetes 1.20, exec probes ignored timeoutSeconds: Probes would continue running indefinitely, possibly beyond the configured deadline, until results were returned. This defect was fixed in Kubernetes V1.20.

LivenessProbe configuration example:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 30        // The first probe is started 30 seconds after the container starts
  periodSeconds: 10              //  Probes are started every 10 seconds
  timeoutSeconds: 3               // Timeout time 3s
  successThreshold: 1             // Success once indicates that the container is healthy
  failureThreshold: 5             // If the container fails for five consecutive times, the container is considered unhealthy. The default value is three times
Copy the code

4. When to use

When to use the starter probe

Kubelet uses startup detectors to know when the application container is started. If such detectors are configured, you can control the container to perform viability and readiness checks after successful startup to ensure that these live, ready detectors do not affect application startup. This can be used to detect the viability of slow-start containers to prevent them from being killed before they are up and running.

When to use survival probe

Kubelet uses survival probes to know when to restart containers. For example, a survival detector can catch a deadlock (the application is running, but cannot continue to perform subsequent steps). Restarting the container in such cases helps make the application more usable in the event of a problem.

When to use the ready probe

Kubelet uses the readiness probe to know when a container is ready and ready to start receiving request traffic. A Pod is considered ready only when all containers in a Pod are ready. One use of this signal is to control which Pod is the back end of the Service. Pods will be removed from Service load balancers before they are ready.

Reference article:

K8s container probe