Kubernetes YAML Best Practices and Strategies

Kubernetes workloads are most often defined in YAML format files.

One of the problems with YAML is that it is difficult to describe constraints or relationships between manifest files. What if you want to check that all images deployed to the cluster have been extracted from the trusted registry? How do I prevent workloads without Pod security policies from being committed to the cluster? Integrated static checking can catch errors and policy violations closer to the development life cycle. And because of the improved availability and security of resource definitions, you can be confident that production workloads follow best practices.

The Kubernetes YAML file static checking ecosystem can be divided into the following categories:

API validators: Such tools validate a given YAML manifest against a Kubernetes API server.
Built-in checkers: These tools come bundled with self-conscious checks for security, best practices, and so on.
Custom validators: Such tools allow you to write custom checks in multiple languages, such as Python and Javascript.

In this article, you’ll learn about six different tools:

Kubeval
Kube-score
Config-lint
Copper
Conftest
Polaris
Copy the code

Let’s Go ~~~

Reference service

Start by deploying a benchmark service for later test comparisons

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text"."hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo
Copy the code

The deployment is complete and verified as follows:

[root@k8s-node001 Test]# kubectl get po NAME READY STATUS RESTARTS AGE http-echo-57dd74545-rtxzm 1/1 Running 0 65s http-echo-57dd74545-trst7 1/1 Running 0 65s [root@k8s-node001 Test]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE HTTP-echo ClusterIP 10.102.221.64 < None > 5678/TCP 70s [root@k8s-node001 Test]# curl 10.102.221.64:5678 hello-worldCopy the code

The above YAML file can be deployed successfully, but does it follow best practices?

Let’s start.

kubeval

The premise of Kubeval is that any interaction with Kubernetes is through its REST API. Therefore, you can use the API schema to verify that a given YAML input conforms to the schema.

Install kubeval

wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
cp kubeval /usr/local/bin
Copy the code

Now let’s change base. Yaml, delete

  selector:
    matchLabels:
      app: http-echo
Copy the code

Base.yaml is then checked using kubeval

[root@k8s-node001 Test]# kubeval  base.yaml
WARN - base.yaml contains an invalid Deployment (http-echo) - selector: selector is required
PASS - base.yaml contains a valid Service (http-echo)
Copy the code

The output sees a WARN indicating that the selector is a required field and then resumes the selector and checks again

[root@k8s-node001 Test]# kubeval  base.yaml 
PASS - base.yaml contains a valid Deployment (http-echo)
PASS - base.yaml contains a valid Service (http-echo)
Copy the code

Check the PASS

The advantage of tools like Kubeval is that we can catch such errors early in the deployment cycle. Also, you don’t need to access the cluster to run the checks – they can be run offline. By default, Kubeval validates resources against the latest, unpublished Kubernetes API schema. See the official website for more usage details

kube-score

Kube-score analyzes the YAML inventory you provide and scores it against the built-in checks of the cluster. First open kube-score.com, a breath of fresh flair, then paste into the input box your written YAML listing, which you’ll examine in base.yaml

The analytical results are as follows

You can see the suggestions for this file, such as resource limits, mirroring tags, Pod network policies, and so on. Not bad, very useful tool…

Of course, kube-Score is not extensible, and you cannot add or adjust policies. If you want to write custom checks to conform to organizational policies, you can use one of four tools: Config-Lint, copper, ConfTest, or Polaris.

Config-lint

Config-lint is a tool for validating configuration files written with YAML, JSON, Terraform, CSV and Kubernetes listings.

Install the Config – lint

Wget tar ZXF - https://github.com/stelligent/config-lint/releases/download/v1.6.0/config-lint_Linux_x86_64.tar.gz config-lint_Linux_x86_64.tar.gz mv config-lint /usr/local/bin/Copy the code

Config-lint does not have built-in checks for Kubernetes listings. You must write your own rules to perform any validation. Rules are written as YAML files, called rule sets, and have the following structure:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
   # list of rules
Copy the code

Suppose we want to check that an image in deployment is always extracted from a trusted repository, such as kubeops.net/app:1.0. The config-Lint rule to enforce such checks is as follows:

- id: MY_DEPLOYMENT_IMAGE_TAG
  severity: FAILURE
  message: Deployment must use a valid image tag
  resource: Deployment
  assertions:
    - every:
        key: spec.template.spec.containers
        expressions:
          - key: image
            op: starts-with
            value: "kubeops.net/"
Copy the code

A complete set of rules looks like this:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
  - id: DEPLOYMENT_IMAGE_REPOSITORY
    severity: FAILURE
    message: Deployment must use a valid image repository
    resource: Deployment
    assertions:
      - every:
          key: spec.template.spec.containers
          expressions:
            - key: image
              op: starts-with
              value: "kubeops.net/"
Copy the code

If you want to test the check, you can save the rule set as check_image_repo.yaml. The check is then performed using config-lint

[root@k8s-node001 Test]# config-lint -rules check_image_repo.yaml base.yaml
[
  {
    "AssertionMessage": "Every expression fails: And expression fails: image does not start with kubeops.net/",
    "Category": "",
    "CreatedAt": "2020-11-02T08:28:43Z",
    "Filename": "base.yaml",
    "LineNumber": 0,
    "ResourceID": "http-echo",
    "ResourceType": "Deployment",
    "RuleID": "DEPLOYMENT_IMAGE_REPOSITORY",
    "RuleMessage": "Deployment must use a valid image repository",
    "Status": "FAILURE"
  }
]
Copy the code

You can see that Every expression fails. Now let’s change the images address to image: kubeops.net/http-echo and check again

[root@k8s-node001 Test]# config-lint -rules check_image_repo.yaml base.yaml
[]
Copy the code

The output is successful if it returns no errors.

Config-lint is a promising framework that lets you write custom checks for Kubernetes YAML listings using YAML DSL. But what if you want to express more complex logic and checks? Does YAML have a limit to this? What if you could express these checks in a real programming language? So Copper

Copper

Copper V2 is a framework that uses custom checks to validate checklists, just like config-Lint. However, Copper does not use YAML definition checking. Instead, tests are written in JavaScript, and Copper provides a library containing some basic helpers to help read Kubernetes objects and report errors.

Install the Copper

https://github.com/cloud66-oss/copper/releases/download/2.0.1/linux_amd64_2.0.1 linux_amd64_2 mv. 0.1 copper chmod + x copper mv copper /usr/local/bin/Copy the code

Like config-Lint, Copper does not provide built-in checking. Let’s customize a check to ensure that the deployment image tag must be non-latest. check_image_repo.js

$$.forEach(function($){
    if ($.kind === 'Deployment') {
        $.spec.template.spec.containers.forEach(function(container) {
            var image = new DockerImage(container.image);
            if (image.tag === 'latest') {
                errors.add_error('no_latest'."latest is used in " + $.metadata.name, 1)}}); }});Copy the code

Perform inspection

[root@k8s-node001 Test]# copper validate --in=base.yaml --validator=check_image_tag.js
Check no_latest failed with severity 1 due to latest is used in http-echo
Validation failed
Copy the code

Now change it to image: kubeops.net/http-echo:v1.0.0

[root@k8s-node001 Test]# copper validate --in=base.yaml --validator=check_image_tag.js
Validation successful
Copy the code

For more usage, see

Conftest

Conftest is a testing framework for configuring data that can be used to check and validate Kubernetes listings. The tests were written using the proprietary query language Rego.

Install Conftest

The tar wget https://github.com/open-policy-agent/conftest/releases/download/v0.21.0/conftest_0.21.0_Linux_x86_64.tar.gz -xzf conftest_0.21.0_linux_x86_64.tar. gz mv Conftest /usr/local/binCopy the code

Like config-Lint and copper, ConfTest does not have any built-in checks.

Create a new directory conftest-checks and a file called check_image_registry. Rego with the following contents:

package main

deny[msg] {

  input.kind == "Deployment"
  image := input.spec.template.spec.containers[_].image
  not startswith(image, "kubeops.net/")
  msg := sprintf("image '%v' doesn't come from kubeops.net repository", [image])
}
Copy the code

Yaml,image: docker. IO /http-echo conftest

[root@k8s-node001 Test]# conftest test --policy ./conftest-checks base.yaml FAIL - base.yaml - image 'docker. IO /http-echo:v1.0.0' doesn't come from kubeops.net repository 2 tests, 1 passed, 0 warnings, 1 failure, 0 exceptionsCopy the code

Yaml,image: kubeops.net/http-echo

[root@k8s-node001 Test]# conftest test --policy ./conftest-checks base.yaml 

2 tests, 2 passed, 0 warnings, 0 failures, 0 exceptions
Copy the code

For more usage, see

Polaris

As a final tool, Polaris can be installed inside a cluster or used as a command line tool to statically analyze Kubernetes listings. When run as a command-line tool, it contains several built-in checks, similar to Kube-Score, covering areas such as security and best practices. In addition, you can use it to write custom checks like config-Lint, copper, and ConfTest. In other words, Polaris combines the best of two categories: built-in and custom checkers.

Install Polaris, where only command line mode is installed

Wget tar ZXF - https://github.com/FairwindsOps/polaris/releases/download/1.2.1/polaris_1.2.1_linux_amd64.tar.gz Polaris_1. 2.1 _linux_amd64. Tar. Gz mv Polaris/usr/local/bin /Copy the code

[root@k8s-node001 Test]# Polaris audit –audit-path base. Yaml Polaris audit –audit-path base. You can take a closer look at the results.

  "PolarisOutputVersion": "1.0"."AuditTime": "0001-01-01T00:00:00Z"."SourceType": "Path"."SourceName": "base.yaml"."DisplayName": "base.yaml"."ClusterInfo": {
    "Version": "unknown"."Nodes": 0."Pods": 1."Namespaces": 0."Controllers": 1
  },
  "Results": [{"Name": "http-echo"."Namespace": ""."Kind": "Deployment"."Results": {},
      "PodResult": {
        "Name": ""."Results": {
          "hostIPCSet": {
            "ID": "hostIPCSet"."Message": "Host IPC is not configured"."Success": true."Severity": "danger"."Category": "Security"."tagNotSpecified": {
                "ID": "tagNotSpecified"."Message": "Image tag is specified"."Success": true."Severity": "danger"."Category": "Images"}}}]},"CreatedTime": "0001-01-01T00:00:00Z"}}]Copy the code

Alternatively, you can output only ratings

[root@k8s-node001 Test]# polaris audit --audit-path base.yaml --format score
66
Copy the code

A new check called checkImageRepo is defined below using the YAML code snippet: config_WITH_CUSTOM_check.yaml

checks:
  checkImageRepo: danger

customChecks:
  checkImageRepo:
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^kubeops.net/.+$
Copy the code

IO /http-echo:v1.0.0 let’s use our custom rules to perform the check

[root@k8s-node001 Test]# polaris audit --config config_with_custom_check.yaml --audit-path base.yaml
{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "base.yaml",
  "DisplayName": "base.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 1,
    "Namespaces": 0,
    "Controllers": 1
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {},
        "ContainerResults": [
          {
            "Name": "http-echo",
            "Results": {
              "checkImageRepo": {
                "ID": "checkImageRepo",
                "Message": "Image registry is not valid",
                "Success": false,
                "Severity": "danger",
                "Category": "Images"
              }
            }
          }
        ]
      },
      "CreatedTime": "0001-01-01T00:00:00Z"
    }
  ]
}
Copy the code

Yaml: “Image registry is not valid”, “Success”: false” kubeops.net/http-echo:v1.0.0 Perform the check again

[root@k8s-node001 Test]# polaris audit --config config_with_custom_check.yaml --audit-path base.yaml 
{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "base.yaml",
  "DisplayName": "base.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 1,
    "Namespaces": 0,
    "Controllers": 1
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {},
        "ContainerResults": [
          {
            "Name": "http-echo",
            "Results": {
              "checkImageRepo": {
                "ID": "checkImageRepo",
                "Message": "Image registry is valid",
                "Success": true,
                "Severity": "danger",
                "Category": "Images"
              }
            }
          }
        ]
      },
      "CreatedTime": "0001-01-01T00:00:00Z"
    }
  ]
}
Copy the code

“Message”: “Image registry is valid”,”Success”: true, For more usage, see

conclusion

Although there are many tools available to validate, score, and organize Kubernetes YAML files, it is important to have a healthy model to design and perform checks. For example, if you are considering a Kubernetes listing through a pipeline, kubeval might be the first step in the pipeline because it validates that the object definition conforms to the Kubernetes API schema. Once this check is successful, you can move on to more detailed testing, such as standard best practices and custom policies. Kube-score and Polaris are good choices. If you have complex requirements and want to customize the checking details, consider using copper, config-Lint, and ConfTest. While confTest and Config-Lint both use more YAML to define custom validation rules, Copper allows access to a true programming language, which makes it attractive. But should you use one of these and write all the checks from scratch? Or should you use Polaris and just write other custom checks? It’s all up to you, what’s right for you is best…

Kubernetes YAML Best Practices and Strategies

Reference service

kubeval

kube-score

Config-lint

Copper

Conftest

Polaris

conclusion

Related Posts

2021 year-end summary, cease to live, struggle

How does Monzo Bank manage 1600 microservices with K8s?

From launch click logo flow and talk about ActivityManagerService