preface

Now, K8s (Kubernetes) has become the fact standard of container arrangement in the industry, and is promoting the popularity and landing of hot technologies such as cloud native application and micro-service architecture. With the choice of distributed architecture, the importance of network security becomes more and more obvious.

The main contents of this paper include the following parts :(1) introduce the security test of K8s container network; (2) Introduce the resource object NetworkPolicy that can be used for network isolation of K8s container; (3) explain several K8s NetworkPolicy experiments in detail; (4) Summary and outlook.

I hope this paper can help readers understand the cloud native network security issues and the technical application of K8s NetworkPolicy. If there are mistakes and omissions in the article, please help correct readers.

Two K8s container network security test

By default, the K8s underlying network is “fully connected”, meaning that all pods running in the same cluster can communicate freely. Therefore, the attack methods existing in traditional physical networks (such as ARP spoofing) and traditional individual applications (such as OWASP Top10) are still “targeted” for container networks. In addition, if an attacker controls some containers of the host machine (or can create containers through choreography), it can also launch penetration attacks against the host machine or other containers in cloud native scenarios.

As shown in Figure 2-1, an attacker can launch a horizontal attack based on a defective container. The attack and defense matrix shown in Figure 2-2 shows the common tactics used by attackers.

In addition, in the cloud native and microservice scenario, the east-west traffic surge of internal network, the boundary becomes more blurred and the container life cycle may be shorter, and other characteristics put forward a new test for “network isolation” and “application container and service protection”.

Introduction to K8s NetworkPolicy

This chapter introduces the concepts and configuration items of K8s NetworkPolicy.

In order to realize fine-grained container Network access isolation policy, THE NetworkPolicy mechanism has been developed by SCI-Network team since version 1.3 of K8s, and has been upgraded to a stable version.

NetworkPolicy is used to restrict the network access of the target Pod. By default, access is allowed to all pods, and access to pods can be restricted after a network policy is set to point to the Pod.

As shown in the NetworkPolicy function diagram drawn by the author in figure 2-1, NetworkPolicy has a strong customizability. It supports selecting a target Pod with “Label Label”, making IP network segments and namespaces for inbound and outbound traffic of the target Pod, and applying Pod as a port-level network access control policy.

Its Label based management mode fits the usage habit of K8s and helps users to quickly arrange the network isolation container at application level or micro-service level. Taking Figure 2-2 as an example, the choreographer can screen resource objects in the Test environment (Label:Test) or production environment (Label:Prod) by Label, and divide and conquer them.

In the process of K8s orchestration, only one NetworkPolicy can not be defined to complete the actual network isolation, and a Policy Controller is needed to implement the Policy. Policy controllers must be provided by third-party network components. Currently, The implementation of NetworkPolicy is supported by Calico, Cilium, Kube-Router, Romana, Weave- Net and other open source projects [1].

In order to help readers who are not familiar with K8s to supplement their preliminary knowledge, Pod, Service, and Namespace in K8s are introduced here. Readers who already understand these concepts are requested to skip these supplementary introductions.

Pod is the minimum dispatch unit of K8s. Each Pod has a separate IP and can be composed of one or more containers. (It is common to put “intimate” containers into a Pod so that they can access each other via localhost. For example, the classic “Nginx+ phP-fpm” combination in a Pod makes scheduling easier.) Containers in the same Pod are automatically assigned to the same physical machine or virtual machine. One of the basic tenets of the K8s network design model is that each Pod has an independent IP address, and it assumes that all pods are in a flat network space that can communicate with each other (which leads to insufficient network isolation between pods).

Because Pod is temporary, the “IP:port” of Pod also changes dynamically. This dynamic change involves a problem in K8S cluster: if a group of back-end Pod acts as a service provider and is invoked by a group of front-end Pod, how can the service caller automatically perceive the change of service provider? This introduces another core concept in K8S, Service.

Service is also the core resource object of K8S (each Service in K8S can be understood as a micro-service in “micro-service architecture”) Pod, Replication Controller (RC), Figure 2-3 shows the logical relationship between duplicate controller and Service.

Figure 2-3

As you can see from Figure 3, k8S Service defines an access point address for a Service through which the front-end application (Pod) accesses a cluster of instances behind it made up of Pod copies. Service and back-end Pod are seamlessly connected through Label Selector (Label key and value pairs can be attached to various object resources such as Node, Pod, Service, and RC). RC can declare that the number of copies of a Pod matches an expected value at any time.

The RC configuration file can attach a Label to the Pod you create; In the Service configuration file, you can select pods for the Service to be created based on Lable, as shown in Figure 2-4.

Figure 2-4

Figure 2-5 is the schematic diagram of K8s cluster. It can be seen from the figure that there are multiple PODS in each Node, and each Service may span multiple nodes or contain multiple services in a Node (Node can be understood as a server in the real world, A Service can be a single-machine or multi-machine Service. Node has no dependency relationship with a Service.

Figure 2-5 (3) Namespace A Namespace is an abstract collection of resources and objects. It can be used to isolate resources for multiple tenants. For example, a Namespace can be used to divide objects in the system into different project groups or user groups. Common pods, Services, Replication Controllers, and deployments all belong to a namespace (default), Node, persistentVolumes, etc. do not belong to any namespace[3].

3.2 K8s NetworkPolicy Configuration Description

The following is an example of the NetworkPolicy YAML format that you can customize:

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

Name: test-network-policy # Indicates the name of the network policy

Namespace: default # Specifies the name of the namespace

spec:

PodSelector: # The Pod range to which this network policy applies

MatchLabels: # In this example, the selection condition is pod role: db containing the "role=db" tagCopy the code

PolicyTypes: # network policyTypes

  • Ingress # inbound network restriction

  • Egress outbound network restriction

Ingress: # Inbound whitelist rule that allows access to target pods

  • From: # Perform network permit on all qualified client pods

    • IpBlock: # Based on the client IP range

      Cidr: 172.17.0.0/16

      except:

      • 172.17.1.0/24

    • NamespaceSelector: # Label based on the namespace in which the client Pod resides

      matchLabels:

      project: myproject
Copy the code

    • PodSelector: # Label based on the client Pod

      matchLabels:

      role: frontend
Copy the code

    ports:

    • protocol: TCP

      Port: 6379

Egress: # Define outbound whitelist rules that the target Pod allows access to

  • To: # Range of server IP addresses that target pods are allowed to access that meet the TO condition

    • ipBlock:

      Cidr: 10.0.0.0/24

    ports:

    • protocol: TCP

      Port: 5978 # and ports Specifies the port number

In addition to user-definable policies, K8s also sets some default global network policies at the Namespace level to facilitate administrators to set unified network policies for the entire Namespace. Such as:

(1) By default, no client is allowed to access all pods in the Namespace; (2) By default, any client is allowed to access all pods in the Namespace; (3) By default, all pods in the Namespace are prohibited from accessing external services; (4) All pods in the Namespace are allowed to access external services by default; (5) By default, any client is forbidden to access all pods in the Namespace and external services.

Four K8s NetworkPolicy experiment

This chapter will briefly introduce the experimental environment of “K8s NetworkPolicy application isolation”, and then introduce several application isolation experiments in detail, hoping to help readers understand the basic usage of K8s NetworkPolicy.

4.1 Experimental Environment The experimental environment in this document is a K8s cluster built with CentOS 7. The cluster has one Master Node and two nodes. The network plug-in Calico has been installed in the cluster.

Note: Calico is a pure three-layer network scheme based on BGP, which can be well integrated with OpenStack, Kubernetes, AWS, GCE and other platforms, and is the mainstream of enterprise applications [4]. Calico also provides rich network policies based on iptables, which not only implements Kubernetes NetworkPolicy, but also provides the function of container network reachability limitation. Refer to Resources [5] for the installation procedure.

4.2 Limiting Traffic to an Application Using “Access Label” The following uses an Nginx Pod that provides services as an example. Set different network access permissions for two client pods to allow pods containing Lable role= NginxClient to access the Nginx container. Denies access to containers that do not contain the Label. To achieve this requirement, the following steps are required.

(1) Create Nginx Pod and add Label “app= Nginx”. The contents of the choreographer file my-nginx.yaml are as follows.

apiVersion: v1

kind: Pod

metadata:

name: nginx

labels:

app: nginx
Copy the code

spec:

containers:

  • name: nginx

    image: nginx

Figure 4-1 shows that the Pod was created successfully.

Figure 4-1

(2) set the networkpolicy for the Nginx Pod created in step 1. The content of the arrangement file networkpolicy-allow-nginxclient.yaml is as follows:

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

name: allow-nginxclient

spec:

podSelector:

matchLabels:

  app: nginx
Copy the code

ingress:

- from:

  - podSelector:

      matchLabels:

        role: nginxclient

  ports:

  - protocol: TCP

    port: 80
Copy the code

The key information of the choreography file includes: the target Pod should contain Label “app=nginx”, the accessible client Pod should contain Label “role= nginxClient”, and the “Nginx Pod port” that the client is allowed to access is 80.

Run the following command to create the NetworkPolicy resource object:

kubectl create -f networkpolicy-allow-nginxclient.yaml

Figure 4-2 shows that the NetworkPolicy is created successfully.

Figure 4-2

(3) Create two client pods, one containing the Label “role= nginxClient” and the other without the Label. To verify the effect of the network policy, run commands in the two PODS to initiate network requests to port 80 of Nginx Pod.

Busybox-client2. yaml contains the following contents:

apiVersion: v1

kind: Pod

metadata:

name: busybox2

namespace: default

spec:

containers:

  • name: busybox2

    Image: busybox: 1.28.4

    command:

    • sleep

    • “3600”

    imagePullPolicy: IfNotPresent

restartPolicy: Always

Busybox-client4. yaml contains the following contents: busybox-client2.yaml contains the following contents:

apiVersion: v1 kind: Pod metadata: name: busybox4 namespace: default labels: role: nginxclient spec: containers:

  • Name: busybox4 image: busybox:1.28.4 Command:
    • sleep
    • “3600”

    imagePullPolicy: IfNotPresent

restartPolicy: Always

Create two pods “busybox2” and “busybox4” with the following command:

kubectl create -f busybox-client2.yaml -f busybox-client4.yaml

Log in to Pod “busybox2” with the following command to execute the command:

kubectl exec -ti busybox2 — sh

Try connecting to port 80 of the Nginx container with the following command:

Wget – timeout = 5 10.244.3.14

Connecting to 10.244.3.14 (10.244.3.14:80) wget: download timed out

Download timed out is displayed on the terminal, indicating that NetworkPolicy takes effect and the Pod of the client without Label role=nginxclient is denied access. Figure 4-3 shows the experiment result.

Figure 4-3

Log in to Pod “busybox4” with the following command to execute the command:

kubectl exec -ti busybox4 –sh

Try connecting to port 80 of the Nginx container:

/ # wget Connecting to 10.244.3.14 (10.244.3.14:80)

wget: can’t open ‘index.html’: File exists

Can’t open ‘index.html’: File exists is displayed on the terminal. This indicates that the Nginx Pod is successfully accessed, NetworkPolicy is in effect, and the client with Label role= NginxClient is allowed to access the Pod.

Figure 4-4 shows the experimental results. Figure 4-4

Note :(1) the screenshot of NetworkPolicy is shown in figure 4-5.

Figure 4-5

(2) Check the Labels experiment screenshot of Pod, as shown in Figure 4-6. Figure 4-6

(3) Figure 4-7 shows the screenshot of Deleting NetworkPolicy. Figure 4-7

4.3 Deny/Allow All Traffic to an application Because the experiment here is similar to 2.2, the experimental process will not be described. To deny all traffic to an application, see the following NetworkPolicy:

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

name: nginx-deny-all

spec:

podSelector:

matchLabels:

  app: nginx
Copy the code

ingress: []

The key information of the marshalling file above includes: the target Pod should contain Label “app: nginx”, ingress (the inbound whitelist rule that allows access to the target Pod) attribute value “[]” ([] stands for deny all).

To allow all traffic to an application, see NetworkPolicy:

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

name: nginx-allow-all

spec:

podSelector:

matchLabels:

  app: nginx
Copy the code

ingress:

  • {}

As you can see, this orchestration file is similar to the previous one, except that the ingress attribute value is “{}” ({} means allow all).

4.4 To reject all traffic destined for a namespace, perform the following steps: (1) Create two namespaces (NS_01 and NS_02).

kubectl create -f ns01.yaml -f ns02.yaml

namespace/ns01 created namespace/ns02 created

Create 2 pods (busybox-ns01 network namespace ns01; Specify the network namespace of busybox-ns02 as NS02, as shown in Figure 4-8.

Figure 4-8

(3) Test connectivity between Pod Busybox-NS01 and Busybox-NS02

The experimental results are shown in Figure 4-9. It can be seen from the figure that although the two PODS are in different namespaces, they are interconnected.

Figure 4-9

(4) compile networkpolicy networkpolicy-ns01-ingress-deny-all.yaml code for network namespace “ns01” :

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

name: networkpolicy-ns01-ingress-deny-all

spec:

podSelector: {}

policyTypes:

  • Ingress

To specify the network policy for network namespace NS01, run the following command:

kubectl apply -f networkpolicy-ns01-ingress-deny-all.yaml -n ns01

Figure 4-10 shows the test results. Figure 4-10

The figure shows that the network policy takes effect.

4.5 Allowed Traffic destined for an application on a network segment

(1) compile NetworkPolicy statement for network segment 10.244.0.0/16 as follows:

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: networkpolicy-ns01-ingress-allow-ip

spec:

podSelector: {}

policyTypes:

  • Ingress

ingress:

  • from:

    • ipBlock:

      Cidr: 10.244.0.0/16

(2) Specify the network policy for namespace NS01. Figure 4-11 shows the result. Figure 4-11

(3) Run the ping command to test the network connectivity

If NetworkPolicy is not deleted, the NetworkPolicy cannot be pinged, as shown in figure 4-12.

Figure 4 to 12

If NetworkPolicy is deleted, the ping succeeds, as shown in Figure 4-13. Figure 4-13

Summary and prospect

Through theoretical analysis and experimental exploration, it can be seen that THE resource object NetworkPolicy of K8s can achieve good results in dealing with the challenge of “micro-service network isolation” in container environment. Its advantages include but are not limited to: (1) Application level (Pod) protection; (2) more suitable for K8s usage habits (NetworkPolicy and Service resource objects find the target Pod through Lable); (3) Relatively comprehensive functions (can affect the following relations: “POD-POD” “POD-network segment” “POD-namespace”); (4) Good compatibility (many K8s network plug-ins provide support; Less constrained by user network technology selection; Compared with eBPF technology, it is less affected by the system kernel version).

Perhaps because of the above advantages, the network isolation of container security products in the market is often seen as the shadow of K8s NetworkPolicy, and the r&d personnel usually base on it to make technical innovation.

K8s NetworkPolicy also has some disadvantages, such as: (1) it can meet the isolation requirements of large-scale complex networks (various plug-ins usually use iptables to implement NetworkPolicy. If there is too much traffic, iptables can be overwhelmed); (2) The granularity of network isolation is not small enough; (3) Insufficient protection of application security and service security.

Generally speaking, K8s NetworkPolicy has advantages and disadvantages, but the defects do not cover the advantages. It can be used as an effective way to protect the network security of K8s cluster at present and in the future, and play an important role in ensuring the network security of K8s environment.

Six references [1] [2] “Kubernetes authoritative guide” take you understand Kubernetes core concepts, ten minutes: www.dockone.io/article/932 [3] the 2. Kubernetes community in Chinese noun explanation: Namespace: www.kubernetes.org.cn/ namesp… [4] the plug-in contrast K8S network: www.jianshu.com/p/d9330360f… [5] “kubeadm fast installation kubernetes v1.14.3 – the first chapter” : www.ziji.work/kubernetes/… [6] the cloud native security technology report: www.nsfocus.com.cn/html/2021/1… [7] the first domestic container on cloud ATT&CK offensive and defensive matrix, ali cloud power enterprise container safe landing “: developer.aliyun.com/article/765…