preface
Now, K8s (Kubernetes) has become the fact standard of container arrangement in the industry, and is promoting the popularity and landing of hot technologies such as cloud native application and micro-service architecture. With the choice of distributed architecture, the importance of network security becomes more and more obvious.
The main contents of this paper include the following parts :(1) introduce the security test of K8s container network; (2) Introduce the resource object NetworkPolicy that can be used for network isolation of K8s container; (3) explain several K8s NetworkPolicy experiments in detail; (4) Summary and outlook.
I hope this paper can help readers understand the cloud native network security issues and the technical application of K8s NetworkPolicy. If there are mistakes and omissions in the article, please help correct readers.
Two K8s container network security test
By default, the K8s underlying network is “fully connected”, meaning that all pods running in the same cluster can communicate freely. Therefore, the attack methods existing in traditional physical networks (such as ARP spoofing) and traditional individual applications (such as OWASP Top10) are still “targeted” for container networks. In addition, if an attacker controls some containers of the host machine (or can create containers through choreography), it can also launch penetration attacks against the host machine or other containers in cloud native scenarios.
As shown in Figure 2-1, an attacker can launch a horizontal attack based on a defective container. The attack and defense matrix shown in Figure 2-2 shows the common tactics used by attackers.
Introduction to K8s NetworkPolicy
This chapter introduces the concepts and configuration items of K8s NetworkPolicy.
In order to realize fine-grained container Network access isolation policy, THE NetworkPolicy mechanism has been developed by SCI-Network team since version 1.3 of K8s, and has been upgraded to a stable version.
NetworkPolicy is used to restrict the network access of the target Pod. By default, access is allowed to all pods, and access to pods can be restricted after a network policy is set to point to the Pod.
As shown in the NetworkPolicy function diagram drawn by the author in figure 2-1, NetworkPolicy has a strong customizability. It supports selecting a target Pod with “Label Label”, making IP network segments and namespaces for inbound and outbound traffic of the target Pod, and applying Pod as a port-level network access control policy.
In the process of K8s orchestration, only one NetworkPolicy can not be defined to complete the actual network isolation, and a Policy Controller is needed to implement the Policy. Policy controllers must be provided by third-party network components. Currently, The implementation of NetworkPolicy is supported by Calico, Cilium, Kube-Router, Romana, Weave- Net and other open source projects [1].
In order to help readers who are not familiar with K8s to supplement their preliminary knowledge, Pod, Service, and Namespace in K8s are introduced here. Readers who already understand these concepts are requested to skip these supplementary introductions.
Pod is the minimum dispatch unit of K8s. Each Pod has a separate IP and can be composed of one or more containers. (It is common to put “intimate” containers into a Pod so that they can access each other via localhost. For example, the classic “Nginx+ phP-fpm” combination in a Pod makes scheduling easier.) Containers in the same Pod are automatically assigned to the same physical machine or virtual machine. One of the basic tenets of the K8s network design model is that each Pod has an independent IP address, and it assumes that all pods are in a flat network space that can communicate with each other (which leads to insufficient network isolation between pods).
Because Pod is temporary, the “IP:port” of Pod also changes dynamically. This dynamic change involves a problem in K8S cluster: if a group of back-end Pod acts as a service provider and is invoked by a group of front-end Pod, how can the service caller automatically perceive the change of service provider? This introduces another core concept in K8S, Service.
Service is also the core resource object of K8S (each Service in K8S can be understood as a micro-service in “micro-service architecture”) Pod, Replication Controller (RC), Figure 2-3 shows the logical relationship between duplicate controller and Service.
Figure 2-3
As you can see from Figure 3, k8S Service defines an access point address for a Service through which the front-end application (Pod) accesses a cluster of instances behind it made up of Pod copies. Service and back-end Pod are seamlessly connected through Label Selector (Label key and value pairs can be attached to various object resources such as Node, Pod, Service, and RC). RC can declare that the number of copies of a Pod matches an expected value at any time.
The RC configuration file can attach a Label to the Pod you create; In the Service configuration file, you can select pods for the Service to be created based on Lable, as shown in Figure 2-4.
Figure 2-4
Figure 2-5 is the schematic diagram of K8s cluster. It can be seen from the figure that there are multiple PODS in each Node, and each Service may span multiple nodes or contain multiple services in a Node (Node can be understood as a server in the real world, A Service can be a single-machine or multi-machine Service. Node has no dependency relationship with a Service.
Figure 2-5 (3) Namespace A Namespace is an abstract collection of resources and objects. It can be used to isolate resources for multiple tenants. For example, a Namespace can be used to divide objects in the system into different project groups or user groups. Common pods, Services, Replication Controllers, and deployments all belong to a namespace (default), Node, persistentVolumes, etc. do not belong to any namespace[3].
3.2 K8s NetworkPolicy Configuration Description
The following is an example of the NetworkPolicy YAML format that you can customize:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
Name: test-network-policy # Indicates the name of the network policy
Namespace: default # Specifies the name of the namespace
spec:
PodSelector: # The Pod range to which this network policy applies
MatchLabels: # In this example, the selection condition is pod role: db containing the "role=db" tagCopy the codePolicyTypes: # network policyTypes
-
Ingress # inbound network restriction
-
Egress outbound network restriction
Ingress: # Inbound whitelist rule that allows access to target pods
-
From: # Perform network permit on all qualified client pods
-
IpBlock: # Based on the client IP range
Cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
-
NamespaceSelector: # Label based on the namespace in which the client Pod resides
matchLabels:
project: myproject Copy the code -
PodSelector: # Label based on the client Pod
matchLabels:
role: frontend Copy the code
ports:
-
protocol: TCP
Port: 6379
-
Egress: # Define outbound whitelist rules that the target Pod allows access to
-
To: # Range of server IP addresses that target pods are allowed to access that meet the TO condition
-
ipBlock:
Cidr: 10.0.0.0/24
ports:
-
protocol: TCP
Port: 5978 # and ports Specifies the port number
-
In addition to user-definable policies, K8s also sets some default global network policies at the Namespace level to facilitate administrators to set unified network policies for the entire Namespace. Such as:
(1) By default, no client is allowed to access all pods in the Namespace; (2) By default, any client is allowed to access all pods in the Namespace; (3) By default, all pods in the Namespace are prohibited from accessing external services; (4) All pods in the Namespace are allowed to access external services by default; (5) By default, any client is forbidden to access all pods in the Namespace and external services.
Four K8s NetworkPolicy experiment
This chapter will briefly introduce the experimental environment of “K8s NetworkPolicy application isolation”, and then introduce several application isolation experiments in detail, hoping to help readers understand the basic usage of K8s NetworkPolicy.
4.1 Experimental Environment The experimental environment in this document is a K8s cluster built with CentOS 7. The cluster has one Master Node and two nodes. The network plug-in Calico has been installed in the cluster.
Note: Calico is a pure three-layer network scheme based on BGP, which can be well integrated with OpenStack, Kubernetes, AWS, GCE and other platforms, and is the mainstream of enterprise applications [4]. Calico also provides rich network policies based on iptables, which not only implements Kubernetes NetworkPolicy, but also provides the function of container network reachability limitation. Refer to Resources [5] for the installation procedure.
4.2 Limiting Traffic to an Application Using “Access Label” The following uses an Nginx Pod that provides services as an example. Set different network access permissions for two client pods to allow pods containing Lable role= NginxClient to access the Nginx container. Denies access to containers that do not contain the Label. To achieve this requirement, the following steps are required.
(1) Create Nginx Pod and add Label “app= Nginx”. The contents of the choreographer file my-nginx.yaml are as follows.
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
Copy the codespec:
containers:
-
name: nginx
image: nginx
Figure 4-1 shows that the Pod was created successfully.
Figure 4-1
(2) set the networkpolicy for the Nginx Pod created in step 1. The content of the arrangement file networkpolicy-allow-nginxclient.yaml is as follows:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-nginxclient
spec:
podSelector:
matchLabels:
app: nginx
Copy the codeingress:
- from:
- podSelector:
matchLabels:
role: nginxclient
ports:
- protocol: TCP
port: 80
Copy the codeThe key information of the choreography file includes: the target Pod should contain Label “app=nginx”, the accessible client Pod should contain Label “role= nginxClient”, and the “Nginx Pod port” that the client is allowed to access is 80.
Run the following command to create the NetworkPolicy resource object:
kubectl create -f networkpolicy-allow-nginxclient.yaml
Figure 4-2 shows that the NetworkPolicy is created successfully.
Figure 4-2
(3) Create two client pods, one containing the Label “role= nginxClient” and the other without the Label. To verify the effect of the network policy, run commands in the two PODS to initiate network requests to port 80 of Nginx Pod.
Busybox-client2. yaml contains the following contents:
apiVersion: v1
kind: Pod
metadata:
name: busybox2
namespace: default
spec:
containers:
-
name: busybox2
Image: busybox: 1.28.4
command:
-
sleep
-
“3600”
imagePullPolicy: IfNotPresent
-
restartPolicy: Always
Busybox-client4. yaml contains the following contents: busybox-client2.yaml contains the following contents:
apiVersion: v1 kind: Pod metadata: name: busybox4 namespace: default labels: role: nginxclient spec: containers:
- Name: busybox4 image: busybox:1.28.4 Command:
- sleep
- “3600”
imagePullPolicy: IfNotPresent
restartPolicy: Always
Create two pods “busybox2” and “busybox4” with the following command:
kubectl create -f busybox-client2.yaml -f busybox-client4.yaml
Log in to Pod “busybox2” with the following command to execute the command:
kubectl exec -ti busybox2 — sh
Try connecting to port 80 of the Nginx container with the following command:
Wget – timeout = 5 10.244.3.14
Connecting to 10.244.3.14 (10.244.3.14:80) wget: download timed out
Download timed out is displayed on the terminal, indicating that NetworkPolicy takes effect and the Pod of the client without Label role=nginxclient is denied access. Figure 4-3 shows the experiment result.
Figure 4-3
Log in to Pod “busybox4” with the following command to execute the command:
kubectl exec -ti busybox4 –sh
Try connecting to port 80 of the Nginx container:
/ # wget Connecting to 10.244.3.14 (10.244.3.14:80)
wget: can’t open ‘index.html’: File exists
Can’t open ‘index.html’: File exists is displayed on the terminal. This indicates that the Nginx Pod is successfully accessed, NetworkPolicy is in effect, and the client with Label role= NginxClient is allowed to access the Pod.
Figure 4-4 shows the experimental results. Figure 4-4
Note :(1) the screenshot of NetworkPolicy is shown in figure 4-5.
Figure 4-5
(2) Check the Labels experiment screenshot of Pod, as shown in Figure 4-6. Figure 4-6
(3) Figure 4-7 shows the screenshot of Deleting NetworkPolicy. Figure 4-7
4.3 Deny/Allow All Traffic to an application Because the experiment here is similar to 2.2, the experimental process will not be described. To deny all traffic to an application, see the following NetworkPolicy:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: nginx-deny-all
spec:
podSelector:
matchLabels:
app: nginx
Copy the codeingress: []
The key information of the marshalling file above includes: the target Pod should contain Label “app: nginx”, ingress (the inbound whitelist rule that allows access to the target Pod) attribute value “[]” ([] stands for deny all).
To allow all traffic to an application, see NetworkPolicy:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: nginx-allow-all
spec:
podSelector:
matchLabels:
app: nginx
Copy the codeingress:
- {}
As you can see, this orchestration file is similar to the previous one, except that the ingress attribute value is “{}” ({} means allow all).
4.4 To reject all traffic destined for a namespace, perform the following steps: (1) Create two namespaces (NS_01 and NS_02).
kubectl create -f ns01.yaml -f ns02.yaml
namespace/ns01 created namespace/ns02 created
Create 2 pods (busybox-ns01 network namespace ns01; Specify the network namespace of busybox-ns02 as NS02, as shown in Figure 4-8.
Figure 4-8
(3) Test connectivity between Pod Busybox-NS01 and Busybox-NS02
The experimental results are shown in Figure 4-9. It can be seen from the figure that although the two PODS are in different namespaces, they are interconnected.
Figure 4-9
(4) compile networkpolicy networkpolicy-ns01-ingress-deny-all.yaml code for network namespace “ns01” :
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: networkpolicy-ns01-ingress-deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
To specify the network policy for network namespace NS01, run the following command:
kubectl apply -f networkpolicy-ns01-ingress-deny-all.yaml -n ns01
Figure 4-10 shows the test results. Figure 4-10
The figure shows that the network policy takes effect.
4.5 Allowed Traffic destined for an application on a network segment
(1) compile NetworkPolicy statement for network segment 10.244.0.0/16 as follows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: networkpolicy-ns01-ingress-allow-ip
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
-
from:
-
ipBlock:
Cidr: 10.244.0.0/16
-
(2) Specify the network policy for namespace NS01. Figure 4-11 shows the result. Figure 4-11
(3) Run the ping command to test the network connectivity
If NetworkPolicy is not deleted, the NetworkPolicy cannot be pinged, as shown in figure 4-12.
Figure 4 to 12
If NetworkPolicy is deleted, the ping succeeds, as shown in Figure 4-13. Figure 4-13
Summary and prospect
Through theoretical analysis and experimental exploration, it can be seen that THE resource object NetworkPolicy of K8s can achieve good results in dealing with the challenge of “micro-service network isolation” in container environment. Its advantages include but are not limited to: (1) Application level (Pod) protection; (2) more suitable for K8s usage habits (NetworkPolicy and Service resource objects find the target Pod through Lable); (3) Relatively comprehensive functions (can affect the following relations: “POD-POD” “POD-network segment” “POD-namespace”); (4) Good compatibility (many K8s network plug-ins provide support; Less constrained by user network technology selection; Compared with eBPF technology, it is less affected by the system kernel version).
Perhaps because of the above advantages, the network isolation of container security products in the market is often seen as the shadow of K8s NetworkPolicy, and the r&d personnel usually base on it to make technical innovation.
K8s NetworkPolicy also has some disadvantages, such as: (1) it can meet the isolation requirements of large-scale complex networks (various plug-ins usually use iptables to implement NetworkPolicy. If there is too much traffic, iptables can be overwhelmed); (2) The granularity of network isolation is not small enough; (3) Insufficient protection of application security and service security.
Generally speaking, K8s NetworkPolicy has advantages and disadvantages, but the defects do not cover the advantages. It can be used as an effective way to protect the network security of K8s cluster at present and in the future, and play an important role in ensuring the network security of K8s environment.
Six references [1] [2] “Kubernetes authoritative guide” take you understand Kubernetes core concepts, ten minutes: www.dockone.io/article/932 [3] the 2. Kubernetes community in Chinese noun explanation: Namespace: www.kubernetes.org.cn/ namesp… [4] the plug-in contrast K8S network: www.jianshu.com/p/d9330360f… [5] “kubeadm fast installation kubernetes v1.14.3 – the first chapter” : www.ziji.work/kubernetes/… [6] the cloud native security technology report: www.nsfocus.com.cn/html/2021/1… [7] the first domestic container on cloud ATT&CK offensive and defensive matrix, ali cloud power enterprise container safe landing “: developer.aliyun.com/article/765…