Containers are changing software development. As the new foundation for CI/CD, containers give you a fast, flexible way to deploy applications, apis, and microservices, and digital success depends on scalability and performance. However, containers and container choreography tools such as Kubernetes are also popular targets for hackers, and if they are not secured effectively, they can put your entire environment at risk. In this article, we’ll discuss best practices for security at each layer of the container stack.

It is important to understand what container safety means. As an application layer construct that relies on a shared kernel, containers can start up faster than VMS. Containers are also much more flexible than VMS in terms of configuration, and can perform everything from mounting storage volumes to disabling security features.

If you bypass container isolation and gain privileges on the host, the container can even run as root under hacker control, and then you’re in real trouble.

There are steps you can take to keep the bad guys out.

Layer 0 – Kernel Kubernetes is an open source platform designed to automate container deployment, extension, and orchestration, and properly configured to help you enhance security. At the kernel level, you can:

  • View the allowed system calls and remove any unnecessary or unwanted system calls
  • Use container sandboxes such as gVisor or Kata Containers to further limit system calls
  • Verify that your kernel version is patched and does not contain any existing vulnerabilities

Layer 1 – Container static Static container security focuses on the Docker image that will be used to build the container. First, reduce the attack surface of the container by removing unnecessary components, packages, and network utilities – the more streamlined the better. Consider using a distroless image that contains only your application and its runtime dependencies.

“Distroless (github.com/GoogleCloud… Distroless is an internal image builder used by Google, including Java images, Node, Python, etc. Distroless contains only the minimal images needed to run the service. It does not include package management tools, shell command lines, and other features. Next, make sure to extract images only from known trusted sources, and then scan them for vulnerabilities and configuration errors. Check their integrity in your CI/CD pipeline and build process, and validate and approve them before running to ensure that hackers haven’t installed any backdoors.

After you run the packaged image, it’s time to debug. Temporary containers will allow you to interactively debug a running container. Monitor for exceptions and suspicious system-level events that could be signs of sabotage, such as unexpected child processes, shells running inside containers, or accidental reads of sensitive files.

Falco, an open source runtime security tool, can help by using system calls to protect and monitor systems in the following ways:

  • Parsing Linux system calls from the kernel at run time
  • Declare flow for powerful rules engines
  • Alert when rules are violated

Layer 2 – Workload (Pod) A Pod is a unit of deployment within Kubernetes, a collection of containers that can share common security definitions and security-sensitive configurations. A Pod security context can set privileges and access controls for a given Pod, for example:

  • Privileged containers within containers
  • Group and user IDS for processes and volumes
  • Fine-grained Linux features (remove or add), such as sys.time
  • Sandbox and forced access control (Seccomp, AppArmor, SELinux)
  • File system Permissions
  • Privilege escalation

To enhance POD-level defenses, you can enforce strict Pod security policies to prevent dangerous workloads from running in the cluster. For greater flexibility and finer control over Pod security, consider the Open Policy Broker (OPA) implemented with the OPA Gatekeeper project.

Layer 3 – Network By default, all pods can talk unrestricted to all other pods in the cluster, which is very advantageous from the attacker’s point of view. If the workload is threatened, an attacker might try to probe the network and see what else they can access. The Kubernetes API is also accessible from within the Pod, providing another rich target.

Strict network control is a key part of container security -pod to POD, cluster to cluster, inside out and inside out. Use built-in network policies to isolate workload traffic and build fine-grained rule sets. Consider implementing a service grid to control traffic between workloads as well as entry/exit, for example by defining namespace-to-namespace traffic.

Application Layer (L7) Attacks – Server-side Request Forgery (SSRF) It’s no surprise that we’ve been hearing a lot about SSRF attacks lately. The SSRF is particularly difficult to defend against in a cloud native environment where apis talk to other apis. Webhooks are particularly notorious. Once the target is found, you can use SSRF upgrade privileges and scan the local Kubernetes network and components, and even dump data on Kubernetes metric endpoints to learn valuable information about your environment — and possibly take it over completely.

Application layer (L7) attacks — Remote Execution code (RCE) RCE is also dangerous in cloud native environments, making it possible to run system-level commands inside containers to grab files, access Kubernetes APIS, run image processing tools, and destroy entire machines.

The first rule of application layer (L7) defense protection is to follow safe coding and architectural practices that mitigate most of your risk. In addition, you can layer network defense in two directions: north and south to monitor and block malicious external traffic targeting your applications and apis; East-west to monitor traffic from container to container, cluster to cluster, and cloud to cloud to make sure you’re not vulnerable to damaged pods.

Layer 4 – Nodes Node level security is equally important. To prevent container outbreaks on VMS or other nodes, limit external administrative access to nodes and the control plane, and be aware of open ports and services. Keep the base operating system to a minimum and harden it with CIS benchmarks. Finally, be sure to scan and patch nodes like any other VM.

Layer 5 – Cluster component Kubernetes All sorts of things happen in a cluster and there is no all-in-one tool or policy to protect it. At higher levels, you should focus on:

  • API server — check your access control and authentication mechanisms and perform other security checks on dynamic Webhooks, Pod security policies, and public network access to the Kubernetes API;
  • Access Control – Enforce the minimum privilege principle for API server and Kubernetes Secret using role-based access Control (RBAC)
  • Service account token – To prevent unauthorized access, limit all secret permissions on the service account and store the service account token
  • Audit logging – Make sure it is enabled
  • Third party components – Pay attention to what is brought into the cluster to know what is running in the cluster and why
  • Kubernetes Version – Kubernetes can have bugs just like any other system and must be updated and patched in a timely manner.
  • Kubelet misconfiguration – Responsible for container choreography and interaction with the container runtime, Kubelet can be abused and attacked in an attempt to promote privileges.

The security of Kubernetes may seem daunting, but by following best practices at every layer of the stack, you can achieve the same high level of protection for the container and the environment. Therefore, you can enjoy the benefits of fast, agile development.