The author | quantico great tiger Alibaba technology experts

Takeaway: How to solve the security isolation problem of multi-tenant cluster is a key problem of cloud on the enterprise. This paper mainly introduces the basic concept and common application form of Kubernetes multi-tenant cluster. Based on Kubernetes native and ACK cluster existing security management ability to quickly achieve multi-tenant cluster related solutions.

What is a multi-tenant cluster?

The concept of “tenant” is not limited to the users of a cluster. It can include a set of workloads composed of computing, network, storage and other resources. In a multi-tenant cluster, different tenants need to be isolated within a cluster (which may be multiple clusters in the future) to avoid attacks by malicious tenants on other tenants to the greatest extent, and to ensure fair distribution of shared cluster resources among tenants.

In terms of the security of isolation, we can classify it into Soft multi-Tenancy and Hard multi-Tenancy.

  • Soft isolation is more targeted at multi-tenancy requirements within an enterprise. By default, no malicious tenant exists in this mode. The purpose of soft isolation is to protect services between internal teams and protect against possible security attacks.
  • And hard isolation for more is the foreign service providers of the services they offer, because the business form does not guarantee that business users in different tenants of the security background, our default think between the tenant and tenant and K8s system exists the possibility of attack each other, so here also need a more strict isolation as security.

The different application scenarios of multi-tenancy are described in more detail in the next section.

Multi-tenant application scenario

Here are two typical enterprise multi-tenant application scenarios and different isolation requirements:

Multi-tenant of an intra-enterprise shared cluster

In this scenario, all users of the cluster are from inside the enterprise, which is also the current usage mode of many K8s cluster customers. Due to the controllable identity of service users, the security risk of this business form is relatively controllable. After all, the boss can directly fire the malicious employees: According to the complexity of the internal personnel structure of the enterprise, we can logically isolate the resources of different departments or teams through the namespace and define the business personnel in the following roles:

  • Cluster administrator: has the ability to manage clusters, such as capacity expansion and node addition. Responsible for creating and assigning namespaces to tenant administrators; Responsible for all kinds of strategies (RAM/RBAC/networkpolicy/quota…). The CRUD.
  • Tenant administrator: has the read-only permission for the RAM of the cluster. Manage the RBAC configuration of tenants;
  • Intra-tenant user: uses K8s resources within the permission range in the corresponding namespace of the tenant.

After establishing access control based on user roles, we also need to ensure network isolation between namespaces, allowing only whitelisted cross-tenant application requests between different namespaces.

In addition, for application scenarios with high business security requirements, we need to limit the kernel capabilities of the application container, which can be combined with seccomp/AppArmor/SELinux and other strategic tools to limit the container’s run-time capabilities.

Of course, Kubernetes’ existing namespace single layer logical isolation is not enough to meet the isolation requirements of some large enterprise applications complex business models. We can focus on Virtual Cluster, which abstractions a higher level of tenant resource model to achieve more refined multi-tenancy management. This makes up for the lack of native namespace capabilities.

Multi-tenancy under the SaaS & KaaS service model

In the SaaS multi-lease scenario, tenants in Kubernetes cluster correspond to each service application instance in the SaaS platform and its own control plane. In this scenario, each service application instance on the platform can be divided into different namespaces. The end users of the service are not able to interact with Kubernetes’ control plane components. These end users can see and use the SaaS console itself, and they use the service or deploy the business through the upper customized SaaS control plane (as shown in the left figure below).

For example, a blogging platform is deployed to run on a multi-tenant cluster. In this scenario, the tenant is each customer’s blog instance and the platform’s own control plane. The control plane of the platform and each hosted blog will run in a different namespace. Customers create and delete blogs and update blog software versions through the platform’s interface, but have no way of understanding how the cluster works.

The KaaS multi-lease scenario is common with cloud service providers, where the services of the business platform are directly exposed to users under different tenants via the Kubernetes control plane. End users can use either the K8s native API or the CRDs/ Controllers interface extended by the service provider. To meet the basic requirements of isolation, tenants need to be logically isolated from each other in terms of access and network and resource quotas.

Different from the shared cluster within the enterprise, the end users here are all from the untrusted domain, among which malicious tenants inevitably execute malicious code on the service platform. Therefore, for the multi-tenant cluster under the SaaS/KaaS service model, we need a higher standard of security isolation. However, the existing native capabilities of Kubernetes are not enough to meet the needs of security. Therefore, we need kernel-level isolation such as security container at the time of container running to strengthen the security of tenants in this business form.

Implement a multi-tenant architecture

When planning and implementing multi-tenant clusters, we can first use Kubernetes’ own resource isolation layer, including the cluster itself, namespace, node, POD and container are different levels of resource isolation model. When the application loads of different tenants share the same resource model, there are security risks between them. Therefore, we need to control the resource domains that each tenant can access in the implementation of multi-tenancy, and at the resource scheduling level, ensure that the container processing sensitive information runs in relatively independent resource nodes as much as possible. From the perspective of resource cost, when there are loads from different tenants sharing the same resource domain, the risk of cross-tenant attacks can be reduced by run-time security and resource scheduling control policies.

Although the existing security and scheduling capabilities of Kubernetes are not sufficient to fully implement multi-tenancy isolation, in application scenarios such as intra-enterprise shared clusters, resource domains are isolated between tenants through namespaces. At the same time, policy models such as RBAC, PodSecurityPolicy and NetworkPolicy can control tenants’ limits on the scope and capability of resource access, and the combination of existing resource scheduling capability can provide considerable security isolation capability. As for service platforms such as SaaS and KaaS, we can achieve kernel-level isolation through the security sandbox container recently launched by Ali Cloud Container Service, which can avoid cross-tenant attacks by malicious tenants by means of escape to the greatest extent.

This section focuses on multi-tenant practices based on Kubernetes’ native security capabilities.

Access control

AuthN & AuthZ & Admission

The authorization of ACK cluster is divided into RAM authorization and RBAC authorization. RAM authorization is used to control the access of the management interface of the cluster, including THE CRUD permission on the cluster (such as cluster visibility, scaling, adding nodes, etc.). RBAC authorization is used for the access control of the Kubernetes resource model inside the cluster, which can be used to specify resources in the granular authorization of the namespace.

ACK authorization management provides predefined role templates of different levels for tenants, supports binding of multiple user-defined cluster roles, and supports batch user authorization. For more information about cluster-related access control authorization on ACK, see the help documentation.

NetworkPolicy

NetworkPolicy controls network traffic between service pods of different tenants and enables service access restrictions between tenants by whitelist.

You can configure NetworkPolicy on a container service cluster that uses the Terway networking plug-in, and here are some examples of policy configuration.

PodSecurityPolicy

PSP is a native cluster-dimension resource model of K8s. It can verify whether its run-time behavior meets the constraints of the corresponding PSP policy at the admission stage of POD creation request in Apiserver. For example, check whether pod uses host network, file system, designated port, PID namespace, etc., and restrict users in the tenant to open privileged container, limit the type of mounting disk, and force read-only mount. In addition, the PSP can add a SecurityContext to a POD based on its binding policy, including the container’s runtime UID, GID, and add or remove core capabilities.

For details on how to enable PSP Admission and use of related policies and permission binding, see here.

OPA

OPA (Open Policy Agent) is a powerful Policy engine that supports the decoupled Policy Decisions service and the community already has a relatively mature integration solution with Kubernetes. When the existing RBAC’s namespace granularity isolation cannot meet the complex security requirements of enterprise applications, OPA can provide object model-level fine-grained access policy control.

Meanwhile, OPA supports seven layers of NetworkPolicy definition and labels/ Annotation based cross-namespace access control, which can be an effective enhancement of K8s native NetworkPolicy.

Resource scheduling correlation

Resource Quotas & Limit Range

In a multi-tenant scenario, different teams or departments share cluster resources. Therefore, resource competition may occur. In this case, you need to set resource quotas for each tenant. ResourceQuota is used to limit the total resource Request and limit occupied by all pods in the tenant’s namespace. LimitRange is used to set the default resource Request and limit values for deploying pods in the tenant’s namespace. In addition, you can limit the tenant’s storage resource quota and object quantity quota.

Detailed guidance on resource quotas can be found here.

Pod Priority/Preemption

Since version 1.14, pod priority and preemption have been stable from beta, where POD priority identifies the priority of pods waiting in pending scheduling queues. When high-priority PODS cannot be scheduled due to insufficient node resources or other reasons, scheduler will try to expel low-priority pods to ensure that high-priority PODS can be scheduled and deployed.

In multi-tenant scenarios, priority and preemption Settings can be used to ensure the availability of important service applications in tenants. Pod Priority can also be used in conjunction with ResouceQuota to limit how much quota tenants can have under a given priority.

Dedicated Nodes

Note: Malicious tenants can circumvent policies enforced by node taint and tolerance mechanisms. The following instructions apply only to clusters of trusted tenants within an enterprise or clusters where tenants do not have direct access to the Kubernetes control plane.

By adding smudges to certain nodes in the cluster, you can use these nodes to designate a few tenants for exclusive use. In multi-tenant scenarios, for example, GPU nodes in a cluster can be reserved for use by service teams that need to use gpus in service applications. Cluster administrators can stain nodes with tags such as Effect: “NoSchedule”, and only pods configured with the corresponding tolerance Settings can be scheduled to that node.

Of course, malicious tenants can also access the node by adding the same tolerance configuration to their pods, so node smears and tolerance mechanisms alone cannot guarantee the exclusivity of the target node on an untrusted multi-tenancy cluster.

See here for information on how to use the node stain mechanism to control scheduling.

Protection of sensitive Information

secrets encryption at REST

In a multi-tenant cluster, different tenant users share the same SET of ETCD storage. In a scenario where end users can access the Kubernetes control plane, we need to protect data in secrets to avoid sensitive information leakage in the case of improperly configured access control policies. For this, you can refer to K8s’ native Secret encryption capability, which you can see here.

ACK also provides an open source solution for secrets encryption based on Ali Cloud KMS service, which can be seen here.

conclusion

Before implementing the multi-tenant architecture, you need to determine the application scenario, including the reliability of users and application loads in tenants and the corresponding security isolation degree. On this basis, the following are the basic requirements for secure isolation:

  • Enable Kubernetes cluster default security configuration: enable RBAC authentication and namespace-based soft isolation; Enable Secrets encryption capability to enhance sensitive information protection; Benchmarks include security configurations based on CIS Kubernetes benchmarks;
  • Enable NodeRestriction, AlwaysPullImages, PodSecurityPolicy, and other admission Controllers.
  • The PSP limits the privileged mode of pod deployment and controls the SecurityContext at which it is run.
  • Configuration NetworkPolicy;
  • Use Resource Quota & Limit Range to Limit the Resource usage of tenants.
  • When the application is running, follow the principle of minimization of permissions, as far as possible to reduce the pod container system permissions;
  • Log everything;
  • Interconnecting with the monitoring system to monitor container application dimensions.

For service models such as SaaS and KaaS, or when we cannot guarantee the trust of the users within the tenant, we need to adopt some stronger isolation measures, such as:

  • Use dynamic policy engines such as OPA for fine-grained access control at the network or Object level;
  • Security containers are used to achieve kernel-level security isolation at container runtime.
  • Comprehensive multi-lease isolation solution for monitoring, logging, storage and other services.

Note: the source of the picture.

“Alibaba Cloud originators pay close attention to technical fields such as microservice, Serverless, container and Service Mesh, focus on cloud native popular technology trends and large-scale implementation of cloud native, and become the technical circle that knows most about cloud native developers.”