Wan Hongming is a core contributor to KubeSphere, focusing on cloud native security.

KubeSphere is a container hybrid cloud management system for cloud native applications built on Kubernetes. It supports multi-cloud and multi-cluster management, provides full-stack automatic o&M capabilities, helps enterprise users simplify DevOps workflow, provides o&M friendly wizard operation interface, and helps enterprises quickly build a powerful and functional container cloud platform.

KubeSphere provides users with many of the features needed to build an enterprise-class Kubernetes environment, For example, multi-cloud and multi-cluster management, Kubernetes resource management, DevOps, application life cycle management, micro-service governance (service grid), log query and collection, service and network, multi-tenant management, monitoring alarms, events, and audit query, storage management, access control, and GPU Support, network policy, mirror warehouse management, and security management.

Thanks to the excellent architecture and design of Kubernetes, KubeSphere has adopted a more lightweight architecture model and flexibly integrated resources to further enrich the K8s ecosystem.

KubeSphere core architecture

The core architecture of KubeSphere is shown below:

There are three core components:

  • Ks-console front-end service component
  • Ks-api Server back-end service component
  • Ks-controller-manager Resource status maintenance component

KubeSphere’s back-end design follows the K8s declarative API style, where all actionable resources are abstracted as CustomResource as possible. Declarative apis are simpler to use than imperative apis and provide better abstraction, telling the program what to expect in the end (what to do) regardless of how to do it.

Typically, in declarative apis:

  1. Your API contains a relatively small number of objects (resources).
  2. Object defines the configuration information of an application or infrastructure.
  3. The object update operation frequency is low.
  4. People are usually required to read or write objects.
  5. The main operations on objects are CRUD style (create, read, update, and delete).
  6. No cross-object transaction support is required: API objects represent the expected state rather than the actual state.

Imperative apis are different from declarative apis. Here are some signs that your API may not be declarative:

  1. The client issues the “do this” command and then gets a synchronous response at the end of the operation.
  2. The client issues the “do this” command, gets an action ID, and then needs to determine whether the request completed successfully.
  3. You will compare your API to RPC.
  4. Store large amounts of data directly.
  5. Normal operations performed on objects are not CRUD style.
  6. Apis are not easy to model with objects.

Data synchronization and data persistence are realized by kube-Apiserver and ETCD, and the state of these resources is maintained by KS-Controller-Manager to achieve the consistency of the final state. If you’re familiar with K8s, you’ll understand the benefits of declarative apis, which are at the heart of KubeSphere.

For example, the configuration of pipeline, user credentials, user entities and alarm notifications in KubeSphere can be abstracted as resource entities. With the help of K8s mature architecture and tool chain, it can be easily combined with K8s to reduce the coupling between components and reduce the complexity of the system.

The core architecture of KS-Api Server

Ks-apiserver is the back-end component of KubeSphere core, which is responsible for data interaction between the front and back ends, proxy distribution of requests, authentication and authentication. The core architecture of KS-Apiserver is shown below:

The development of KS-APiserver uses the Go-restful framework, which can add multiple filters to the request link for dynamic interception of request and response, and realize authentication, authentication, audit logical forwarding and reverse proxy functions. The API style of KubeSphere also learns K8s mode as much as possible to facilitate the use of RBAC for permission control.

Decoupling through CRD + Controller greatly simplifies the integration with third-party tools and software.

The K8s community also provides a rich tool chain, with Controller-Runtime and Kubebuiler providing quick scaffolding for development.

API aggregation and permission control

Ks-apiserver can be extended to realize API aggregation and further realize function expansion, aggregation query and other functions. The API development process needs to follow the following specifications, in order to integrate with KubeSphere’s tenant system, resource system.

  • API polymerization
  • Access control
  • CRD + controller

API specification

Group by API group
/apis/{api-group}/{version}/{resources}
# sample
/apis/apps/v1/deployments
/kapis/iam.kubesphere.io/v1alpha2/users
# api core
/api/v1/namespaces

Use path to distinguish between different actions
/api/{version}/watch/{resources}
/api/{version}/proxy/{resources}/{resource}

Use path to differentiate between different resource levels
/kapis/{api-group}/{version}/workspaces/{workspace}/{resources}/{resource}
/api/{version}/namespaces/{namespace}/{resource}
Copy the code

The purpose of the specification API:

  1. Better abstraction of resources. Object abstraction is better for declarative apis
  2. Better API management, version, grouping, stratification, more convenient API expansion
  3. Better integration with access control, can easily from the request for metadata, apigroup, scope, version, the verb

Access control

The core of KubeSphere permission control is RBAC role-based access control.

Key objects are: Role, User, and RoleBinding.

A Role defines the resources that a Role can access.

Roles are divided according to resource layers. Cluster Role, Workspace Role, and Namespace Role Define the resources that can be accessed by a role at the current layer.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: role-grantor
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["rolebindings"]
  verbs: ["create"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterroles"]
  verbs: ["bind"]
  # ignoring resourceNames means that any ClusterRole binding is allowed
  resourceNames: ["admin"."edit"."view"]
- nonResourceURLs: ["/healthz"."/healthz/*"] The '*' in # nonResourceURL is a global wildcard
  verbs: ["get"."post"]
Copy the code

RoleBinding Binds a role to a Subject. Principals can be groups, users, or service accounts.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-grantor-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: role-grantor
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: user-1
Copy the code

CRD + controller

Custom Resource is an extension of the Kubernetes API, which can be dynamically registered to expand the K8s API. Users can use Kubectl to create and access objects in it, just as they do with built-in resources.

Resources are abstracted using CRD and maintained using Controller to monitor resource changes. The core of controller is Reconcile. As in Reconcile, resource states are maintained in a passive, timed manner until a declared state is reached.

Using the User resource as an example, we can define the CRD of the structure to abstract the User:

apiVersion: iam.kubesphere.io/v1alpha2
kind: User
metadata:
  annotations:
    iam.kubesphere.io/last-password-change-time: "2021-05-12T05:50:07Z"
  name: admin
  resourceVersion: "478503717"
  selfLink: /apis/iam.kubesphere.io/v1alpha2/users/admin
  uid: 9e438fcc-f179-4254-b534-e913dfd7a727
spec:
  email: [email protected]
  lang: zh
  description: 'description'
  password: $2a$10$w312tzLTvXObnfEYiIrk9u5Nu/reJpwQeI66vrM1XJETWtpjd1/q2
status:
  lastLoginTime: "2021-06-08T06:37:36Z"
  state: Active
Copy the code

The corresponding API is:

# to create
POST /apis/iam.kubesphere.io/v1alpha2/users

# remove
DELETE /apis/iam.kubesphere.io/v1alpha2/users/{username}

# modified
PUT /apis/iam.kubesphere.io/v1alpha2/users/{username}
PATCH /apis/iam.kubesphere.io/v1alpha2/users/{username}

# query
GET /apis/iam.kubesphere.io/v1alpha2/users
GET /apis/iam.kubesphere.io/v1alpha2/users/{username}
Copy the code

Ks-apiserver is responsible for writing this data to the K8s and synchronizing it to each copy by the Informer.

Ks-controller-manager monitors data changes to maintain resource status. Create a user, for example, through the POST/apis/iam kubesphere. IO/v1alpha2 / users create a user, the user synchronization controller for the user resource condition.

func (c *userController) reconcile(key string) error {
	// Get the user with this name
	user, err := c.userLister.Get(key)

        iferr ! =nil {
            // The user may no longer exist, in which case we stop
            // processing.
            if errors.IsNotFound(err) {
                utilruntime.HandleError(fmt.Errorf("user '%s' in work queue no longer exists", key))
                return nil
            }
            klog.Error(err)
            return err
        }

	ifuser, err = c.encryptPassword(user); err ! =nil {
		klog.Error(err)
		return err
	}

	ifuser, err = c.syncUserStatus(user); err ! =nil {
		klog.Error(err)
		return err
	}

	// synchronization through kubefed-controller when multi cluster is enabled
	if c.multiClusterEnabled {
		iferr = c.multiClusterSync(user); err ! =nil {
			c.recorder.Event(user, corev1.EventTypeWarning, controller.FailedSynced, fmt.Sprintf(syncFailMessage, err))
			return err
		}
	}

	c.recorder.Event(user, corev1.EventTypeNormal, successSynced, messageResourceSynced)
	return nil
}
Copy the code

Complex logic is handled by controllers through declarative apis for easy decoupling. It can be easily integrated with other systems and services, for example:

/apis/devops.kubesphere.io/v1alpha2/namespaces/{namespace}/pipelines
/apis/devops.kubesphere.io/v1alpha2/namespaces/{namespace}/credentials
/apis/openpitrix.io/v1alpha2/namespaces/{namespace}/applications
/apis/notification.kubesphere.io/v1alpha2/configs
Copy the code

Corresponding permission control policies:

Define a role that can add, delete, change, or query user resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: user-manager
rules:
- apiGroups: ["iam.kubesphere.io"]
  resources: ["users"]
  verbs: ["create"."delete"."patch"."update"."get"."list"]
Copy the code

Define a role that can create pipeline resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: devops-manager
rules:
- apiGroups: ["devops.kubesphere.io"]
  resources: ["pipelines"]
  verbs: ["create"."delete"."patch"."update"."get"."list"]
Copy the code

This article is published by OpenWrite!