Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

This article has participated in the “Digitalstar Project” and won a creative gift package to challenge the creative incentive money.

1 introduction

This is a K8S CSI system architecture diagram

2 Related Explanations

CSI is short for Container Storage Interface.

The purpose of CSI is to define an industry standard “container storage interface” to enable storage vendors to develop a CSI compliant plug-in that can work in multiple container choreography systems.

CSI components are typically deployed in containers, reducing environment dependency.

3 K8S object

3.1 PersistentVolume

Cluster-level resources, persistent storage volumes, created by the cluster administrator or External Provisioner.

The PV lifecycle is independent of the Pod that uses the PV, and the storage device details are stored in the.spec of the PV.

Recovery strategy

  • Retain: Retention policy. When the PVC is deleted, the PV and external storage resources still exist.
  • Delete: deletes the policy. When the PVC bound to the PV is deleted, the PV object is deleted from the K8S cluster and external storage resources are deleted.
  • Resycle (deprecated)

Include status:

  • Available
  • Bound
  • Released

3.2 PersistentVolumeClaim

Persistent storage volume declaration, namespace-level resources, created by the user or the StatefulSet controller (according to VolumeClaimTemplate).

PVC is similar to Pod in that Pod consumes Node resources and PVC consumes PV resources. A Pod can request specific levels of resources (CPU and memory), while a PVC can request specific volume sizes and Access Mode.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: nginx-pvc
spec:
   storageClassName: cbs
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 10Gi
Copy the code

Include status:

  • Pending
  • Bound

3.3 StorageClass

StorageClass is a cluster-level resource created by the cluster administrator. Defines the template for dynamically creating a PV.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cbs
parameters:
  type: cbs
provisioner: cloud.tencent.com/qcloud-cbs
reclaimPolicy: Delete
volumeBindingMode: Immediate
Copy the code

3.4 VolumeAttachment

Record PV mounting information, such as the node to which PV is mounted and the volume Plugin to which PV is mounted.

The AD Controller creates a VolumeAttachment, and the external-Attacher monitors the VolumeAttachment and mounts and unmounts storage based on its status attributes.

3.5 CSINode

CSINode records csi Plugin information (such as nodeId, driverName, topology information, etc.).

When Node Driver Registrar registers the CSI plugin with Kubelet, the Node Driver Registrar will create (or update) a CSINode object to record the CSI plugin information

ApiVersion: storage.k8s. IO /v1beta1 kind: CSINode Metadata: name: 10.1.2.11 spec: Drivers: -allocatable: count: 18 name: com.tencent.cloud.csi.cbs nodeID: ins-ig73rt44 topologyKeys: - topology.com.tencent.cloud.csi.cbs/zoneCopy the code

4 Storage Components and their functions

4.1 Volume Plugin

Expand the volume management capabilities of various storage types and integrate the operation capabilities of third-party storage with the K8S storage system.

Call an interface or command of a third-party storage device to create/delete, attach/detach, and mount/umount data volumes. Volume Plugin is invoked to create/delete, attach/detach, and mount/umount data volumes in the previous analysis components.

According to the location of source code, volume plugin can be divided into in-tree and out-of-tree

  • in-tree

In K8S source code internal implementation, and K8S release, management, update iteration is slow, poor flexibility.

  • out-of-tree

The code is independent of K8S and is implemented by storage vendors, including CSI and Flexvolume.

csi plugin

external plugin

External plugins include external-provisioner, external-attacher, external-resizer, external-Snapshotter, etc. The External Plugin assists the CSI Plugin component to perform storage-related operations. External Plugin is responsible for objects such as Watch PVC and volumeAttachment, and then invokes volume Plugin to complete storage operations. Such as the external-Provisioner Watch PVC object, then call the CSI Plugin to create the storage, and finally create the PV object; External-attacher watch volumeAttachment object, then call csi Plugin to attach/dettach; External-resizer watch PVC object, and then call csi Plugin to do storage expansion operations, etc.

4.2 kube controller — the manager

PV controller

Responsible for pv and PVC binding and life cycle management (such as creating/deleting underlying storage, creating/deleting PV objects, changing the state of PV and PVC objects).

The PV controller invokes the Volume Plugin (in-tree) to create or delete underlying storage and PV objects.

AD controller

The full name of AD Cotroller is Attachment/Detachment controller, which is mainly responsible for creating and deleting VolumeAttachment objects. Volume and call the plugin for storage devices Attach/Detach operation (the volume mount of data to a specific node node/remove mount from the specific node node), and update the node. The Status. VolumesAttached, etc.

The Attach/Detach operation logic of different volume plugins is different. For example, if the storage is used by the out-of-tree volume plugin, the Attach/Detach operation only changes the state of the VolumeAttachment object. Instead of actually mounting the data volume to the node or unmounting it from the node, the actual node storage mount/unmount operation is called by the volume Manager in Kubelet.

4.3 kubelet

volume manager

It mainly manages Attach/Detach of volumes (the function is the same as that of AD Controller, which component is controlled by kubelet startup parameter to perform the operation), mount/umount and other operations.

5 Process Analysis

5.1 Creating and Mounting a Vm

5.1.1 the in – tree

[1] User creates PVC

[2] PV Controller watch to create PVC, looking for a suitable PV binding

[3, 4] When no suitable PV is found, the Volume Plugin is called to create the volume and the PV object, which is then bound to the PVC object

[5] The user creates a POD to mount PVC

[6] Kube-scheduler watch to create a pod and find an appropriate node to schedule it

[7, 8] After Pod scheduling is completed, AD controller/ Volume Manager watch does not attach to the volume declared by Pod, and volume Plugin will be invoked to attach

[9] Volume Plugin Attaches the volume to the Pod node by attach operation, and becomes a device such as /dev/vdb

[10, 11] After attach operation is completed, the volume Manager watch does not mount the volume declared by POD, and the volume Plugin will be called to mount the volume

[12] Volume Plugin mounts the /dev/vdb device obtained in step [9] on node to the specified directory

5.1.2 the Out – tree

[1] User creates PVC

[2] PV Controller watch to create PVC, looking for a suitable PV binding. When looking for suitable PV, PVC object will be updated, add the annotation: volume. The beta. Kubernetes. IO/storage – provisioner, let began to start creating external – provisioner component object storage and PV operation

[3] External-provisioner watch for PVC creation Volume. Beta. Kubernetes. IO/storage – the value of the provisioner, judge whether to responsible for the create operation, call the csi – plugin ControllerServer to create storage, and create the PV object

[4] PV Controller Watch to PVC, find the appropriate PV (created in the previous step) to bind to it

[5] The user creates a Pod to mount PVC

[6] Kube-scheduler watch to create a Pod and find an appropriate node to schedule it

[7, 8] After POD scheduling is completed, the AD controller/ Volume Manager watch does not attach to the volume declared by POD, and cSI-Attacher will be called to attach. You’re actually just creating the volumeAttachement object

[9] External-Attacher component Watch to the creation of volumeAttachment object, call CSI-plugin for attach operation

[10] Csi-plugin ControllerServer attaches volume to a Pod node by attaching it to a device such as /dev/vdb

[11, 12] After attach operation is completed, the volume manager watch does not mount the volume declared by Pod, so cSI-mounter will be called to mount the volume

[13] Csi-mounter calls csi-plugin NodeServer to mount the /dev/vdb device obtained in step (10) on node node to the specified directory

The appendix

emptyDir

A volume of type emptyDir is created when a Pod is scheduled to a host, and containers within the same Pod can read and write from the same file in emptyDir. Once the Pod leaves the host, the data in emptyDir is permanently deleted.

So currently, EmptyDir volumes are mainly used as temporary Spaces, such as:

  • The Web server writes logs
  • TMP temporary directory for files

The steps are as follows:

First, you need to declare a volume named nginxData inside the POD

volumes:
  - name: nginxdata
    emptyDir: {}
Copy the code

Then, the volume can be hung in the container

VolumeMounts: # mountPath Indicates the mount point path of the volume in the container. - mountPath: /var/www/html name: nginxdataCopy the code

Yaml for testing:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-dp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-dp
  template:
    metadata:
      labels:
        app: test-dp
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.5
        volumeMounts:
          - mountPath: /var/www/html
            name: nginxdata
      volumes:
        - name: nginxdata
          emptyDir: {}
Copy the code

IO ~empty-dir directory nginxdata: /var/lib/kubelet/ Pods /< Pod uid>/volumes/kubernetes. IO ~empty-dir

If you log in to the container created by this Pod, you can also see a directory named /var/www/html, which is the same directory as the nginxdata directory on the host.

hostPath

The hostPath Volume mounts the Pod to directories or files on the host so that the container can be stored using the host’s high-speed file system.

The disadvantage is that in K8S, pods are dynamically scheduled on each node. When a Pod starts on the current node and stores files locally via hostPath, the next time it is scheduled to start on another node, the files stored on the previous node will not be available.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-dp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-dp
  template:
    metadata:
      labels:
        app: test-dp
    spec:
      containers:
      - name: nginx
        image: Nginx: 1.19.5
        volumeMounts:
          - mountPath: /var/www/html
            name: nginxdata
      volumes:
        - name: nginxdata
          hostPath:
            path: /data
Copy the code