takeaway

The last article covered an overview of shared storage and a simple case demonstration. I’m going to do PV and PVC in this video.

PV is an abstraction of shared storage on the underlying network. Shared storage is defined as a “resource”, for example, Node is also a resource that container applications can consume. PV is created and configured by an administrator and is directly related to the implementation of shared storage.

PVC is a user’s “request” for storage resources, just as A Pod consumes Node resources, PVC can consume PV resources. PVC can apply for specific storage space and access mode.

StorageClass marks the features and performance of storage resources. Administrators can define storage resources into certain categories, just like a Profile of a storage device. According to the description of StorageClass, you can intuitively learn the features of various storage resources, and apply for storage resources according to the requirements of applications for storage resources. Storage volumes can be created on demand.

PV

PV as a storage resource, you can set key information such as storage capability, access mode, storage type, reclamation policy, and back-end storage type.

ApiVersion: V1 kind: PersistentVolume metadata: name: PV1 spec: capacity: storage: 5Gi accessModes: # # access mode - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle recovery strategy storageClassName: missile NFS: path: / server: 172.17.0.2Copy the code

Kubernetes supports the following PV types:

◎ AWSElasticBlockStore: ElasticBlockStore provided by AWS public Cloud.

◎ AzureFile: File provided by Azure public cloud.

◎ AzureDisk: Disk provided by Azure public cloud.

◎ CephFS: An open source shared storage system

◎ Fibre Channel: Fibre Channel storage devices.

◎ FlexVolume: A plug-in storage mechanism.

◎ Flocker: An open source shared storage system

◎ GCEPersistentDisk: PersistentDisk provided by GCE public cloud.

◎ Glusterfs: An open source shared storage system

◎ HostPath: host directory, only used for standalone test.

◎ iSCSI: iSCSI storage device.

◎ Local: Local storage device, you can currently provide Local PV through specified Block devices, or through community-developed sig-storage-local-static-provisioner plug-in (github.com/kubernetes-…). To manage the life cycle of a Local PV.

◎ NFS: Network file system.

◎ Portworx Volumes: Storage services provided by Portworx

◎ Quobyte Volumes: Quobyte’s storage services.

◎ Ceph Block Device (RBD) : Ceph Block storage.

◎ ScaleIO Volumes: Storage devices for DellEMC.

◎ StorageOS: Storage service provided by StorageOS.

◎ VsphereVolume: Storage system provided by VMWare.

Key Configuration Parameters

1. Storage Capacity

Describes the capabilities of the storage device and supports the setting of storage space (storage= XX).

2. Volume Mode

VolumeMode =xx. The options include Filesystem and Block. The default value is Filesystem.

Currently, the following PV types support block device types:

AWSElasticBlockStore, AzureDisk, FC, GCEPersistentDisk, iSCSI, Local Volume, RBD (Ceph Block Device), VsphereVolume (Alpha)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Block
  fc:
    targetWWNs: ["url"]
    lun: 0
    readOnly: false
Copy the code

3. Access Modes

This parameter describes the permission of an application to access storage resources.

◎ ReadWriteOnce (RWO) : Indicates the read and write permission and can be mounted only by a single Node.

◎ ReadOnlyMany (ROX) : read-only permission to be mounted by multiple nodes.

◎ ReadWriteMany (RWX) : Indicates the read and write permission, which can be mounted by multiple nodes.

4. Storage category (Class)

The storageClassName parameter specifies the name of a StorageClass resource object. PVCS with a specific category can only be bound to PVCS that request this category. PVS that are not bound to any category can only be bound to PVCS that do not request any category.

5. Reclaim Policy

Through persistentVolumeReclaimPolicy fields,

◎ Retain: Retain data. Manual processing is required.

◎ Recycle space: Deletes files, for example, rm -rf /thevolume/*.

◎ Delete Delete: The Volume is deleted from the back-end storage connected to the PV

Clear internal volumes of EBS, GCE PD, Azure Disk, and OpenStack Cinder devices.

6. Mount Options

When mounting a PV to a Node, you may need to set additional mount parameters based on the characteristics of back-end storage. For details, see the mountOptions field in the PV definition.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  mountOptions:
  - hard
  - nolock
  - nfsvers=3
  gcePersistentDisk:
    fsType: ext4
    pdName: gce-disk-1
Copy the code

Currently, you can set mount parameters for the following PV types:

AWSElasticBlockStore, AzureDisk, AzureFile, CephFS, Cinder (OpenStack Block Storage), GCEPersistentDisk, Glusterfs, NFS, Quobyte Volumes, Ceph Block Device (RBD), StorageOS, VsphereVolume, iSCSI

7. Node Affinity

Limiting Volume access to certain nodes is set in the nodeAffinity field. Pods using these volumes are scheduled to nodes that meet the criteria.

This parameter applies only to Local storage volumes.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node
Copy the code

The stages of the PV life cycle

◎ Available: Not yet bound to a PVC

◎ Bound: Already Bound with a PVC.

The bound PVC has been deleted and the resources have been Released, but not recycled by the cluster.

◎ Failed: Automatic resource reclamation Failed.

PVC

PVC applies for storage resources, including storage space request, access mode, PV selection criteria, and storage type.

ApiVersion: v1 kind: PersistentVolumeClaim Metadata: name: PVC spec: accessModes: # ReadWriteOnce Resources: # select resource, 8Gi storage requests: storage: 8Gi storageClassName: slow # select selector: matchLabels: release "stable" matchExpressions: - {key: environment, operator: In, values: [dev]}Copy the code

The key configuration

1. Resources

Description Requests for storage resources. Currently, only request.storage is supported, that is, the size of storage space

2. AccessModes

This parameter describes the access permission on storage resources. The value is the same as that for PV

3. Volume Modes

Used to describe the DESIRED PV storage volume mode, including file systems and block devices.

4. PV Selector

Setting the Label Selector enables the PVC to filter the various PVS that already exist in the system.

The selection criteria can be set using matchLabels and matchExpressions, and if both fields are set, the Selector logic will be that both sets of criteria are matched

5. Storage Category (Class)

PVC can set the required backend storage category (specified by the storageClassName field) during definition to reduce dependence on the details of backend storage features. Only PVS with this Class configured can be selected by the system and bound to the PVC

PVC can also be set without Class requirements. If the storageClassName field is set to empty (storageClassName=””), it indicates that the PVC does not require a specific Class, and the system only matches and binds PVVS without a Class. PVC can also do not set the storageClassName field at all. In this case, the admission Controller named DefaultStorageClass will be performed according to whether the system is enabled

6. DefaultStorageClass is disabled

Equivalent to PVC, the storageClassName value is set to empty (storageClassName=””), that is, only PV with no Class can be matched and bound to it.

7. Enable DefaultStorageClass

The cluster administrator must have defined the default StorageClass. If DefaultStorageClass does not exist in the system, it is equivalent to not enabling DefaultStorageClass. If a default StorageClass exists, the system automatically creates a PV for the PVC (back-end storage using the default StorageClass) and binds them. Methods for the cluster administrator to set the default StorageClass, adding an annotation in StorageClass definition “StorageClass. Kubernetes. IO/is the default – class = true”. If the administrator defines multiple StorageClass as default, the system cannot create corresponding PVS for PVC because they are not unique.

PVCS and PVCS are limited by Namespace. PVCS are limited by Namespace when selecting PVCS. Only PVCS in the same Namespace can be bound to PVCS. Pod is also limited by Namespace when referencing PVC. Only PVCS in the same Namespace can be mounted to Pod.

When both Selector and Class are set, the system will select PVS that match both conditions.

In addition, if the resource supply is in dynamic mode, that is, the administrator does not define PV in advance and only gives it to the system to automatically create PV dynamically through StorageClass, the system will not be able to supply any storage resources for the PVC when it sets Selector.

With dynamic provisioning enabled, once the PVC is deleted by the user, the PVS bound to it are also deleted according to their default recycling policy “Delete”. If the PV (user data) needs to be retained, change the automatic PV reclamation policy from Delete to Retain after the dynamic binding is successful.

PV and PVC life cycle

Think of PV as the available storage resources and PVC as the demand for storage resources.

(1) Resource supply

K8s supports two resource provisioning modes: Static and Dynamic. The result of resource provisioning is the creation of good PVS.

Static mode: The cluster administrator manually creates many PVS and sets the features of the back-end storage when defining PVS.

Dynamic mode: The cluster administrator does not need to manually create a PV. Instead, the cluster administrator sets the StorageClass to describe the back-end storage and marks it as a certain type. In this case, the PVC is required to declare the storage type. The system automatically creates the PV and binds it to the PVC. A PVC can declare Class as “” to indicate that dynamic mode is prohibited for that PVC.

(2) Resource binding

After the PVC is defined, the system selects a PV that meets the REQUIREMENTS of the PVC based on its requirements for storage resources (storage space and access mode). Once it is found, it binds the PV to the DEFINED PVC, and the application can use the PVC. If there is no PVS in the system, the PVC will process Pending state until there is a PVS in the system. Once PV is bound to PVC, it will be monopolized by PVC and can no longer be bound with other PVC. When the storage space applied by PVC is less than that of PV, the entire PV space can be used by PVC, which may cause a waste of resources. If the resource provisioning is in dynamic mode, the system will automatically create a PV and bind it to the PVC after finding a suitable StorageClass for the PVC.

(3) Resource use

The Pod uses the Volume definition to mount the PVC to a path inside the container for use. The Volume is of type Persistent VolumeClaim and can be used exclusively after the container has mounted a PVC. Multiple PODS can be mounted to the same PVC.

volumes:
  - name: pv
    persistentVolumeClaim:
      claimName: pvc
Copy the code

(4) Resource release

After the storage resources are used up, you can delete the PVC. The PVCS bound to the PVC are marked as released, but they cannot be bound to other PVCS immediately. Data written through the previous PVC may remain on the storage device, and the PV can only be used again after it has been wiped.

(5) Resource recovery

For a PV, the administrator can set a reclamation policy to set how to deal with legacy data after the PVCS bound to it release resources. The storage space of PV can only be reclaimed for new PVC binding and use.

The principle of USING PVC for PV, PVC, StorageClass and Pod in static resource supply mode and dynamic resource supply mode is explained in the two figures respectively.

In static resource supply mode, the storage management mechanism is bound through PV and PVC and used by Pod

In dynamic resource supply mode, resources are dynamically bound through StorageClass and PVC (PV is automatically generated by the system), and storage management mechanism is used by Pod

StorageClass

StorageClass, as an abstract definition of storage resources, shields the details of back-end storage for PVC applications set by users. On the one hand, users’ attention to details of storage resources is reduced; on the other hand, administrators’ manual management of PV is reduced. The system automatically creates and binds PV, realizing dynamic resource supply.

StorageClass consists of the name, back-end storage provider (Privisioner), and back-end storage parameter configuration. Once created, the StorageClass cannot be modified. If you need to modify it, you can only delete it and rebuild it.

The following example defines a StorageClass named standard, whose provider is AWS -ebs, and whose parameter is set to type and value is Gp2:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
privisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
Copy the code

The key configuration

1. Provider (Privisioner)

Describes a provider of storage resources, also known as a back-end storage driver.

2. Parameters

Parameter Settings for the back-end storage resource provider. Different Provisioner include different parameter Settings. Some parameters may not be displayed, and the Provisioner will use its default value.

Set the default StorageClass

First, you need to enable the Admission Controller named DefaultStorageClass, which is added in the command line argument — Admission Control of Kube-Apiserver

--admission-control=... ,DefaultStorageClassCopy the code

Then, set an annotation in the StorageClass definition:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow
  annotations:
    storageclass.beta.kubernetes.io/is-default-class="true"
privisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
Copy the code

After kubectl create command is used to create StorageClass, you can see that StorageClass named gold is marked as default:

kubectl get sc
Copy the code

CSI storage mechanism

The Container Storage Interface (CSI) mechanism is used to establish a standard set of Storage management interfaces between Kubenetes and external Storage systems to provide Storage services for containers.

CSI stores the key components and deployment architecture of the plug-in

It consists of two components: CSI Controller and CSI Node

1, the CSI Controller

Manage and operate storage resources and storage from the perspective of storage service. In K8S, it is recommended to deploy as a single-instance Pod, which can be deployed using StatefulSet and Deployment controllers. Set the number of copies to 1 to ensure that only one controller instance can be run for a storage plug-in.

Two containers are deployed within this Pod:

(1) Secondary sidecar container for communication with Master (Kubernetes Controller Manager),

The sidecar container can also contain extenal-Attacher and Extenal-Privisioner, with the following functions:

◎ External-Attacher: Monitors changes to the VolumeAttachment resource object and triggers ControllerPublish and ControllerUnpublish operations on CSI endpoints.

◎ External-provisioner: Monitors changes to the PersistentVolumeClaim resource object, triggering the CreateVolume and DeleteVolume operations on CSI endpoints

(2) CSI Driver storage Driver container, provided by a third party, needs to realize the above interface

The two containers use local sockets (Unix Domain Sockets (UDS) and communicate with each other using the gPRC protocol. The SIDecar container calls the CSI interface of the CSI Driver container through sockets. The CSI Driver container is responsible for specific volume storage operations.

2, CSI Node

DaemonSet is recommended for managing and operating volumes on hosts (nodes), with each Node running a Pod.

Deploy the following two containers on this Pod:

(1) Node-driver-Register, an auxiliary sidecar container that communicates with Kubelet, is mainly used to register storage drivers in Kubelet.

(2) CSI Driver storage Driver container, provided by a third-party storage provider, whose main function is to receive calls from Kubelet, needs to implement a series of CSI interfaces related to Node. Examples include the NodePublishVolume interface (used to mount volumes to a target path within the container), the NodeUnpublishVolume interface (used to unload volumes from the container), and so on.

The Node-driver-Register container communicates with Kubelet through a Unix Socket in the hostPath directory of the Node host. The CSI Driver container communicates with Kubelet through a Unix Socket in another hostPath directory on the Node host. At the same time, kubelet’s working directory (/var/lib/kubelet) needs to be mounted to the CSI Driver container. Used to manage the Volume (including mount and unmount) for pods.

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

I’m Liusy, a programmer who likes to work out.

For more dry goods and the latest news, please follow the official account: Ancient False God

If it is helpful to you, a point of attention is my biggest support!!