I just want to save a file. Why the trouble? - k8s Relationship between PV, PVC, and StorageClass

Docker

When we use Docker, it is relatively easy to set the Volume. We just need to map the path of the specified Volume in the container, and then use the path in the container.

Like this:

# tomcat
  tomcat01:
    hostname: tomcat01
    restart: always
    image: jdk-tomcat:v8
    container_name: tomcat8-1
    links:
      -  mysql:mysql
    volumes:
      - / home/soft/docker/tomcat/webapps: / usr/local/apache tomcat -- 8.5.39 / webapps
      - / home/soft/docker/tomcat/logs: / usr/local/apache tomcat - 8.5.39 / logs
      - /etc/localtime:/etc/localtime
    environment:
      JAVA_OPTS: -Dspring.profiles.active=prod
      TZ: Asia/Shanghai
      LANG: C.UTF-8
      LC_ALL: zh_CN.UTF-8
    env_file:
      - /home/soft/docker/env/tomcat.env
Copy the code

Why set Volume? Because of course we’re going to persist data, we’re going to store data on hard disk.

k8s

When you get to K8S, things get a little more complicated, and a bunch of concepts emerge:

Pv
Pvc
StorageClass
Provisioner
.

Forget the complicated concepts, I just want to save a file, is there an easy way?

Yeah, let’s go over the basics.

As we know, files in containers are temporarily stored on disk and are lost when the Container crashes. Kubelet will restart the container, but the container will restart in a clean state. So we use Volume to persist data.

Docker also has the concept of volumes, but it is only lightly and loosely managed. A Docker volume is a directory on disk or in another container. Docker provides volume drivers, but its functionality is very limited.

Kubernetes supports many types of volumes. Pod can use any number of volume types simultaneously.

Temporary volume types have the same lifetime as Pod, but persistent volumes can be longer than Pod. Kubernetes also destroys temporary volumes when Pod no longer exists; However, Kubernetes does not destroy persistent volumes. For any type of volume in a given Pod, data is not lost during container restart.

At the core of the volume is a directory that may hold data that can be accessed by containers in pods. The specific volume type adopted will determine how the directory is formed, what media is used to store data, and what is stored in the directory.

To use a volume, set it to the volume provided by Pod in the.spec.volumes field and declare the mounting position of the volume in the container in the.spec.containers[*]. VolumeMounts field. Each volume is mounted to a specified path in the mirror. A volume cannot be mounted to or hard linked to other volumes. Each container in a Pod configuration must independently specify the mount location for each volume.

From the above concept, we know that there are different types of Volume, including temporary and persistent. So let’s talk about the simple one first, which is to solve the need of “I just want to save a file, is there an easy way?”

hostPath

The hostPath volume can mount files or directories from the host node’s file system to your Pod. Here’s an example:

apiVersion: v1
kind: Pod
metadata:
  name: test-webserver
spec:
  containers:
  - name: test-webserver
    image: k8s.gcr.io/test-webserver:latest
    volumeMounts:
    - mountPath: /var/local/aaa
      name: mydir
    - mountPath: /var/local/aaa/1.txt
      name: myfile
  volumes:
  - name: mydir
    hostPath:
      # ensure that the directory where the file resides was created successfully.
      path: /var/local/aaa
      type: DirectoryOrCreate
  - name: myfile
    hostPath:
      path: /var/local/aaa/1.txt
      type: FileOrCreate
Copy the code

HostPath can simply solve the problem of storing files on the host.

But here’s a caveat:

HostPath volumes have many security risks. It is best to avoid using HostPath whenever possible. When a HostPath volume must be used, it should be limited to the required files or directories and mounted read-only.

Another limitation of using hostPath is that our Pod cannot drift randomly, it needs to be fixed to one node, because once it drifts to other nodes, there will be no corresponding data on the host computer. Therefore, when we use hostPath, we will use nodeSelector.

emptyDir

EmptyDir is also a more common storage type.

The hostPath shown above defines the host directory. EmptyDir is like specifying implicitly.

Kubernetes creates a temporary directory on the host that will later be bound to mount to the Volume directory declared by the container. The Pod container uses the volumeMounts field to declare which Volume to mount, and uses the mountPath field to define the Volume directory in the container

When a Pod is dispatched to a Node, the emptyDir volume is created and remains as long as the Pod is running on the Node. As the name suggests, the volume is initially empty. Although containers in pods may or may not mount the emptyDir volume in the same path, these containers can all read and write the same files in the emptyDir volume. When a Pod is removed from a node for some reason, the data in the emptyDir volume is also permanently deleted.

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: k8s.gcr.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
  volumes:
  - name: cache-volume
    emptyDir: {}
Copy the code

Kubectl describe: EmptyDir (a temporary directory that shares a pod’s lifetime)

. Containers: nginx: Container ID: docker://07b4f89248791c2aa47787e3da3cc94b48576cd173018356a6ec8db2b6041343 Image: Nginx: 1.8... Environment: <none> Mounts: /usr/share/nginx/html from nginx-vol (rw) ... Volumes: nginx-vol: Type: EmptyDir (a temporary directory that shares a pod's lifetime)Copy the code

PV and PVC

PersistentVolume (PV): indicates a PersistentVolume
PVC(PersistentVolumeClaim): Persistent volume declaration

The relationship between PV and PVC is similar to the relationship between interfaces and implementations in Java.

PVC is a statement of user storage. PVC is similar to Pod in that Pod consumes nodes and PVC consumes PV resources. Pod can request CPU and memory, while PVC can request specific storage space and access patterns. For real storage users, you don’t need to worry about the underlying storage implementation details, just use PVC directly.

PV is an abstraction of underlying shared storage and is created and configured by administrators. It is related to the implementation of underlying shared storage technologies, such as Ceph, GlusterFS, NFS, and hostPath, which are interconnected with shared storage through plug-in mechanisms.

Let’s look at an example:

For example, operations personnel can define such an NFS-type PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.2441.4.
    path: "/"
Copy the code

PVC describes the properties of persistent storage that Pod wants to use. For example, Volume storage size, read and write permissions, and so on.


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: manual
  resources:
    requests:
      storage: 1Gi

Copy the code

For user-created PVC to actually be used by the container, it must be bound to one of the eligible PVS.

The first condition is the SPEC fields for PV and PVC. For example, the size of PV storage must meet the requirements of PVC.
The second condition is that the storageClassName field in PV and PVC must be the same

After successfully binding PVC to PV, Pod can declare the use of this PVC in its own YAML file just as it would with a regular Volume type such as hostPath

apiVersion: v1
kind: Pod
metadata:
  labels:
    role: web-frontend
spec:
  containers:
  - name: web
    image: nginx
    ports:
      - name: web
        containerPort: 80
    volumeMounts:
        - name: nfs
          mountPath: "/usr/share/nginx/html"
  volumes:
  - name: nfs
    persistentVolumeClaim:
      claimName: nfs
Copy the code

The volumes of type hostPath and emptyDir we used earlier are not “persistent” and can either be cleaned up by Kubelet or “migrated” to other nodes. Therefore, in most cases, the implementation of persistent volumes relies on a remote storage service, such as remote file storage (e.g., NFS, GlusterFS), remote block storage (e.g., remote disks provided by the public cloud), and so on.

StorageClass

The way we managed PV manually was called Static Provisioning.

A large Kubernetes cluster is likely to have tens of thousands of PVCS, which means operations must create thousands of PVS beforehand. To make matters worse, as new PVCS continue to be submitted, o&S have to keep adding new PVS that meet the requirements, or the new Pod will fail because PVCS cannot bind to PV. In practice, this is almost impossible to do manually. So Kubernetes provides a mechanism for automatically creating PVS called Dynamic Provisioning.

The core of the Dynamic Provisioning mechanism is an API object called StorageClass. The StorageClass object is used to create the PV template.

Specifically, the StorageClass object defines the following two parts:

First, the properties of PV. For example, storage type, Volume size, and so on.
Second, the storage plug-ins needed to create this PV. For example, Ceph and so on.

With these two pieces of information, Kubernetes can find a StorageClass based on the PVC submitted by the user. Kubernetes then calls the storage plug-in declared by the StorageClass to create the required PV.

In the following example, PV is created automatically.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
# specify the storage class to be used. This storage class will automatically create PVS that meet the requirements
 storageClassName: fast
 resources:
    requests:
      storage: 30Gi
Copy the code

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
Copy the code

The StorageClass serves as the PV template. In addition, only the PV and PVC that belong to the same StorageClass can be bound together. Another important function of the StorageClass is to specify the Provisioner for PV. At this point, Kubernetes can automatically create PVS for you if your storage plug-in supports Dynamic Provisioning.

Local PV

Kubernetes relies on PV and PVC to implement a new feature, which is called: Local Persistent Volume, also known as Local PV.

Local PV implements a function very similar to hostPath plus nodeAffinity. For example, a Pod can declare the use of a Local PV, which is actually a Volume of hostPath type. If the directory corresponding to hostPath is already created on nodeA, then I need to add nodeAffinity=nodeA to Pod to use this Volume. In theory, this is possible, but in fact, we should never use a directory on a host as PV, because the storage behavior of the local directory is completely uncontrollable, and the disk on which it is located can be written at any time, or even cause the whole host to break down. Therefore, generally speaking, the storage medium corresponding to Local PV is an extra disk or block device mounted on the host computer, which can be considered as “one PV one disk”.

One big difference between Local PV and normal PV is that Local PV ensures that Pod is always correctly scheduled to the node where it is requested. For ordinary PV, Kubernetes always schedules Pod to a node. Then the Volume directory on the node is persisted, and then the binding mount between the Volume directory and the container is completed. However, for Local PV, the available disks on the node must be prepared in advance, because they may be mounted differently on different nodes, and some nodes may not even have such disks. Therefore, In this case, the scheduler must be able to know the relationship between all nodes and disks corresponding to Local PV, and then schedule pods based on this information. In fact, Volume distribution is considered during scheduling.

Example:

Create a PV for the local disk

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/vol1 
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node-1
Copy the code

Among them:

Lcal. path Write the corresponding disk path
The node must be specified using.spec.nodeAffinity
.spec.volumeMode can be FileSystem (Default) or Block
Make sure you run StorageClass first (the file written below)

Then write the StorageClass file

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Copy the code

Among them:

Provisioner is kubernetes. IO /no-provisioner. This is because local PV does not support Dynamic Provisioning, so there is no way to automatically create the PROVISIoner when the PVC is created
VolumeBindingMode is a WaitForFirstConsumer. WaitForFirstConsumer is a WaitForFirstConsumer. There are two WaitForFirstConsumer types: WaitForFirstConsumer and Immediate, which must be deferred.

Create another PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: local-storage

Copy the code

Note that storageClassName indicates the name of the storageClassName created by ourselves: local-storage

After applying this file, use kubectl get PVC command to see that its state is Pending. Although the pv matching is available at this time, it will not be bound, and it is still waiting.

And then we’ll write a POD to apply this PVC

kind: Pod
apiVersion: v1
metadata:
  name: example-pv-pod
spec:
  volumes:
    - name: example-pv-storage
      persistentVolumeClaim:
       claimName: example-local-claim
  containers:
    - name: example-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: example-pv-storage
Copy the code

In this way, a local PV is deployed on the pod, so that even if the pod is missing and it is created again on the node, the files written to it can be persisted in a specific location.

How to delete this PV must follow the process, otherwise it will fail to delete

Delete the pod using this PV
Remove the disk from node (one disk per PV)
Delete the PVC
Delete the pv

conclusion

In this article, we discuss several types of Kubernetes storage, including temporary storage such as hostPath, emptyDir, and real persistent storage. We also discuss related concepts, such as PVC, PV, StorageClass, etc. The following figure summarizes these concepts:

reference

Geek Time: An in-depth look at the Kubernetes course
Kubernetes. IO/useful/docs/con…
www.qikqiak.com/k8strain/st…
www.kubernetes.org.cn/4078.html
Haojianxun. Making. IO / 2019/01/10 /…

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

I just want to save a file. Why the trouble? – k8s Relationship between PV, PVC, and StorageClass

Docker

k8s

hostPath

emptyDir

PV and PVC

StorageClass

Local PV

conclusion

reference

I just want to save a file. Why the trouble? – k8s Relationship between PV, PVC, and StorageClass

Docker

k8s

hostPath

emptyDir

PV and PVC

StorageClass

Local PV

conclusion

reference

Related Posts

Spring isolation levels and propagation behavior

[Java] Java garbage collection mechanism

Python automatically finds articles you like — GitHub Project Introduction