K8s uses the new NFS Provisioner to configure subdir

Wechat public account: operation and development story, author: Jiang Zong

background

There is nothing new about NFS as volume storage in K8S, which is one of the simplest and most accessible types of file storage. Recently there was a requirement to use NFS storage in K8S, which was recorded as follows, and there are also some SAO operations and pits encountered in the process, which were also recorded as follows.

Access the NFS Provisioner GitHub repository and you will be notified that the repository has been personally archived and is read-only. Old NFS address: github.com/kubernetes-…

The Deprecated repository has been removed to another GitHub address and is no longer updated. The Deprecated repository has been removed to the specified repository.

New Warehouse address:Github.com/lorenzofare…Found one more feature in the updated repository than in the old repository: added a parameterpathPatternIn effect, you can configure the subdirectory of PV by setting this parameter.

Out of curiosity to deploy the new NFS, the following YAML configuration files can be found in the deploy directory in your project. My configuration here has been slightly changed according to my environment, such as the IP address of the NFS service. You can change your NFS server address and path as required. This practice is on K8S 1.19.0

NFS client – provisioner deployment

class.yaml

$ cat class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
Copy the code

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: K8s. GCR. IO/sig - storage/NFS - subdir - external - provisioner: v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 172.1633.4.
            - name: NFS_PATH
              value: /
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.1633.4.
            path: /
Copy the code

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"."list"."watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get"."list"."watch"."create"."delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get"."list"."watch"."update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get"."list"."watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create"."update"."patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get"."list"."watch"."create"."update"."patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: kube-system
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
Copy the code

Note: If the image cannot be pulled, you can pull the image from a foreign machine and import it to RBAC. There is no need to change the class. Yaml file when configuring subdirectories, as described below

Create all resource files

kubectl apply -f class.yaml -f deployment.yaml -f rbac.yaml
Copy the code

Create a PVC with a simple example

$ cat test-pvc-2.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc-2
  namespace: nacos
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

$ cat test-nacos-pod-2.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos-c1-sit-tmp-1
  labels:
    appEnv: sit
    appName: nacos-c1-sit-tmp-1
  namespace: nacos
spec:
  serviceName: nacos-c1-sit-tmp-1
  replicas: 3
  selector:
    matchLabels:
      appEnv: sit
      appName: nacos-c1-sit-tmp-1
  template:
    metadata:
      labels:
        appEnv: sit
        appName: nacos-c1-sit-tmp-1
    spec:
      dnsPolicy: ClusterFirst
      containers:
      - name: nacos
        image: www.ayunw.cn/library/nacos/nacos-server:1.4.1
        ports:
        - containerPort: 8848
        env:
        - name: NACOS_REPLICAS
          value: "1"
        - name: MYSQL_SERVICE_HOST
          value: mysql.ayunw.cn
        - name: MYSQL_SERVICE_DB_NAME
          value: nacos_c1_sit
        - name: MYSQL_SERVICE_PORT
          value: "3306"
        - name: MYSQL_SERVICE_USER
          value: nacos
        - name: MYSQL_SERVICE_PASSWORD
          value: xxxxxxxxx
        - name: MODE
          value: cluster
        - name: NACOS_SERVER_PORT
          value: "8848"
        - name: PREFER_HOST_MODE
          value: hostname
        - name: SPRING_DATASOURCE_PLATFORM
          value: mysql
        - name: TOMCAT_ACCESSLOG_ENABLED
          value: "true"
        - name: NACOS_AUTH_ENABLE
          value: "true"
        - name: NACOS_SERVERS
          value: nacos-c1-sit-0.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848 nacos-c1-sit-1.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848 nacos-c1-sit-2.nacos-c1-sit-tmp-1.nacos.svc.cluster.local:8848
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 500m
            memory: 5Gi
          requests:
            cpu: 100m
            memory: 512Mi
        volumeMounts:
        - name: data
          mountPath: /home/nacos/plugins/peer-finder
          subPath: peer-finder
        - name: data
          mountPath: /home/nacos/data
          subPath: data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        storageClassName: "managed-nfs-storage"
        accessModes:
          - "ReadWriteMany"
        resources:
          requests:
            storage: 10Gi
Copy the code

View data in PVC and NFS storage

# ll
total 12
drwxr-xr-x 4 root root 4096 Aug  3 13:30 nacos-data-nacos-c1-sit-tmp-1-0-pvc-90d74547-0c71-4799-9b1c-58d80da51973
drwxr-xr-x 4 root root 4096 Aug  3 13:30 nacos-data-nacos-c1-sit-tmp-1-1-pvc-18b3e220-d7e5-4129-89c4-159d9d9f243b
drwxr-xr-x 4 root root 4096 Aug  3 13:31 nacos-data-nacos-c1-sit-tmp-1-2-pvc-26737f88-35cd-42dc-87b6-3b3c78d823da

# ll nacos-data-nacos-c1-sit-tmp-1-0-pvc-90d74547-0c71-4799-9b1c-58d80da51973
total 8
drwxr-xr-x 2 root root 4096 Aug  3 13:30 data
drwxr-xr-x 2 root root 4096 Aug  3 13:30 peer-finder
Copy the code

Configuring subdirectories

Remove the class.yaml created earlier, add the pathPattern parameter, and then regenerate the SC

$ kubectl delete -f class.yaml

$ vi class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
  # Add the following parameters
  pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"
Copy the code

Create a PVC to test whether the generated PV directory generates subdirectories

$ cat test-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-pvc-2
  namespace: nacos
  annotations:
    nfs.io/storage-path: "test-path-two" # not required, depending on whether this annotation was shown in the storage class description
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi
Copy the code

Create a resource

kubectl apply -f class.yaml -f test-pvc.yaml
Copy the code

Viewing the result On the machine where NFS is mounted, the generated directory is found to have been generated, and the subdirectory hierarchy is based on the rule of “namespace/annotation name”. It just conforms to the pathPattern rule defined in the StorageClass above.

# pwd
/data/nfs# ll nacos/
total 4
drwxr-xr-x 2 root root 4096 Aug  3 10:21 nacos-pvc-c1-pro

# tree -L 2 .. ├ ─ nacos ├ ── NacOS-PVC-C1-Pro 2 directories, 0 filesCopy the code

Provisioner high availability

Single points of failure should be avoided as much as possible in a production environment, so consider the provisioner configuration for the updated high availability architecture as follows:

$ cat nfs-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-system
spec:
  3 copies of POD are configured for high availability
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      imagePullSecrets:
        - name: registry-auth-paas
      containers:
        - name: nfs-client-provisioner
          image: www.ayunw.cn/nfs-subdir-external-provisioner:v4.0.2-31-gcb203b4
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
              Set high availability to allow elections
            - name: ENABLE_LEADER_ELECTION
              value: "True"
            - name: NFS_SERVER
              value: 172.1633.4.
            - name: NFS_PATH
              value: /
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.1633.4.
            path: /
Copy the code

Reconstruction of resources

kubectl delete -f nfs-class.yaml -f nfs-deployment.yaml
kubectl apply -f nfs-class.yaml -f nfs-deployment.yaml
Copy the code

Check whether provisioner high availability is in effect

# kubectl get po -n kube-system | grep nfs
nfs-client-provisioner-666df4d979-fdl8l    1/1     Running   0          20s
nfs-client-provisioner-666df4d979-n54ps    1/1     Running   0          20s
nfs-client-provisioner-666df4d979-s4cql    1/1     Running   0          20s
# kubectl logs -f --tail=20 nfs-client-provisioner-666df4d979-fdl8l -n kube-systemI0803 06:04:41.406441 1 Leaderelection.go: 147] Attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs... ^C#kubectl logs -f --tail=20 -n kube-system nfs-client-provisioner-666df4d979-n54psI0803 06:04:41.961617 1 Leaderelection.go: 147] Attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs... ^C [root@qing-core-kube-master-srv1 nfs-storage]# kubectl logs -f --tail=20 -n kube-system Nfs-client-provisioner-666df4d979-s4cql I0803 06:04:39.574258 1 Leaderelection. go:242] Attempting to acquire leader lease kube-system/nfs-provisioner-baiducfs... I0803 06:04:39.593388 1 Leaderelection.Go :252] Successfully acquired lease Kube-system/NFS-provision-Baiducfs I0803 06:04:39.593519 1 event.go:278] event (v1.objectreference {Kind:"Endpoints", Namespace:"kube-system", Name:"nfs-provisioner-baiducfs", UID:"3d5cdef6-57da-445e-bcd4-b82d0181fee4", APIVersion:"v1", ResourceVersion:"1471379708", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84 became leader I0803 06:04:39.593559 1 Controller.go :820 nfs-provisioner-baiducfs_nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84! I0803 06:04:39.694505 1 Controller. Go :869] Start provisioner controller nfs-provisioner-baiducfs_nfs-client-provisioner-666df4d979-s4cql_590ac6eb-ccfd-4653-9de5-57015f820b84!Copy the code

Successfully acquired lease Kube-system/NFS-provision-Baiducfs enables the third POD to be successfully elected as the leader node and the high availability takes effect.

An error

The following error was reported when the Describe POD discovery was encountered during operation

Mounting arguments: --description=Kubernetes transient mount for /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root --scope -- The mount -t NFS 172.16.41.7: / data/nfs_storage /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root Output: Running scope as unit: run-rdcc7cfa6560845969628fc551606e69d.scope mount: /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program. Warning FailedMount 10s kubelet, node1.ayunw.cn MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32 Mounting command: systemd-runCopy the code

Solution: The cause is that no NFS client is installed on the node to which POD is scheduled. You only need to install NFS client nfs-utils.

K8s uses the new NFS Provisioner to configure subdir

background

NFS client – provisioner deployment

class.yaml

deployment.yaml

rbac.yaml

Configuring subdirectories

Provisioner high availability

An error

Related Posts

Can naked resignation Still spank big company interviewer? I chose Meituan after I got the offer from Ali and Byte from Four sides!

Spark Data skew and its solution

Thread sharing of JVM runtime data areas