The problem

There was a problem deploying ElasticSearch via Kubernetes today. Because ElasticSearch is a stateful application and requires persistent storage, there is no cloud storage, NFS, CEPHfs, etc on hand. Kubernetes provides the default yaml resource file for ElasticSearch to use emptydir.

Storage

The Elasticsearch StatefulSet will use the EmptyDir volume to store data. EmptyDir is erased when the pod terminates, here it is used only for testing purposes. Important: please change the storage to persistent volume claim before actually using this StatefulSet in your setup!

To put it bluntly, you need to be free to choose your own persistent storage. So it is necessary to use Local disk, Local PV mode, PV resource file as follows:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-holder-01
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /home/k8s/localpv  The directory on the node
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubernetes-node03
Copy the code

/home/k8s/localpv: /home/k8s/localpv: /home/k8s/localpv

Error: MountVolume.NewMounter Initialization failed for volume “local-pv-xxxxx” : Path /home/k8s/localPV does not exist

The /home/k8s/localpv path could not be found.

why

Because my Kubernetes cluster is a RKE cluster installed via Rancher, not a native installation via Kubeadm, the cluster components all run inside the Docker container, so Kubelet runs inside the container and cannot read the path on the Node Node.

The official FAQ also explains the problem

Volume does not exist with containerized kubelet

If your kubelet is running in a container, it may not be able to access the path on the host.

In order to allow the kubelet to access the path on the host, you must prefix hostDir with the prefix of the host filesystem in the kubelet container or mount the directory of the local volumes into the kubelet container at the same path.

For example, if the root filesystem of the host is mounted at /rootfs in the kubelet container, you need to prefix the hostDir with /rootfs. This requires recreating the local PV objects. You can delete them all and wait for them to be discovered again.

Another solution is to add a bind in the kubelet deployment configuration to mount the parent directory of local volumes into the kubelet container at the same path. This requires restarting the kubelet container.

For Rancher clusters, users must configure additional local volumes (or the parent directory) via Extra Binds.

Solutions are as follows:

services:
    kubelet:
      extra_binds:
        - "/host/dev:/dev"
        - "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins:z"
Copy the code