background

  • Cloudpods: Is our open source multi-cloud management platform that runs on Top of Kubernetes and includes a complete private cloud implementation.
  • Rook is a distributed storage orchestration system designed to provide storage solutions on Kubernetes. It does not provide storage itself, but provides an adaptation layer between Kubernetes and the storage system to simplify the deployment and maintenance of the storage system. The Ceph stores it supports are production-available states for Stable.
  • Ceph is an open source distributed storage system. Its main functions include RBD block storage and CephFS distributed file system storage.

The Cloudpods service runs on top of the Kubernetes cluster in a containerized manner. After deploying Cloudpods according to the deployment document/multi-node installation document, the environment has a complete Kubernetes cluster.

However, Cloudpods built-in private cloud virtual machines use local storage. This article mainly introduces how to use Rook to deploy Ceph cluster on computing nodes in Cloudpods Kubernetes cluster. The Ceph cluster managed by Rook is then exposed as a private cloud VIRTUAL machine that interconnects with Cloudpods.

Cloudpods built-in private cloud provides virtualization capabilities, Rook-managed Ceph provides distributed storage, and these services are containerized and run on Kubernetes. The nodes where Cloudpods run virtual machines are also called compute nodes, which are also Kubernetes nodes. As long as there are independent raw disks on the compute nodes, Rook can be used to deploy Ceph on the compute nodes. Combining these technologies can easily achieve a cloud native hyper-converged private cloud.

Environment to prepare

  • Cloudpods: Multi-node deployment versions higher than V3.6

    • 3 compute nodes with separate raw disks for Ceph (also used as storage nodes)

  • Kubernetes: v1.15.9 (Cloudpods default cluster)

  • Rook: v1.7 version

  • Operating system: CentOS 7

  • Kernel version: 3.10.0-1062.4.3. El7. Yn20191203. X86_64

    • If the kernel version recommended by CephFS is 4.17 or later, you can upgrade our official 5.4 kernel

IO /docs/ Rook /v… .

Deploy Ceph using Rook

Using Rook to deploy a Ceph cluster on an existing Cloudpods Kubernetes cluster, this assumes that a multi-node Cloudpods cluster has been deployed according to the deployment/multi-node installation document.

Node information

Suppose there are three nodes node-{0,1,2}, the disk information of each node is as follows, sd{b,c,d} are raw disks without partitions, reserved for Ceph:

$LSBLK NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT SDA 8:00931.5G 0 Disk ├─ SDA1 8:101m 0 Part ├─ SDA2 8:20512M 0 Part /boot └─sda3 8 0 0 0 part/SDB 8 0 0 0 0 0 0 disk SDC 8 0 0 0 0 0Copy the code

Use Kubectl get Nodes to see the nodes that are already in the Kubernetes cluster:

$kubectl get Nodes NAME STATUS ROLES AGE VERSION Cloudbox Ready master 34D v1.15.9- bea. 0 node-0 Ready < None > 12d 0 node-1 Ready < None > 11d v1.15.9-beta.0 node-2 Ready < None > 11d v1.15.9-beta.0Copy the code

Then label the corresponding node role=storage-node:

$kubectl label node node-0 role=storage-node $kubectl label node node-1 role=storage-node $kubectl label node $kubectl get nodes -l role NAME STATUS ROLES AGE VERSION role cloudbox Ready Master 34D v1.15.9-beta.0 node-0 Ready <none> 12D v1.15.9-beta.0 storage-node node-1 Ready < None > 11d v1.15.9-beta Storage-node node-2 Ready < None > 11d v1.15.9-beta.0 storage-nodeCopy the code

Also, run the climc host-list command (climc is the cloud platform’s command-line tool) and you can see that these three nodes are added to the cloud platform as computing nodes for Cloudpods:

$ climc host-list +--------------------------------------+----------+-------------------+----------------+--------------+----------------- ------------+---------+---------+-------------+----------+-----------+------------+------------+--------------+--------- ---+ | ID | Name | Access_mac | Access_ip | Ipmi_Ip | Manager_URI | Status | enabled | host_status | mem_size | cpu_count | node_count | sn | storage_type | host_type | +--------------------------------------+----------+-------------------+----------------+--------------+----------------- ------------+---------+---------+-------------+----------+-----------+------------+------------+--------------+--------- -- -- -- + | 0 d8023ad a3c ebf9-4-8294 - fd170f4ce5c6 | node - 0 | 38: ea: a7:8 d: 94:78 | 172.16.254.127 | 172.16.254.2 | https://172.16.254.127:8885 | running | true | online 32 | | 128695 | 2 | 6 cu3505m2g | rotate | hypervisor | | C02470b3 46 f7-852-9666 - e - 9 bda8074a72e | | node - 1 ec: f4: bb: d7: c4: e0 | 172.16.254.124 | 172.16.254.5 | https://172.16.254.124:8885 | running | true | online 48 | 2 | | 96432 | 62 cnf52 | rotate | hypervisor | | B45 c2d9 5811-2-47 e4-8 c08 - a5d479d03009 | 2 | node - d4: ae: how e: 90:9 c | 172.16.254.126 | 172.16.254.3 | https://172.16.254.126:8885 | running | true | online 24 | | 128723 | 2 | 8 q1pb3x | rotate | hypervisor | +--------------------------------------+----------+-------------------+----------------+--------------+----------------- ------------+---------+---------+-------------+----------+-----------+------------+------------+--------------+--------- ---+Copy the code

Deploy the Rook component

Download the Rook code:

# clone rook source $git clone - single - branch, branch in release 1.7 https://github.com/rook/rook.git $CD rookCopy the code

To deploy the Rook operator service:

# the apply rook CRD $CD cluster/examples/kubernetes/ceph/pre - k8s - 1.16 / $kubectl apply - f CRDS. Yaml # apply the rook Operator service $CD.. $kubectl apply -f common.yaml -f operator.yaml RESTARTS for rook-ceph-operator pod becomes Running $kubectl -n rook-ceph get Pods NAME READY STATUS RESTARTS AGE rook-ceph-operator-68964f4b87-pc87m 1/1 Running 0 7m38sCopy the code

Create a CEPh cluster

Yaml: Rook cluster.yaml: Rook cluster.yaml: Rook cluster.yaml

$ cp cluster.yaml cluster-env.yaml
Copy the code

Yaml and cluster-env.yaml diff contents:

$ diff -u cluster.yaml cluster-env.yaml
Copy the code

For details, see Diff. Note the following:

  • spec.image: Registry.cn-beijing.aliyuncs.com/yunionio/ceph:v14.2.22 instead, here need to use v14 version of the image, the corresponding ceph version for the nautilus, Later versions may have incompatible CloudPods

  • Spec.net work.provider: changed to host, indicating that ceph related containers use hostNetwork, so that services outside the Kubernetes cluster can be used

  • Placement: Kubernetes ceph pod is assigned to a node where role=storage-node

  • Spec. storage: storage configuration

    • UseAllNodes: We specify role=storage-node nodes to run ceph. This value must be set to false
    • Nodes: Sets the storage path of each node. The storage path can be a disk or directory
Yaml 2021-10-09 10:49:53.731596210 +0800 +++ cluster-env.yaml 2021-10-09 17:50:01.859112585 +0800 @@-21,7 + 21, 7 # @ @ versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/. # If you  want to be more precise, You can always use a timestamp tag to quay. IO/ceph/ceph: v16.2.6-20210918 # This tag took not contain a new ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities - image: Quay. IO/ceph/ceph: v16.2.6 + image: Registry.cn-beijing.aliyuncs.com/yunionio/ceph:v14.2.22 # been to unsupported versions of Ceph. Currently `nautilus`, `octopus`, and `pacific` are supported. # Future versions such as `pacific` would require this to be set to `true`. # Do not set to @@rulesNamespace: rook-ceph network: # enable host networking - #provider: host + provider: host # enable the Multus network provider #provider: multus #selectors: @@ # To control where various services will be scheduled by Kubernetes, use the placement configuration sections below. # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and # tolerate taints with a key of 'storage-node'. -# placement: -# all: -# nodeAffinity: -# requiredDuringSchedulingIgnoredDuringExecution: -# nodeSelectorTerms: -# - matchExpressions: -# - key: role -# operator: In -# values: -# - storage-node -# podAffinity: -# podAntiAffinity: -# topologySpreadConstraints: -# tolerations: -# - key: storage-node -# operator: Exists + placement: + all: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: role + operator: In + values: + - storage-node + podAffinity: + podAntiAffinity: + topologySpreadConstraints: + tolerations: + - key: storage-node + operator: Exists # The above placement information can also be specified for mon, osd, and mgr components # mon: # Monitor deployments may contain an anti-affinity rule for avoiding Monitor @@-207,8 +207,8 @@ # osd: rook-ceph-osd-priority-class # mgr: rook-ceph-mgr-priority-class storage: # cluster level storage configuration and selection - useAllNodes: true - useAllDevices: true + useAllNodes: false + useAllDevices: false #deviceFilter: config: # crushRoot: "Custom -root" # specify a non-default root label for the CRUSH map @@ -219,17 +219,22 @@ # encryptedDevice: "true" # the default value for this option is "false" # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named # nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes. IO /hostname' label. - # Nodes: - # - name: "172.17.4.201" - # devices: # specific devices to use for storage can be specified for each node - # - name: "sdb" - # - name: "nvme01" # multiple osds can be created on high performance devices - # config: - # osdsPerDevice: "5" - # - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths - # config: # configuration can be specified at the node level which overrides the cluster level config - # - name: "172.17.4.301" - # deviceFilter: "^sd." + Nodes: + - name: "node-0" + devices: # specific devices to use for storage can be specified for each node + - name: "sdb" + - name: "sdc" + - name: "sdd" + - name: "node-1" + devices: + - name: "sdb" + - name: "sdc" + - name: "sdd" + - name: "node-2" + devices: + - name: "sdb" + - name: "sdc" + - name: "sdd" # when onlyApplyOSDPlacement is false, will merge both placement.All() and placement.osd onlyApplyOSDPlacement: false # The section for configuring management of daemon disruptions during upgrade or fencing.Copy the code

After compiling cluster-env.yaml, use the following command to create the ceph cluster:

$kubectl apply -f cluster - env. Yaml cephcluster. Ceph. Rook. IO/rook - ceph created # to check the rook - ceph pod health $in the namespace kubectl -n rook-ceph get pods NAME READY STATUS RESTARTS AGE rook-ceph-crashcollector-dl380p-55f6cc56c9-b8ghc 1/1 Running 0 3m3s rook-ceph-crashcollector-r710-7d8659858-mrqgq 1/1 Running 0 2m20s rook-ceph-crashcollector-r720xd-1-5b686487c5-hvzdb 1/1 Running 0 3m10s rook-ceph-csi-detect-version-ffdsf 0/1 Completed 0 26m rook-ceph-mgr-a-759465b6c7-cslkp 1/1 Running 0 3m13s rook-ceph-mon-a-657c4c6769-ljtr9 1/1 Running 0 18m rook-ceph-mon-b-7db98b99d4-99pft 1/1 Running 0 18m rook-ceph-mon-c-7f84fc475d-5v599 1/1 Running 0 10m rook-ceph-operator-68964f4b87-pc87m 1/1 Running 0 68m rook-ceph-osd-0-7cc5cb94cb-dxznm 1/1 Running 0 2m32s rook-ceph-osd-1-f4d47ddf9-7vgh7 1/1 Running 0 2m35s rook-ceph-osd-2-5d7667b8d8-d5tnp 1/1 Running 0 2m20s rook-ceph-osd-3-c9c56cd77-8sbzj 1/1 Running 0 2m32s rook-ceph-osd-4-88565589c-rnpmg 1/1 Running 0 2m35s rook-ceph-osd-5-7d7c554b6c-pvsfx 1/1 Running 0 2m35s rook-ceph-osd-6-6c7596c844-jg9qt 1/1 Running 0 2m20s rook-ceph-osd-7-55f9987ddf-pjthz 1/1 Running 0 2m32s rook-ceph-osd-8-6949b69dd6-685wp 1/1 Running 0 2m20s rook-ceph-osd-prepare-dl380p-c6nc8 0/1 Completed 0 3m3s rook-ceph-osd-prepare-r710-zkmjz 0/1 Completed 0 3m3s $kubectl -n rook-ceph get cephcluster NAME $kubectl -n rook-ceph get cephcluster NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH rook-ceph /var/lib/rook 3 29m Ready Cluster created successfully HEALTH_OKCopy the code

After deploying the CEPH cluster, we need to deploy toolbox.yaml Pod to get the cluster connection information:

$ kubectl apply -f toolbox.yaml deployment.apps/rook-ceph-tools created $ kubectl -n rook-ceph get pods | grep tools $kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph exec -it $ Rook -ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash # 172.16.254.126:6789172.16. 254.124:6789172.16. 254.127:6789 [root @ rook - f55 ceph - tools - 885579 - QPNHH /] $cat The/etc/ceph/ceph. Conf [global] mon_host = 172.16.254.126:6789172.16. 254.124:6789172.16. 254.127:6789 / client. Admin Keyring = /etc/ceph/keyring AQBHTWFhFQzrORAALLIngo/OOTDdnUf4vNPRoA== [root@rook-ceph-tools-885579f55-qpnhh /]$ cat /etc/ceph/keyring [client.admin] Key = AQBHTWFhFQzrORAALLIngo/OOTDdnUf4vNPRoA = = # to check the health cluster [root @ rook - f55 ceph - tools - 885579 - QPNHH /] $ceph status cluster:  id: 233cf123-7a1a-4a7b-b6db-1cee79ec752b health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 38m) mgr: a(active, since 33m) osd: 9 osds: 9 up (since 33m), 9 in (since 34m); 30 remapped pgs data: pools: 1 pools, 256 pgs objects: 0 objects, 0 B usage: 57 MiB used, 20 TiB / 20 TiB avail pgs: 226 Active +clean 30 Active +clean+ REMapped [root@rook-ceph-tools-885579f55-qpnhh /]$ceph OSD Status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 node-0 6172k 3725G 0 0 0 0 exists,up 1 node-1 7836k 279G 0 0 0 0 exists,up 2 node-2 5596k 931G 0 0 0 0 exists,up 3 node-0 6044k 3725G 0 0 0 0 exists,up 4 node-1 5980k 279G 0 0 0 0 exists,up 5 node-1 5980k 279G 0 0 0 0 exists,up 6 node-2 6236k 3726G 0 0 0 0 exists,up 7 node-0 7772k 3725G 0 0 0 0 exists,up 8 node-2 7836k 3726G 0 0 0 0 Exists,up # Create a cloudPods-test pool for later VIRTUAL machine tests [root@rook-ceph-tools-885579f55-qpnhh /]$ceph osd pool create Cloudpods-test 64 64 pool 'cloudPods-test' created # create pool RBD [root@rook-ceph-tools-885579f55-qpnhh /]$RBD pool init cloudpods-test [root@rook-ceph-tools-885579f55-qpnhh /]$ ceph osd lspools 1 cloudpods-testCopy the code

The Cloudpods virtual machine uses Ceph deployed by Rook

After deploying the Ceph cluster in the Kubernetes cluster using Rook, import the connection information of the Ceph cluster to the Cloudpods private cloud platform and make it available to the virtual machine.

The ceph connection information from the previous steps is as follows:

  • Mon_host: 172.16.254.126:6789172.16. 254.124:6789172.16. 254.127:6789
  • keyring: AQBHTWFhFQzrORAALLIngo/OOTDdnUf4vNPRoA==
  • rbd pool: cloudpods-test

Then log in to cloudPods front-end to create cepH RBD storage and associate it with the host as follows:

  1. Create a block storage named rook-ceph and fill in the following information:

  1. By default, the rook-ceph block storage is offline and needs to be associated with the platform’s private cloud host. The host will detect the connectivity of the CEPH cluster and obtain the corresponding pool capacity information:

  1. After the rook-ceph block storage is associated with the host, the state becomes “online” and 20TB of capacity is acquired:

  1. Used to create a VMrook-cephStorage: Add disks on the VM creation page and set the storage type toCeph RBD:

  1. After the VM is created, log in to the VM using VNC or SSH.

You can find that /dev/sda(system disk) and /dev/sdb(data disk) are mounted to the VM. The underlying RBD block devices of CEPH are both mounted to the VM, because the speed of the mechanical disk used by cepH’s underlying block device is 99 MB/s, which meets the expectations.

  1. Then, run the climc command to check the host where the VM is located. It is found that the CEPh-test-VM is running on a Node node-1 that is also a storage node in the CEPh cluster, achieving a hyperconverged architecture:
$ climc server-list --details --search ceph-test-vm +--------------------------------------+--------------+--------+---------------+--------+---------+------------+-------- ---+------------+---------+ | ID | Name | Host | IPs | Disk | Status | vcpu_count | vmem_size | Hypervisor | os_type | +--------------------------------------+--------------+--------+---------------+--------+---------+------------+-------- + -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- + | e0 e2d ffd8ec7c - 1-4427-89-81 b6ce184185 | ceph - test - vm | | node - 1 172.16.254.252 | 235520 | running | 2 | 2048 | kvm | Linux | +--------------------------------------+--------------+--------+---------------+--------+---------+------------+-------- ---+------------+---------+Copy the code

Other operating 

  • Cleaning up a Cluster For details about how to delete the Ceph Cluster deployed by Rook