I also recorded some problems in using KubeSphere recently. I hope it can be helpful to others

Kubesphere application upload problem

8.1 File Uploading 413

Ingress413 error: Kubesphere uses ingress-nginx controller. You can add k-V in the annotation to support it.

Solution: Apply route customization Max body size

Kubernetes. Making. IO/ingress – ngi…

8.2 Large File Upload Back-end 504

Large file upload back end response 504 solution:

proxy read timeoutnginx.ingress.kubernetes.io/proxy-read-timeout
Copy the code

Cross domain problems

Kubesphere supports cross-domain use of ingress-nginx, which can be added in annotations by referring to the link below

Kubernetes. Making. IO/ingress – ngi…

The test environment can be used to use hosts, resolve domain names to local, front-end use nginx to do static file service, reverse proxy back-end API, you can refer to the example:

server {
  listen 80;
  server_name localhost;
  #Forcing HTTPS forward
  # rewrite ^(.*)$ https://$hostThe $1 permanent;location / { index index.html; root /smart-frontend; try_files $uri $uri/ /index.html; client_body_buffer_size 200m; charset utf-8; } location /api { proxy_pass http://smart-backend:8080/api; proxy_read_timeout 1200; client_max_body_size 1024m; } gzip on; # enable gzip gzip_vary on; gzip_min_length 1k; Gzip_buffers 4 16k; gzip_comp_level 6; Gzip_types text/plain Application /javascript application/ X-javascript text/ CSS application/ XML text/javascript application/x-httpd-php image/jpeg image/gif image/png image/x-icon; }Copy the code

10 Adding a Node

In the later period, as services gradually improve and cluster node resources are insufficient, new node nodes are added and data disks of node nodes are added to ceph nodes

10.1 Adding a Node to a CEPh Cluster

  • The system configuration

  • Free Key Configuration

  • Hosts configuration

  • Docker is installed and migrated to the data disk

  • Cgroup enabled

  • Ceph data node added

Ceph cluster configuration Add data disk node for node03 cluster (if the data store class is sufficient, you can not add data node)

[root@node03 docker]# mkfs.xfs /dev/vdd [root@node03 docker]# mkdir -p /var/local/osd3 [root@node03 docker]# mount Add VDD to /etc/fstab [root@node03 docker]# yum -y install yum-plugin-priorities epel-release [root@node03 yum.repos.d]# chmod 777 -R /var/local/osd3/ [root@node03 yum.repos.d]# chmod 777 -R /var/local/osd3/* [root@master ceph]# ceph-deploy install node03 [root@master ceph]# ceph-deploy GatherKeys on the master node  master [root@master ceph]# ceph-deploy osd prepare node03:/var/local/osd3Copy the code
  • Activate the osd
[root@master ceph]# ceph-deploy osd activate node03:/var/local/osd3
Copy the code
  • Check the status
[root@master ceph]# ceph-deploy osd list master node01 node02 node03
Copy the code
  • Copy the key
[root@master ceph]# ceph-deploy admin master node01 node02 node03
Copy the code
  • Set permissions on node03
[root@node03 yum.repos.d]# chmod +r /etc/ceph/ceph.client.admin.keyring
Copy the code
  • Configure the MDS on the master
[root@master ceph]# ceph-deploy mds create node01 node02 node03
Copy the code
  • Check the status
[root@master ceph]# ceph health [root@master ceph]# ceph - The data needs to be backfilled due to the new node. [root@master conf]# ceph -s cluster 5b9eb8D2-1C12-4f6D-AE9C-85078795794b Health HEALTH_ERR 44 PGS backfill_wait 1 PGS Backfilling 1 PGS Inconsistent 45 PGS stuck unclean Recovery 1/55692 objects Degraded (0.002%) Recovery 9756/55692 Objects Misplaced (17.518%) 2 Scrub errors monmap E1: 1 mons at {master= 172.16.60.2:6689/0} election epoch 35, quorum 0 Master OSdmap E2234:4 OSDS: 4 up, 4 in; 45 remapped pgs flags sortbitwise,require_jewel_osds pgmap v5721471: 192 pgs, 2 pools, 104 GB data, 27846 objects 230 GB used, 1768 GB / 1999 GB AVAIL 1/55692 Objects Misplaced (0.002%) 9756/55692 objects Misplaced (17.518%) 146 active+clean 44 active+remapped+wait_backfill 1 active+remapped+backfilling 1 active+clean+inconsistent recovery io 50492 kB/s, 13 objects/s client io 20315 B/s wr, 0 op/s rd, 5 op/s wrCopy the code
  • The final problem, due to the new Node node, the new ceph data node needs data synchronization
[root@master conf]# ceph -s cluster 5b9eb8d2-1c12-4f6d-ae9c-85078795794b health HEALTH_ERR 1 pgs inconsistent 2 scrub Errors monmap e1:1 mons at {master= 172.16.60.2:6689/0} election epoch 35, quorum 0 Master OSdmap E2324:4 OSDS: 4 up, 4 in flags sortbitwise,require_jewel_osds pgmap v5723479: 192 pgs, 2 pools, 104 GB data, 27848 objects 229 GB used, 1769 GB / 1999 GB avail 191 active+clean 1 active+clean+inconsistent client io 78305 B/s wr, 0 op/s rd, 18 op/s wR fix [root@master conf]# ceph-s cluster 5b9eb8D2-1C12-4f6D-AE9C-85078795794b health HEALTH_OK monmap e1: 1 mons at {master= 172.16.60.2:6689/0} election epoch 35, quorum 0 Master OSdmap E2324:4 OSDS: 4 up, 4 in flags sortbitwise,require_jewel_osds pgmap v5724320: 192 pgs, 2 pools, 104 GB data, 27848 objects 229 GB used, 1769 GB / 1999 GB avail 192 active+clean client io 227 kB/s wr, 0 op/s rd, 7 op/s wr#Synchronization is complete
[root@master conf]# ceph health
HEALTH_OK
Copy the code

10.2 Node Adding a Node

Kubesphere for the convenience of new nodes, provides a new convenient in the footsteps of a key, may refer to: kubesphere.com.cn/docs/v2.1/z…

Modify the host. Ini

Master anSIBLE_connection =local IP = 172.16.60.2node01 ANSIBLE_host =172.16.60.3 IP = 172.16.60.3node02 Ansible_host =172.16.60.4 IP =172.16.60.4 node03 ANSIBLE_host =172.16.60.5 IP =172.16.60.5 [kube-master] master [kube-node] master node01 node02 node03Copy the code

Run the add-nodes.sh script in the “/script” directory. After the expansion script is successfully executed, you can see the cluster node information containing the new node. You can choose infrastructure from the menu of the KubeSphere console and enter the host management page to view the information, or run the Kubectl get node command through the Kubectl tool. View details about nodes in the expanded cluster.

[root@master scripts]# ./add-nodes.sh
Copy the code

Check the validation

[root@master conf]# kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE Kernel-version container-Runtime master Ready Master 136d V1.15.5 172.16.60.2 < None > CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node01 Ready node,worker 136d v1.15.5 172.16.60.3 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node02 Ready node,worker 136d v1.15.5 172.16.60.4 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node03 Ready worker 10m V1.15.5 172.16.60.5 < None > CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://19.3.5 [root@master conf]# kubectl label node node-role.kubernetes.io/node= common.yaml hosts.ini plugin-qingcloud.yaml [root@master conf]# kubectl label node node03 node-role.kubernetes.io/node= node/node03 labeled [root@master conf]# kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE Kernel-version container-Runtime master Ready Master 136d V1.15.5 172.16.60.2 < None > CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node01 Ready node,worker 136d v1.15.5 172.16.60.3 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node02 Ready node,worker 136d v1.15.5 172.16.60.4 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker:// 18.6.2node03 Ready node,worker 11m v1.15.5 172.16.60.5 < None > CentOS Linux 7 (Core) 3.10.0-693. El7. X86_64 docker: / / 19.3.5 root @ master the conf #Copy the code

Eleven K8s cluster resources are uneven

The usage of K8S resources is unbalanced. Some system services are running on nodes due to the deployment of nodeSelect. Node2 occupies a large memory, causing an exception alarm or cluster restart

You can do this by looking at

kubectl get pods -o wide --all-namespaces |grep node02 |awk '{print $1,  $2}'
Copy the code

Schedule some system applications to master via NodesELECT to reduce memory stress on Node2.

`kubectl  get nodes --show-labels`
Copy the code

View the system build on Node2 and add Nodeselector to reschedule

      nodeSelector:
        node-role.kubernetes.io/master: master
Copy the code

View the existing Kubesphere System Deployment on Node2

With scheduling complete, check that the memory load for Node2 has come down

Kubesphere Devops project

Node03 is added, and the devops project is in the queue for a week. At this time, the instance running the job is not initialized. Log in to the cluster, and the Base Pod on Node03 is mirrored on the Pull Agent. Save the base image and load it on node03

[root@master ~]# kubectl describe pods -n kubesphere-devops-system $(kubectl get pods -n kubesphere-devops-system |grep -E "^base" |awk '{print $1}')
Copy the code

13 Kubesphere application installation

At present, my kubesphere cluster is 2.1. After adding the REPO in the project, I still need to manually trigger the synchronization of mirror data in the background. I added several HELM’s REPO, and it seems that the chart inside is not displayed in the Web interface. I created a new application and then selected the app store from Kubesphere. There were only a few charts in the store, but I could not find the chartscharts of the added helm source. Inside the server, I could use the command search. For the time being, the consulting community has received a reply. I remember that there is a task in the background of V2.0 version to synchronize charts. For the current version of 2.1, the helm command is first used to manually install helm in the cluster

[root@master common-service]# helm install -n consul --namespace common-service -f consul/values-production.yaml consul/  NAME: consul LAST DEPLOYED: Tue Jan 14 17:56:27 2020 NAMESPACE: common-service STATUS: DEPLOYED RESOURCES: ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE consul-0 0/2 Pending 0 0s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE consul ClusterIP None <none> 8400/TCP,8301/TCP,8301/UDP,8300/TCP,8600/TCP,8600/UDP 1s consul-ui ClusterIP 10.233.59.7 < none > 80 / TCP 1 s = = > v1 / StatefulSet NAME READY AGE consul 0/3 0 s = = > v1beta1 / PodDisruptionBudget NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE consul-pdb 1 N/A 0 1s NOTES: ** Please be patient while the chart is being deployed ** Consul can be accessed within the cluster on port 8300 at consul.common-service.svc.cluster.local In order to access to the Consul Web UI: kubectl port-forward --namespace common-service svc/consul-ui 80:80 echo "Consul URL: http://127.0.0.1:80" Please take into account that you need to wait until a cluster leader is elected before using the Consul Web UI. In order to check the status of the cluster you can run the following command: kubectl exec -it consul-0 -- consul members Furthermore, to know which Consul node is the cluster leader run this other command: kubectl exec -it consul-0 -- consul operator rafCopy the code

Specific questions can be reference post: kubesphere.com.cn/forum/d/669…

Self-organized K8S study notes, there is a rise can be a quick learning exchange: github.com/redhatxl/aw… Support KubeSphere, a domestic container management platform, and make a modest contribution to the community.