Two more:
- “KubeSphere error combat”
- KubeSphere Mistake Combat II
I have also documented some usage problems with kubesphere in the future, which I hope will help others to experience a silky container management platform.
Delete abnormal containers
Consul has been deployed using the helm, and is deleted later
[root@master ~]# helm delete consul --purge
Copy the codeA POD status on Consul has been Terminating
[root@master ~]# kubectl get pods -n common-service
NAME READY STATUS RESTARTS AGE
consul-1 1/2 Terminating 1 24d
redis-master-0 1/1 Running 1 17d
redis-slave-0 1/1 Running 1 8d
redis-slave-1 1/1 Running 1 17d
Copy the codeCheck the status
[root@master ~]# kubectl describe pods consul-1 -n common-service
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedSync 3m41s (x4861 over 22h) kubelet, node02 error determining status: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Copy the codeDisposal suggestions:
- Upgrade to Docker 18. This version uses the new Containerd and has a number of bug fixes.
- If the terminating state occurs, container experts can be provided to conduct investigation. It is not recommended to forcibly delete the terminating state directly, which may cause some business problems.
Suspected BUG in version 17 dockerd. Kubectl -n cn-staging delete pod apigateway-6dc48bf8b6-clcwk –force –grace-period=0 can be used to forcibly delete pods, but docker ps can still see the container
[root@master ~]# kubectl -n common-service delete pod consul-1 --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "consul-1" force deleted
[root@master ~]# kubectl get pods -n common-service
NAME READY STATUS RESTARTS AGE
redis-master-0 1/1 Running 1 17d
redis-slave-0 1/1 Running 1 8d
redis-slave-1 1/1 Running 1 17d
Copy the codeView it on Node2
/ root @ node02 ~ # docker ps - a | grep consul b5ea9ace7779 fc6c0a74553d "/ entrypoint. Sh/run..." 3 weeks ago Up 3 weeks k8s_consul_consul-1_common-service_5eb39c90-8503-4125-a2f0-63f177e36293_1 13192855eb6f Mirrorgooglecontainers/pause - amd64:3.1 "/ pause" three weekes line Exited (0) 23 hours a line k8s_POD_consul-1_common-service_5eb39c90-8503-4125-a2f0-63f177e36293_0Copy the codeIf there are Finalizers in the metadata, the resource is typically created by a program that has its Finalizers logo in the metadata. This means that once the resource is removed, it needs to be cleaned up by the app that created it, removing the tokens from the finalizers before the resource can be completely removed. For example, some resources created by Rancher have Finalizers in them.
Manually edit your resource definitions and delete finalizers. If you can check your resources, they’re already deleted.
15 K8S log exceptions are removed
After upgrading from V2.0 to V2.1 after checking kubesphere no log
Check whether the relevant Pods responsible for log collection are normal,Fluent Bit + ElasticSearch
[root@master ~]# kubectl get po -n kubesphere-logging-system
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-curator-elasticsearch-curator-158086m9zv5 0/1 Completed 0 2d13h
elasticsearch-logging-curator-elasticsearch-curator-158095fmdlz 0/1 Completed 0 37h
elasticsearch-logging-curator-elasticsearch-curator-158103bwf8f 0/1 Completed 0 13h
elasticsearch-logging-data-0 1/1 Running 1 8d
elasticsearch-logging-data-1 1/1 Running 774 69d
elasticsearch-logging-discovery-0 1/1 Running 478 56d
elasticsearch-logging-kibana-94594c5f-q7sht 1/1 Running 1 22d
fluent-bit-2b9kj 2/2 Running 2 23h
fluent-bit-bf52m 2/2 Running 2 23h
fluent-bit-pkb9f 2/2 Running 2 22h
fluent-bit-twd98 2/2 Running 2 23h
logging-fluentbit-operator-56c6b84b94-4nzzn 1/1 Running 1 23h
logsidecar-injector-5cbf7bd868-cr2kh 1/1 Running 1 11d
logsidecar-injector-5cbf7bd868-mp46g 1/1 Running 1 22d
Copy the codeBefore we know that logs are stored by ES, kubesphere-logging-system maps es service to NodePort mode, check index, find only Jaeger’s
curl elasticsearch-logging-data.kubesphere-logging-system.svc:9200/_cat/indices
Copy the code
After viewing the index is normal
View the Fluent Bit logs
[root@master ~]# kubectl-n kubespher-logging -system logs -f Fluent-bit-2b9KJ-c fluent-bit I0207 13:53:25.667667 1 fluentbitdaemon.go:135] Start Fluent-Bit daemon... Fluent Bit V1.0.5 Copyright (C) Treasure Data [2020/02/07 13:53:26] [info] [storage] initializing Copyright (C) Treasure Data [2020/02/07 13:53:26] [info] [storage] initializing... [2020/02/07 13:53:26] [ info] [storage] in-memory [2020/02/07 13:53:26] [ info] [storage] normal synchronization mode, checksum disabled [2020/02/07 13:53:26] [ info] [engine] started (pid=15) [2020/02/07 13:53:26] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443 [2020/02/07 13:53:26] [ info] [filter_kube] local POD info OK [2020/02/07 13:53:26] [ info] [filter_kube] testing connectivity with API server... [2020/02/07 13:53:36] [ warn] net_tcp_fd_connect: getaddrinfo(host='kubernetes.default.svc'): Name or service not known [2020/02/07 13:53:36] [error] [filter_kube] upstream connection error [2020/02/07 13:53:36] [ warn] [filter_kube] could not get meta for POD fluent-bit-2b9kjCopy the codeBefore, docker container data was migrated to the data disk due to the disk problem of the system disk. However, the log collection was abnormal due to the link form.
Step 1. Add containersLogMountedPath to ConfigMap Ks-Installer. The path is filled based on the actual environment
[root@master docker]# docker info -f '{{.DockerRootDir}}'
/data/docker
[root@master docker]# ll /var/lib/docker
lrwxrwxrwx. 1 root root 12 Oct 10 19:01 /var/lib/docker -> /data/docker
Copy the code
Step 2. Wait a few minutes for the Installer to automatically update the ConfigMap of the Fluent-bit operator. Until containersLogMountedPath is updated to ConfigMap (try not to modify this ConfigMap directly to prevent future upgrades).
Step 3. Restart the Flunet Bit
#Delete the fluent software - bit Daemonset
[root@master ~]# kubectl scale -n kubesphere-logging-system deployment logging-fluentbit-operator --replicas=0
deployment.extensions/logging-fluentbit-operator scaled
[root@master ~]# kubectl delete -n kubesphere-logging-system daemonsets fluent-bit
daemonset.extensions "fluent-bit" deleted
#Restart Fluent-bit Operator Deployment
[root@master ~]# kubectl scale -n kubesphere-logging-system deployment logging-fluentbit-operator --replicas=1
deployment.extensions/logging-fluentbit-operator scaled
#Check whether Fluent-bit is up[root@master ~]# kubectl get po -n kubesphere-logging-system NAME READY STATUS RESTARTS AGE elasticsearch-logging-curator-elasticsearch-curator-158086m9zv5 0/1 Completed 0 2d13h elasticsearch-logging-curator-elasticsearch-curator-158095fmdlz 0/1 Completed 0 37h elasticsearch-logging-curator-elasticsearch-curator-158103bwf8f 0/1 Completed 0 13h elasticsearch-logging-data-0 1/1 Running 1 8d elasticsearch-logging-data-1 1/1 Running 774 69d elasticsearch-logging-discovery-0 1/1 Running 478 56d elasticsearch-logging-kibana-94594c5f-q7sht 1/1 Running 1 22d fluent-bit-5rzpv 0/2 ContainerCreating 0 3s fluent-bit-nkzdv 0/2 ContainerCreating 0 3s fluent-bit-pwhw7 0/2 ContainerCreating 0 3s fluent-bit-w5t8k 0/2 ContainerCreating 0 3s logging-fluentbit-operator-56c6b84b94-d7vgn 1/1 Running 0 5s logsidecar-injector-5cbf7bd868-cr2kh 1/1 Running 1 11d logsidecar-injector-5cbf7bd868-mp46g 1/1 Running 1 22dCopy the codeAfter the Fluent-bit of all Nodes is started, you can view that the log is recovered
Reference: github.com/kubesphere/…
Reference: github.com/kubesphere/…
Sixteen K8S storage
If pod runs abnormally, storage is abnormal, and cepH status is abnormal
[root@master test]# ceph -s cluster 774df8bf-d591-4824-949c-b53826d1b24a health HEALTH_WARN mon.master low disk space Monmap e1:1 MONS at {master=10.234.2.204:6789/0} Election epoch 14, quorum 0 Master OSdmap E3064:3 OSDS: 3 up, 3 in flags sortbitwise,require_jewel_osds pgmap v9076023: 192 pgs, 2 pools, 26341 MB data, 8231 objects 64888 MB used, 127 GB / 190 GB avail 192 active+clean client io 17245 B/s wr, 0 op/s rd, 4 op/s wrCopy the codeKubelet has GC by default, where docker files are cleaned manually
# view file
[root@master overlay2]# docker system dfTYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 34 12 8.463GB 5.225GB (61%) Containers 46 21 836.6kB 836.5kB (99%) Local Volumes 4 0 59.03MB Build Cache 00 0B# clean up files
[root@master overlay2]# docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] y
Copy the code17 change the kube-proxy mode to iptables and ipvs
- Modeprobe between each node
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
yum install -y ipset ipvsadm
kubectl get configmap kube-proxy -n kube-system -oyaml
Copy the codeIpvsadm has no rules
[root@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
->RemoteAddress:Port Forward Weight ActiveConn InActConn
Copy the code- Change the ipvs mode of kube-proxy’s configmap to ipvs on the master node
Delete previous Kube-proxy pod
[root@master ~]# kubectl get pod -n kube-system|grep kube-proxy|awk '{print "kubectl delete po "$1" -n kube-system"}'|sh pod "kube-proxy-2wnst" deleted pod "kube-proxy-bfrk9" deleted pod "kube-proxy-kvslw" deletedCopy the code
View through ipvsadm has been switched over.
18 Application Installation
The installation of applications using terminals was recorded in debugging practice 2. In kubesphere2.1, you can install applications in the added application repository in the enterprise space using a web interface. The steps are recorded again
- Add the REPO to the application repository in the enterprise space
- The application installation selection in a specific project comes from the application template
- Select the REPo source and search for the desired Charts package
19 Service Governance
KubeSphere very much enforces its application service governance capabilities, using Istio’s Sidercar mode to inject envoy columns of canary publishing, load balancing, traffic detection management, traffic limiting fuse degradation, etc. Currently, I have tested the governance of microservices in my own application, and I feel it is very useful. I will have the opportunity to record the process later.
Self-organized K8S study notes, there is a rise can be a quick learning exchange: github.com/redhatxl/aw… Support KubeSphere, a domestic container management platform, and make a modest contribution to the community.