Problem description

Kubectl create -f kubectl get Pods kubectl get Pods kubectl create -f kubectl get Pods Kubectl get Pods

# kubectl create -f mysql-rc.yaml
replicationcontroller "mysql" created
# kubectl get pods
NAME          READY     STATUS              RESTARTS   AGE
mysql-nznsb   0/1       ContainerCreating   0          12m
Copy the code

My mysql – rc. Yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: Mysql/mysql - server: 8.0.18-1.1.13
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
Copy the code

Troubleshooting and resolution of problems

Use the kubectl describe command to view recent pod events

# kube describe pod mysql Name: mysql-dKH46 Namespace: default Node: 127.0.0.1/127.0.0.1 Start Time: Sun, 04 Jul 2021 19:48:13 +0800 Labels: app=mysql Status: Pending IP: Controllers: ReplicationController/mysql Containers: mysql: Container ID: Image: mysql/mysql - server: 8.0.18-1.1.13 Image ID: Port: 3306/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Volume Mounts: <none> Environment Variables: MYSQL_ROOT_PASSWORD: 123456 Conditions: Type Status Initialized True Ready False PodScheduled True No volumes. QoS Class: BestEffort Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath TypeReason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {default-scheduler} Normal Scheduled Successfully assigned mysql-dKH46 to 127.0.0.1 1m 28s 4 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request. details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: No such file or directory)" 1m 2s 5 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"registry.access.redhat.com/rhel7/pod-infrastructure:latest\""Copy the code

The following error was reported while pulling the mirror

Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request. details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory)"
Copy the code

Reason is to find/etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt this certificate, and then use ll command to find the address as a soft link, and link to the file does not exist

# ll/etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt LRWXRWXRWX. 1 root root in 27 July 4 very /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem # ll /etc/rhsm/ca | grep redhat | wc -l 0Copy the code

I searched online and learned that RHSM series is the subscription service package of redhat. Centos is recompiled from redhat, so RHSM is also needed. The missing certificate location reported in the error message is actually a symbolic link. The missing certificate location is in /etc/rhsm/ca/Redhat-uep.pem. Previously, python-rhsm-certificates were available through the python-rhsm-certificates package, but centos 7 indicates that this package has been replaced by the subscription manager-rhsm-certificates package. The trick point is that there is a bug in the newly changed package, the message is correct after packaging, actually there is no certificate, you can check the issue here if you are interested

The issue provides a way to do this without downloading packages or extracting certificates from older python-rhsm-certificates packages, simply by executing the following command

openssl s_client -showcerts -servername registry.access.redhat.com -connect registry.access.redhat.com:443 </dev/null 2>/dev/null | openssl x509 -text > /etc/rhsm/ca/redhat-uep.pem
Copy the code

After extracting the certificate, pod can be started correctly

# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
mysql-nznsb   1/1       Running   0          29m
Copy the code

Reference links:

  • Kubernetes error in CentOS 7
  • Centos Issue 0014785
  • Kubernetes Debug Pods and ReplicationControllers