The previous article prepared the offline resources required for an offline installation of OCP, including the installation of the Image, all the sample Image Streams, and all the RedHat Operators in OperatorHub. This article starts with the installation of the OCP (Openshift Container Platform) cluster, including DNS resolution, load balancing configuration, Ignition profile generation, and cluster deployment.

Multiple files are required during the OCP installation: the installation profile, the Kubernetes deployment checklist, and the Ignition profile (including Machine Types). The installation profile will be transformed into a Kubernetes deployment checklist, which will then be wrapped into the Ignition profile. The installer uses these Ignition profiles to create the Openshift cluster. All the original installation configuration files are modified when you run setup, so you should back them up before installing.

1. Installation process

To install OCP, we need a boot host (Bootstrap). This host has access to all OCP nodes. Boot the host to start a temporary control plane, which starts the rest of the OCP cluster and is then destroyed. Boot hosts for cluster installation using the Ignition configuration file, which describes how to create an OCP cluster. The Ignition profile generated by the installer contains certificates that expire after 24 hours, so cluster installation must be completed before the certificates expire.

To boot the cluster installation, perform the following steps:

  • Boot the host up and start hostingMasterResources required to start a node.
  • MasterThe node remotely obtains resources from the boot host and completes the boot.
  • MasterNodes are built by bootstrap hostsEtcdThe cluster.
  • Boot the host to use the new oneEtcdCluster Startup TemporaryKubernetesControl plane.
  • The temporary control plane generates the control plane on the Master node.
  • The temporary control plane closes and transfers control to the production control plane.
  • Instruct the host to inject OCP components into the generation control plane.
  • Setup closes the boot host.

After the boot installation process is complete, the OCP cluster is deployed. The cluster then downloads and configures the remaining components required for daily operations, including creating compute nodes and installing other services through Operator.

2. Prepare server resources

The server planning is as follows:

  • Install three controller plane nodesEtcd, control plane components, andInfrasBasic components.
  • Two compute nodes, running real load.
  • A boot host can be deleted after the cluster deployment is complete.
  • A base node for preparing the offline resources mentioned in the previous section, as well as deploying DNS and load balancing.
  • A mirror node to deploy a private mirror repositoryQuay.
The host type The operating system Hostname vCPU memory storage IP FQDN
Mirror node RHEL 7.6 registry 4 8GB 150GB 192.168.57.70 registry.openshift4.example.com
Based on the node RHEL 7.6 bastion 4 16GB 120GB 192.168.57.60 bastion.openshift4.example.com
Guide the host RHCOS bootstrap 4 16GB 120GB 192.168.57.61 bootstrap.openshift4.example.com
Control plane RHCOS master1 4 16GB 120GB 192.168.57.62 master1.openshift4.example.com
Control plane RHCOS master2 4 16GB 120GB 192.168.57.63 master2.openshift4.example.com
Control plane RHCOS master3 4 16GB 120GB 192.168.57.64 master3.openshift4.example.com
Compute nodes RHCOS or RHEL 7.6 worker1 2 8GB 120GB 192.168.57.65 worker1.openshift4.example.com
Compute nodes RHCOS or RHEL 7.6 worker2 2 8GB 120GB 192.168.57.66 worke2.openshift4.example.com

3. Configure the firewall

Let’s look at the port number assignment for each node.

Ports to be opened between all nodes (compute nodes and the control plane) :

agreement port role
ICMP N/A Test the network connectivity
TCP 9000-9999. Service ports of a node, including those used by Node9100-9101.Port and Cluster Version Operator9099port
10250–10259 Default port reserved by Kubernetes
10256 openshift-sdn
UDP 4789 Communication port of the VXLAN or GENEVE protocol
6081 Communication port of the VXLAN or GENEVE protocol
9000–9999 Service ports of a node, including those used by Node9100-9101.port
30000–32767 Kubernetes NodePort

Ports on the control plane that need to be opened to other nodes:

agreement port role
TCP 2379–2380 Etcd service port
6443 Kubernetes API

In addition, two layer 4 load balancers are configured, one to expose the cluster API and one to expose the Ingress:

port role internal external describe
6443 Boot hosts and use the control plane. After initializing the cluster control plane on the boot host, manually delete the boot host from the load balancer x x Kubernetes API server
22623 Boot hosts and use the control plane. After initializing the cluster control plane on the boot host, manually delete the boot host from the load balancer x Machine Config server
443 Ingress Controller or Router x x HTTPS traffic
80 Ingress Controller or Router x x HTTP traffic

4. To configure DNS

According to the official documentation, OCP clusters using the UPI infrastructure require the following DNS records. In each record,

is the cluster name and

is the cluster base domain specified in the install-config.yaml file, as shown in the following table:

component DNS record describe
Kubernetes API api.<cluster_name>.<base_domain>. This DNS record must point to the load balancer of the control plane node. This record must be resolvable by clients outside the cluster and by all nodes in the cluster.
api-int.<cluster_name>.<base_domain>. This DNS record must point to the load balancer of the control plane node. This record must be resolvable by clients outside the cluster and by all nodes in the cluster.
Routes *.apps.<cluster_name>.<base_domain>. DNS wildcard record pointing to the load balancer. The backend of this load balancer is the node where the Ingress Router resides, which is the compute node by default. This record must be resolvable by clients outside the cluster and by all nodes in the cluster.
etcd etcd-<index>.<cluster_name>.<base_domain>. OCP requires that the DNS records of each ETCD instance point to the control plane node where the instance is running. Etcd instances are distinguished by values, which are defined as0At the beginning,n-1End, wherenIs the number of control plane nodes in the cluster. All nodes in the cluster must be able to parse this record.
_etcd-server-ssl._tcp.<cluster_name>.<base_domain>. Because ETCD uses ports2380For external services, an SRV DNS record of each ETCD node with priority 0, weight 10 and port 2380 needs to be established

There are many ways to deploy the DNS service, and I certainly recommend using CoreDNS as the cloud native standard. Since you need to add SRV records here, you need CoreDNS in conjunction with the ETCD plug-in. All of the following operations are performed on the underlying node.

Install and start etcd with yum:

$ yum install -y etcd
$ systemctl enable etcd --nowCopy the code

Then download the CoreDNS binary:

$$tar ZXVF wget HTTP: / / https://github.com/coredns/coredns/releases/download/v1.6.9/coredns_1.6.9_linux_amd64.tgz Coredns_1.6.9_linux_amd64. TGZ $mv coreDNS /usr/local/binCopy the code

Create Systemd Unit file:

$ cat > /etc/systemd/system/coredns.service <<EOF [Unit] Description=CoreDNS DNS server Documentation=https://coredns.io  After=network.target [Service] PermissionsStartOnly=true LimitNOFILE=1048576 LimitNPROC=512 CapabilityBoundingSet=CAP_NET_BIND_SERVICE AmbientCapabilities=CAP_NET_BIND_SERVICE NoNewPrivileges=true User=coredns WorkingDirectory=~ ExecStart=/usr/local/bin/coredns -conf=/etc/coredns/Corefile ExecReload=/bin/kill -SIGUSR1 $MAINPID Restart=on-failure [Install] WantedBy=multi-user.target EOFCopy the code

Create a CoreDNS user:

$ useradd coredns -s /sbin/nologinCopy the code

Create a CoreDNS configuration file:

$cat > / etc/coredns Corefile < < EOF. : {# 53 monitored TCP and UDP port 53 template IN A apps.openshift4.example.com {match .*apps\.openshift4\.example\.com # regular expression matching request DNS Name answer "{{.name}} 60 IN A 192.168.57.60" # DNS reply fallthrough} Etcd test.com {path /skydns # etcd /skydns # etcd /skydns All subsequent DNS records will be stored in this path http://localhost:2379 # etcd access address, multiple Spaces separated fallthrough # This parameter is optional, and the etCD certificate is set for Prometheus. Prometheus Enable the POLLING policy for DNS records forward. 192.168.57.1 log # Print logs} EOFCopy the code

The Template plug-in is used to achieve universal domain name resolution.

Start CoreDNS and set boot autoboot:

$ systemctl enable coredns --nowCopy the code

Verify generic domain name resolution:

@ $dig + short apps.openshift4.example.com 127.0.0.1 192.168.57.60 $dig + short x.apps.openshift4.example.com @ 127.0.0.1 192.168.57.60Copy the code

Add other DNS records:

$ alias etcdctlv3='ETCDCTL_API=3 etcdctl' $ etcdctlv3 put /skydns/com/example/openshift4/api '{" host ", "192.168.57.60", "TTL" : 60}' $etcdctlv3 put/skydns/com/example/openshift4 / API - int '{" host ", "192.168.57.60", "TTL" : 60}' $etcdctlv3 put/skydns/com/example/openshift4 / etcd - 0 '{" host ", "192.168.57.62", "TTL" : 60}' $etcdctlv3 put/skydns/com/example/openshift4 / etcd - 1 '{" host ", "192.168.57.63", "TTL" : 60}' $etcdctlv3 put/skydns/com/example/openshift4 / etcd - 2 '{" host ", "192.168.57.64", "TTL" : 60}' $etcdctlv3 put/skydns/com/example/openshift4 / _tcp / _etcd - server - SSL/x1 '{"host":"etcd-0.openshift4.example.com","ttl":60,"priority":0,"weight":10,"port":2380}' $ etcdctlv3 put /skydns/com/example/openshift4/_tcp/_etcd-server-ssl/x2 '{"host":"etcd-1.openshift4.example.com","ttl":60,"priority":0,"weight":10,"port":2380}' $ etcdctlv3 put /skydns/com/example/openshift4/_tcp/_etcd-server-ssl/x3 '{"host":" etcd-2.openShift4.example.com "," TTL ":60,"priority":0,"weight":10,"port":2380 Put/skydns/com/example/openshift4 / bootstrap '{" host ":" 192.168.57.61 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 master1 '{" host ":" 192.168.57.62 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 master2 '{" host ":" 192.168.57.63 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 master3 '{" host ":" 192.168.57.64 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 worker1 '{" host ":" 192.168.57.65 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 worker2 '{" host ":" 192.168.57.66 ", "TTL" : 60}' $etcdctlv3 put / skydns/com/example/openshift4 / registry '{" host ":" 192.168.57.70 ", "TTL" : 60}'Copy the code

Verify DNS resolution:

$yum install - y bind - utils $dig + short api.openshift4.example.com @ 127.0.0.1 192.168.57.60 $dig + short Api-int.openshift4.example.com @ 127.0.0.1 192.168.57.60 $dig + short etcd-0.openshift4.example.com @ 127.0.0.1 Etcd-1 $dig +short Etcd-2.openshift4.example.com @ 127.0.0.1 192.168.57.64 $dig + short - t the SRV _etcd - server-ssl._tcp.openshift4.example.com @ 127.0.0.1 10 33 2380 etcd-0.openshift4.example.com. 10. 33 2380 etcd-1.openshift4.example.com 10 33, 2380 Etcd-2.openshift4.example.com + short bootstrap.openshift4.example.com @ $dig 127.0.0.1 192.168.57.61 $dig + short Master1.openshift4.example.com @ 127.0.0.1 192.168.57.62 $dig + short master2.openshift4.example.com @ 127.0.0.1 192.168.57.63 $dig + short master3.openshift4.example.com @ 127.0.0.1 192.168.57.64 $dig + short Worker1.openshift4.example.com @ 127.0.0.1 192.168.57.65 $dig + short worker2.openshift4.example.com @ 127.0.0.1 192.168.57.66Copy the code

5. Configure load balancing

Load balancing I chose to use Envoy to prepare the configuration file first:

Bootstrap

# /etc/envoy/envoy.yaml
node:
  id: node0
  cluster: cluster0
dynamic_resources:
  lds_config:
    path: /etc/envoy/lds.yaml
  cds_config:
    path: /etc/envoy/cds.yaml
admin:
  access_log_path: "/dev/stdout"
  address:
    socket_address:
      address: "0.0.0.0"
      port_value: 15001Copy the code

LDS

# /etc/envoy/lds.yaml
version_info: "0"
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_openshift-api-server
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 6443
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        stat_prefix: openshift-api-server
        cluster: openshift-api-server
        access_log:
          name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_machine-config-server
  address:
    socket_address:
      address: "::"
      ipv4_compat: true
      port_value: 22623
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        stat_prefix: machine-config-server
        cluster: machine-config-server
        access_log:
          name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_ingress-http
  address:
    socket_address:
      address: "::"
      ipv4_compat: true
      port_value: 80
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        stat_prefix: ingress-http
        cluster: ingress-http
        access_log:
          name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: listener_ingress-https
  address:
    socket_address:
      address: "::"
      ipv4_compat: true
      port_value: 443
  filter_chains:
  - filters:
    - name: envoy.tcp_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
        stat_prefix: ingress-https
        cluster: ingress-https
        access_log:
          name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdoutCopy the code

CDS

# /etc/envoy/cds.yaml version_info: "0" resources: - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: openshift-api-server connect_timeout: 1s type: strict_dns dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN load_assignment: cluster_name: openshift-api-server endpoints: - lb_endpoints: - endpoint: address: Socket_address: address: 192.168.57.61 port_value: 6443 - endpoint: address: socket_address: address: 192.168.57.62 port_value: 6443 - endpoint: address: socket_address: address: 192.168.57.63 port_value: 6443 - endpoint: Address: socket_address: address: 192.168.57.64 port_value: 6443 - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: machine-config-server connect_timeout: 1s type: strict_dns dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN load_assignment: cluster_name: machine-config-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 192.168.57.61 port_value: 22623 - endpoint: address: socket_address: address: 192.168.57.62 port_value: 22623 - endpoint: address: socket_address: address: 192.168.57.63 port_value: 22623 - endpoint: address: Socket_address: address: 192.168.57.64 port_value: 22623 - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: ingress-http connect_timeout: 1s type: strict_dns dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN load_assignment: cluster_name: ingress-http endpoints: -lb_endpoints: - endpoint: address: socket_address: address: 192.168.57.65 port_value: 80 - endpoint: address: Socket_address: address: 192.168.57.66 port_value: 80 - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: ingress-https connect_timeout: 1s type: strict_dns dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN load_assignment: cluster_name: ingress-https endpoints: -lb_endpoints: - endpoint: address: socket_address: address: 192.168.57.65 port_value: 443 - endpoint: address: Socket_address: address: 192.168.57.66 port_value: 443Copy the code

Configure unread to look at my ebook: Envoy’s Chinese Guide

Start the Envoy:

$ podman run -d --restart=always --name envoy --net host -v /etc/envoy:/etc/envoy envoyproxy/envoyCopy the code

6. Prepare for installation

Generate an SSH private key and add it to the agent

During the installation, OCP installation debugging and disaster recovery will be performed on the base node, so the SSH key must be configured on the base node, which will be used by the Ssh-Agent to execute the installation program.

The core user on the underlying node can use this private key to log in to the Master node. During cluster deployment, the private key is added to the ~/.ssh/authorized_keys list of the core user.

The procedure for creating a key is as follows:

Create an SSH key without password authentication:

$ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/new_rsaCopy the code

② Start the ssh-Agent process as a background task.

$ eval "$(ssh-agent -s)"Copy the code

③ Add SSH private key to ssh-agent:

$ ssh-add ~/.ssh/new_rsaCopy the code

During the cluster installation, you will be prompted to enter the SSH public key. In this case, use the public key new_rsa.pub created earlier.

Obtaining the installation program

If you install online, you also need to download the installation program on the base node. But this is an offline installation, and the installer was extracted in the previous article, so you don’t need to download it again.

Create an installation configuration file

Create an installation directory to store the files required for the installation:

$ mkdir /ocpinstallCopy the code

Customize install-config.yaml and save it in the /ocpinstall directory. The configuration file must be named install-config.yaml. Configuration file contents:

apiVersion: v1 baseDomain: example.com compute: - hyperthreading: Enabled name: worker replicas: 0 controlPlane: hyperthreading: Enabled name: master replicas: 3 metadata: name: openshift4 networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 networkType: OpenShiftSDN serviceNetwork: -172.30.0.0/16 Platform: None: {} fips: false pullSecret: '{"auths": ... }' sshKey: 'ssh-rsa ... 'additionalTrustBundle: | -- -- -- -- -- BEGIN CERTIFICATE -- -- -- -- -- ellipsis, attention to front two empty here -- -- -- -- -- END CERTIFICATE -- -- -- -- -- imageContentSources: - mirrors: - registry.openshift4.example.com/ocp4/openshift4 source: quay.io/openshift-release-dev/ocp-release - mirrors: - registry.openshift4.example.com/ocp4/openshift4 source: Quay. IO/openshift - release - dev/ocp - v4.0 - art - devCopy the code
  • BaseDomain: All DNS records within Openshift must be subdomains of this base and contain cluster names.
  • compute: Compute node configuration. This is an array and each element must be hyphenated-At the beginning.
  • Hyperthreading: Enabled Enables synchronous multi-threading or hyperthreading. Synchronous multithreading is enabled by default to improve the performance of the machine kernel. To disable this function, disable both the control plane and compute nodes.
  • Compute. Replicas: computes the number of nodes. Because we are manually creating the compute node, we will set this to 0.
  • Controlplane. replicas: controls the number of plane nodes. The number of control plane nodes must be the same as that of ETCD nodes. To achieve high availability, set this parameter to 3.
  • metadata.name: Cluster name. In the previous DNS record<cluster_name>.
  • Cidr: Defines the IP address segment for assigning Pod IP addresses, which cannot overlap with the physical network.
  • hostPrefix: Specifies the subnet prefix length assigned to each node. For example, if thehostPrefixSet to23, a given CIDR is assigned to each node/ 23Subnets, allowing $510 (2^{32-23} -2)$Pod IP addresses.
  • ServiceNetwork: indicates the address pool of the Service IP address. Only one address pool can be set.
  • pullSecret: Pull Secret, used in the previous article, can be done by commandcat /root/pull-secret.json|jq -cTo compress it into one line.
  • sshKey: The public key created above can be used by running a commandcat ~/.ssh/new_rsa.pubLook at it.
  • additionalTrustBundle: trust certificate of private mirror repository Quay, which can be run on the mirror nodecat /data/quay/config/ssl.certLook at it.
  • imageContentSources: From the frontoc adm release mirrorThe output of.

Back up the installation configuration file for future reuse:

$ cd /ocpinstall
$ cp install-config.yaml  install-config.yaml.20200604Copy the code

Create the Kubernetes deployment manifest

Install-config. yaml will be deleted after Kubernetes deployment manifest is created, please backup this file first!

Create Kubernetes deployment manifest file:

$ openshift-install create manifests --dir=/ocpinstallCopy the code

Compile /cluster-scheduler-02-config.yml file and set mastersSchedulable to FLase to prevent Pod scheduling to control nodes.

Create the Ignition profile

Install-config. yaml will be removed after you create the Ignition profile, be sure to back it up first!

$ cp install-config.yaml.20200604 install-config.yaml
$ openshift-install create ignition-configs --dir=/ocpinstallCopy the code

Generated files:

├ ─ ─ auth │ ├ ─ ─ kubeadmin - password │ └ ─ ─ kubeconfig ├ ─ ─ the bootstrap. Ign ├ ─ ─ master. Ign ├ ─ ─ the metadata. The json └ ─ ─ worker. IgnCopy the code

Prepare an HTTP service, here using Nginx:

$ yum install -y nginxCopy the code

Modify the Nginx configuration file /etc/nginx/nginx/.conf to change port 8080 (because the load balancer already uses port 80). Then start the Nginx service:

$ systemctl enable nginx --nowCopy the code

Copy the Ignition configuration file to the HTTP service’s Ignition directory:

$ mkdir /usr/share/nginx/html/ignition
$ cp -r *.ign /usr/share/nginx/html/ignition/Copy the code

Obtain the BIOS file of RHCOS

Download the BIOS file for bare-metal installation and upload it to the Nginx directory:

$ mkdir /usr/share/nginx/html/install $ wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.3-x86_64-metal.x86_64.raw.gz - O The/usr/share/nginx/HTML/install/rhcos 4.4.3 - x86_64 - metal. X86_64. Raw. GzCopy the code

Obtain the ISO file of RHCOS

Local download RHCOS ISO file: mirror.openshift.com/pub/openshi… And upload it to vSphere. The steps are as follows:

① Log in to vSphere and click Save.

② Select a Data Store and select Upload File in the window on the right.

3 Select the downloaded ISO file and upload it to the ESXI host.

7. Install the cluster

Bootstrap

Create the BOOTstrap VM, select Red Hat Enterprise Linux 7 (64-bit) as the OPERATING system, mount the ISO file, configure cpus, memory, and hard disks based on the preceding table, and power on the vm. Then follow these steps:

① On the RHCOS Installer installation screen, press Tab to enter boot parameters.

Coreos. inst = yes: coreos.inst = yes

Ip=192.168.57.61::192.168.57.1:255.255.255.0:bootstrap.openshift4.example.com: ens192: none nameserver = 192.168.57.60 coreos.inst.install_dev=sda Coreos. Inst. Image_url = http://192.168.57.60:8080/install/rhcos-4.4.3-x86_64-metal.x86_64.raw.gz Coreos. Inst. Ignition_url = http://192.168.57.60:8080/ignition/bootstrap.ignCopy the code

The IP =… IP =$IPADDRESS::$DEFAULTGW:$NETMASK:$HOSTNAMEFQDN:$IFACE: None

As shown in the figure:

③ If the installation fails, the system enters Emergency Shell and checks whether the network and domain name resolution are normal. If the preceding parameters are incorrect, reboot the shell and go back to the first step.

After the installation is successful, run the SSH -i ~/. SSH /new_rsa [email protected] command to log in to the bootstrap node from the base node and verify that:

  • Whether the network configuration conforms to their own Settings:
    • hostname
    • ip route
    • cat /etc/resolv.conf
  • Verify that the bootstrap service is successfully started:
    • podman psCheck whether the service is running as a container
    • usess -tulnpCheck whether ports 6443 and 22623 are enabled.

Bootstrap starts with podman, and then starts the temporary control plane inside the container. The temporary control plane runs in the container through CRIO, which is a bit tricky. Look directly at the command:

$ podman ps -a --no-trunc --sort created --format "{{.Command}}"

start --tear-down-early=false --asset-dir=/assets --required-pods=openshift-kube-apiserver/kube-apiserver,openshift-kube-scheduler/openshift-kube-scheduler,openshift-kube-controller-manager/kube-controller-manager,openshift-cluster-version/cluster-version-operator
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yaml
render --dest-dir=/assets/cco-bootstrap --cloud-credential-operator-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:244ab9d0fcf7315eb5c399bd3fa7c2e662cf23f87f625757b13f415d484621c3
bootstrap --etcd-ca=/assets/tls/etcd-ca-bundle.crt --etcd-metric-ca=/assets/tls/etcd-metric-ca-bundle.crt --root-ca=/assets/tls/root-ca.crt --kube-ca=/assets/tls/kube-apiserver-complete-client-ca-bundle.crt --config-file=/assets/manifests/cluster-config.yaml --dest-dir=/assets/mco-bootstrap --pull-secret=/assets/manifests/openshift-config-secret-pull-secret.yaml --etcd-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aba3c59eb6d088d61b268f83b034230b3396ce67da4f6f6d49201e55efebc6b2 --kube-client-agent-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8eb481214103d8e0b5fe982ffd682f838b969c8ff7d4f3ed4f83d4a444fb841b --machine-config-operator-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31dfdca3584982ed5a82d3017322b7d65a491ab25080c427f3f07d9ce93c52e2 --machine-config-oscontent-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b397960b7cc14c2e2603111b7385c6e8e4b0f683f9873cd9252a789175e5c4e1 --infra-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7862a735f492a18cb127742b5c2252281aa8f3bd92189176dd46ae9620ee68a --keepalived-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a882a11b55b2fc41b538b59bf5db8e4cfc47c537890e4906fe6bf22f9da75575 --coredns-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b25b8b2219e8c247c088af93e833c9ac390bc63459955e131d89b77c485d144d --mdns-publisher-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dea1fcb456eae4aabdf5d2d5c537a968a2dafc3da52fe20e8d99a176fccaabce --haproxy-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7064737dd9d0a43de7a87a094487ab4d7b9e666675c53cf4806d1c9279bd6c2e --baremetal-runtimecfg-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:715bc48eda04afc06827189883451958d8940ed8ab6dd491f602611fe98a6fba --cloud-config-file=/assets/manifests/cloud-provider-config.yaml --cluster-etcd-operator-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9f7a02df3a5d91326d95e444e2e249f8205632ae986d6dccc7f007ec65c8af77
render --prefix=cluster-ingress- --output-dir=/assets/ingress-operator-manifests
/usr/bin/cluster-kube-scheduler-operator render --manifest-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:187b9d29fea1bde9f1785584b4a7bbf9a0b9f93e1323d92d138e61c861b6286c --asset-input-dir=/assets/tls --asset-output-dir=/assets/kube-scheduler-bootstrap --config-output-file=/assets/kube-scheduler-bootstrap/config
/usr/bin/cluster-kube-controller-manager-operator render --manifest-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:187b9d29fea1bde9f1785584b4a7bbf9a0b9f93e1323d92d138e61c861b6286c --asset-input-dir=/assets/tls --asset-output-dir=/assets/kube-controller-manager-bootstrap --config-output-file=/assets/kube-controller-manager-bootstrap/config --cluster-config-file=/assets/manifests/cluster-network-02-config.yml
/usr/bin/cluster-kube-apiserver-operator render --manifest-etcd-serving-ca=etcd-ca-bundle.crt --manifest-etcd-server-urls=https://localhost:2379 --manifest-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:187b9d29fea1bde9f1785584b4a7bbf9a0b9f93e1323d92d138e61c861b6286c --manifest-operator-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:718ca346d5499cccb4de98c1f858c9a9a13bbf429624226f466c3ee2c14ebf40 --asset-input-dir=/assets/tls --asset-output-dir=/assets/kube-apiserver-bootstrap --config-output-file=/assets/kube-apiserver-bootstrap/config --cluster-config-file=/assets/manifests/cluster-network-02-config.yml
/usr/bin/cluster-config-operator render --config-output-file=/assets/config-bootstrap/config --asset-input-dir=/assets/tls --asset-output-dir=/assets/config-bootstrap
/usr/bin/cluster-etcd-operator render --etcd-ca=/assets/tls/etcd-ca-bundle.crt --etcd-metric-ca=/assets/tls/etcd-metric-ca-bundle.crt --manifest-etcd-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aba3c59eb6d088d61b268f83b034230b3396ce67da4f6f6d49201e55efebc6b2 --etcd-discovery-domain=test.example.com --manifest-cluster-etcd-operator-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9f7a02df3a5d91326d95e444e2e249f8205632ae986d6dccc7f007ec65c8af77 --manifest-setup-etcd-env-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31dfdca3584982ed5a82d3017322b7d65a491ab25080c427f3f07d9ce93c52e2 --manifest-kube-client-agent-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8eb481214103d8e0b5fe982ffd682f838b969c8ff7d4f3ed4f83d4a444fb841b --asset-input-dir=/assets/tls --asset-output-dir=/assets/etcd-bootstrap --config-output-file=/assets/etcd-bootstrap/config --cluster-config-file=/assets/manifests/cluster-network-02-config.yml
render --output-dir=/assets/cvo-bootstrap --release-image=registry.openshift4.example.com/ocp4/openshift4@sha256:4a461dc23a9d323c8bd7a8631bed078a9e5eec690ce073f78b645c83fb4cdf74
/usr/bin/grep -oP Managed /manifests/0000_12_etcd-operator_01_operator.cr.yamlCopy the code
$ crictl pods POD ID CREATED STATE NAME NAMESPACE ATTEMPT 17a978b9e7b1e 3 minutes ago Ready bootstrap-kube-apiserver-bootstrap.openshift4.example.com kube-system 24 8a0f79f38787a 3 minutes ago Ready bootstrap-kube-scheduler-bootstrap.openshift4.example.com kube-system 4 1a707da797173 3 minutes ago Ready bootstrap-kube-controller-manager-bootstrap.openshift4.example.com kube-system 4 0461d2caa2753 3 minutes ago Ready cloud-credential-operator-bootstrap.openshift4.example.com openshift-cloud-credential-operator 4 ab6519286f65a 3 minutes  ago Ready bootstrap-cluster-version-operator-bootstrap.openshift4.example.com openshift-cluster-version 2 457a7a46ec486  8 hours ago Ready bootstrap-machine-config-operator-bootstrap.openshift4.example.com default 0 e4df49b4d36a1 8 hours ago Ready etcd-bootstrap-member-bootstrap.openshift4.example.com openshift-etcd 0Copy the code

If there is no problem, you can continue to observe the log while continuing with journalctl-b-f -u bootkube.service

The default RHCOS user is core. To obtain root permission, run sudo su (no password required).

Master

For the controller node, create the VM first and modify the boot parameters as follows:

Ip=192.168.57.62::192.168.57.1:255.255.255.0:master1.openshift4.example.com: ens192: none nameserver = 192.168.57.60 coreos.inst.install_dev=sda Coreos. Inst. Image_url = http://192.168.57.60:8080/install/rhcos-4.4.3-x86_64-metal.x86_64.raw.gz Coreos. Inst. Ignition_url = http://192.168.57.60:8080/ignition/master.ignCopy the code

After the controller node is successfully installed, it restarts. You can also log in to the controller node using the SSH key.

Then repeat the same steps to create the other two controller nodes, taking care to change the boot parameters (IP and host name). Before creating compute nodes, run the following command on basic nodes to create production control planes:

$openShift-install --dir=/ocpinstall wait-for bootstrap-complete --log-level=debug Debug OpenShift Installer 4.4.5 DEBUG Built from commit 15eac3785998a5bc250c9f72101a4a9cb767e494 INFO Waiting up to 20m0s for the Kubernetes API at https://api.openshift4.example.com:6443... INFO API V1.17.1 Up INFO Waiting up to 40m0s for bootstrapping to complete... DEBUG Bootstrap status: complete INFO It is now safe to remove the bootstrap resourcesCopy the code

Remove the bootstrap host from the load balancer after It is now safe to remove the bootstrap resources. This article uses an endpoint from cds.yaml to remove the bootstrap host. And then reload.

Observe the boot node log:

$ journalctl -b -f -u bootkube.service ... Jun 05 00:24:12 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:12.108179 1 WaitForCEO.go :67] Waiting on condition EtcdRunningInCluster in etCD CR /cluster to be True 05 00:24:21 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:21.595680 1 WaitForCEO.go :67] Waiting on condition EtcdRunningInCluster in etCD CR /cluster to be True 05 00:24:26 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:26.250214 1 WaitForCEO.go :67] Waiting on condition EtcdRunningInCluster in etCD CR /cluster to be True 05 00:24:26 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:26.306421 1 WaitForCEO.go :67] Waiting on condition EtcdRunningInCluster in etCD CR /cluster to be True 05 00:24:29 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:29.097072 1 WaitForCEO.go :64] Cluster etCD Operator bootStrapped successfully Jun 05 00:24:29 bootstrap.openshift4.example.com bootkube.sh[12571]: I0605 00:24:29.097306 1 waitForCEO.go :58] cluster-etcd-operator bootstrap etCD Jun 05 00:24:29 bootstrap.openshift4.example.com podman[16531]: 2020-06-05 00:24:29.120864426 +0000 UTC m=+17.965364064 Container died 77971b6ca31755a89b279fab6f9c04828c4614161c2e678c7cba48348e684517 (image = quay. IO/openshift - release - dev/ocp - v4.0 - art - dev @ sha256:9 f7a02df3a5d91326d95e444e2e249f8205632ae986d6dccc7f007ec65c 8af77, name=recursing_cerf) Jun 05 00:24:29 bootstrap.openshift4.example.com bootkube.sh[12571]: bootkube.service completeCopy the code

Worker

For the compute node, create the VM first and modify the boot parameters as follows:

Ip=192.168.57.65::192.168.57.1:255.255.255.0:worker1.openshift4.example.com: ens192: none nameserver = 192.168.57.60 coreos.inst.install_dev=sda Coreos. Inst. Image_url = http://192.168.57.60:8080/install/rhcos-4.4.3-x86_64-metal.x86_64.raw.gz Coreos. Inst. Ignition_url = http://192.168.57.60:8080/ignition/worker.ignCopy the code

After the compute node is successfully installed, the compute node restarts. You can also log in to the compute node using the SSH key.

Then repeat the same steps to create the other compute nodes, taking care to change the boot parameters (IP and host name).

Log on to the cluster

You can log in to the cluster as a default system user by exporting the cluster Kubeconfig file. The Kubeconfig file, which contains information about the cluster used by the CLI to connect clients to the correct cluster and API Server, was created during the OCP installation.

$ mkdir ~/.kube
$ cp /ocpinstall/auth/kubeconfig ~/.kube/config
$ oc whoami
system:adminCopy the code

Approval of CSR

When a node is added to the cluster, two pending certificate signing requests (CSRS) are generated for each node that is added. These CSRS must be confirmed to have been approved or self-approved if necessary.

$oc get the node NAME STATUS ROLES AGE VERSION master1.openshift4.example.com Ready master, worker 6 h25m v1.17.1 Master2.openshift4.example.com Ready master worker six h39m v1.17.1 master3.openshift4.example.com Ready master and the worker 6 h15m v1.17.1 worker1.openshift4.example.com NotReady worker 5 h8m v1.17.1 worker2.openshift4.example.com NotReady worker 5 h9m v1.17.1Copy the code

The output lists all the nodes created. Look for Pending certificate signing requests (CSR) and ensure that each node added to the cluster sees client and server requests with Pending or Approved status. CSR approval requests for Pending states:

$ oc adm certificate approve xxxCopy the code

Or execute the following command to approve all CSRS:

$ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approveCopy the code

Operator Automatically initializes

After the control plane is initialized, ensure that all operators are in the Available state, that is, ensure that the values of the Available fields of all operators are True:

$oc Get Clusteroperators NAME VERSION AVAILABLE Misc, DEGRADED SINCE Authentication 4.4.5 True False False 150m Cloud-credential 4.4.5 True False False 7H7M cluster-AutoScaler 4.4.5 True False False 6H12m console 4.4.5 True False False 150M CSI-snapshot-Controller 4.4.5 True False False 6h13m DNS 4.4.5 True False False 6h37m ETCD 4.4.5 True False False 6h19m image-Registry 4.4.5 True False False 6h12m ingress 4.4.5 True False False 150m Insights 4.4.5 True False False 6h13m kube-apiserver 4.4.5 True False False 6h15m kube-controller-manager 4.4.5 True False False 6h36m Kube-scheduler 4.4.5 True False False 6h36m kube-storage-version-migrator 4.4.5 True False False 6h36M machine-API 4.4.5 True False False 6H37M machine-config 4.4.5 True False False 6H36M Marketplace 4.4.5 True False False 6H12m Monitoring 4.4.5 True False False 6h6m network 4.4.5 True False False 6H39M node- Tuning 4.4.5 True False False 6H38m Openshift-apiserver 4.4.5 True False False 6h14m openshift-controller-manager 4.4.5 True False False 6h12m openshift-controller-manager 4.4.5 True False False 6h12m Openshift-samples 4.4.5 True False False 6h11m operator-lifecycle-manager 4.4.5 True False False 6h37m Operator-lifecycle -manager-catalog 4.4.5 True False False 6h37m operator-lifecycle-manager-packageserver 4.4.5 True False False 6h15m service-ca 4.4.5 True False False 6h38m service-catalog-apiserver 4.4.5 True False False 6h38m Service-catalog-controller-manager 4.4.5 True False False 6h39M storage 4.4.5 True False False 6h12mCopy the code

If the Operator is abnormal, diagnose and rectify the fault.

Complete the installation

Finally, to complete the cluster installation, execute the following command:

$ openshift-install --dir=/ocpinstall wait-for install-complete --log-level=debugCopy the code

Notice The url and password for accessing the Web Console are displayed at the end. It doesn’t matter if you forget your password, you can view the file/ocpinstall/auth/kubeadmin – the password for the password.

To access the Web Console locally, add hosts:

192.168.57.60 console-openshift-console.apps.openshift4.example.com
192.168.57.60 oauth-openshift.apps.openshift4.example.comCopy the code

Browser to https://console-openshift-console.apps.openshift4.example.com, the input output of the above user name password to log in. After the first login, you will be prompted:

You are logged in as a temporary administrative user. Update the Cluster OAuth configuration to allow others to log in.Copy the code

You can use htpasswd to customize the administrator account as follows:

① htpasswd -c B -b users. Htpasswd admin XXXXX

② Download the users.htpasswd file to the local PC.

③ Open Global Configuration on the Web Console.

Go to OAuth, click enter, add Identity Providers of type HTPasswd and upload the users.htpasswd file.

④ To exit the current user, note that the following interface is displayed:

Select htpasswd and enter the username and password you created earlier to log in.

If the user password input window is displayed after exit, it is still the verification of kube:admin. If the above prompt is not displayed, you can manually enter the Web Console address to automatically jump.

⑤ If you log in to OAuth Details, you can see the Administrator menu item.

oauths.config.openshift.io "cluster" is forbidden: User "admin" cannot get resource "oauths" in API group "config.openshift.io" at the cluster scopeCopy the code

Therefore, you need to grant the cluster administrator permission:

$ oc adm policy add-cluster-role-to-user cluster-admin adminCopy the code

Screenshot of the Web Console:

To delete the default account, run the following command:

$ oc -n kube-system delete secrets kubeadminCopy the code

8. Reference materials

  • OpenShift 4.2 vSphere Install with Static IPs
  • OpenShift Container Platform 4.3 Deployment Record
  • Chapter 1. Installing on bare metal

Kubernetes 1.18.2 1.17.5 1.16.9 1.15.12 Offline installation package publishing address store.lameleg.com, welcome to experience. The latest SEALos V3.3.6 is used. Host name resolution configuration optimization, LVSCARE mount /lib/module to solve ipvS loading problems on startup, fix LVSCARE community netlink incompatibility with 3.10 kernel, SealOS generate 100 year certificate and other features. More features github.com/fanux/sealo… . Join the SealOS group by scanning the QR code below. The sealOS robots that have integrated sealOS can see sealOS in real time.