Author’s brief introduction

Zhang Zhibo, r&d Director of SUSE Rancher Greater China, has been active in r&d. He has experienced the technological transformation from OpenStack to Kubernetes, and has rich r&d and practical experience in the underlying operating system Linux, virtual KVM and Docker container technology.

Background: I once maintained RancherOS, an open source container operating system for Docker. Cos-toolkit, an emerging project that continues the Container OS philosophy, offers a deeper level of abstraction. This paper will analyze the design concept and use method of COS-Toolkit step by step, and share some personal thoughts.

The reflection of RancherOS

When RancherOS was launched, Docker was in its heyday. RancherOS’s goal was to become a Container OS for Docker. In addition to the basic Immutable OS standard feature, we made a number of magic changes: dockerd was changed to system-Docker instead of Systemd, so that we can manage some system services like containers. Extreme tailoring of the kernel, hoping to make it to run Docker OS at the lowest cost; Rootfs in user space is essentially an Ubuntu/CentOS/Buildroot image running container. The use of YAML configuration lists for system initialization simplifies the burden of system administrators through this abstraction, and so on.

Over time, some of the magic ideas became unsustainable upstream, such as system-Docker components that remained stuck in early versions and could not evolve. The problem with lean kernels is that without a strong community behind them, it is difficult to maintain these kernels and system components on an ongoing basis and provide a reliable guarantee of compatibility. The default console is built on the Buildroot project, and adding a package each time is tedious and complex, and Buildroot itself strives for simplicity that sometimes fails to meet the needs of the server. It is difficult to meet users’ customization requirements. Users usually need to wait for the update of the official version. The cost of user-defined technology is very high and it is difficult to get started.

RancherOS gained a small community of users in the process, and tried to commercialize on a small scale. It turned the idea into reality and was a good open source project for a specific group of people. However, from a sustainable perspective, stopping maintenance is a more rational option. However, you can see that the community is still publishing some BurmillaOS (burmillaos.org/), a derivative of RancherOS.

COS – Toolkit to take over the next leg

Cos-toolkit is a toolkit for building ContainerOS. Previously RancherOS was itself a Linux distribution, and cos-toolkit provides greater abstraction for building such a distribution. Drawing on the experience of previous open source projects and incorporating the mainstream needs of the current market, COS-Toolkit brings the following key features:

  • Build and upgrade in Container mode. Cos-toolkit uses the open source tool Luet (github.com/mudler/luet… Packages (github.com/rancher-san… OS. This approach builds on the strong community behind each Linux distribution, and coS-Toolkit does not need to focus on how a particular package is compiled and integrated, nor does it break the habits of every distribution enthusiatic.

  • Support OTA updates. The OTA of Container OS can build the updated content and publish it to the image repository. With Kubernetes’ extended orchestration capabilities, system upgrades are performed on all nodes. The upgrade action triggers a specific Container image update. And applied to the OS built by cOS- Toolkit.

  • Cloud-init support, based on Systemd, and immutable features. Cloud-init support directly reflects the friendliness of the public Cloud environment; And systemD-based is an important choice for sustainable development; Support for Immutable takes precedence throughout the design and is a standard feature of almost all Container OS.
  • Simple and convenient customization. Cos-toolkit is no longer just a distribution. It provides more powerful custom build capabilities. In addition to referring to the packages maintained in CoS-Toolkit, users can also customize packages for extension. Cos-toolkit also simplifies various OEM configurations (Rancher-sandbox.github. IO/COs-Toolkit…

The COS-Toolkit project builds some base distributions by default:

  • Blue: Based on Fedora
  • Green: Based on openSUSE
  • Orange: Based on Ubuntu

Since the core engineers are from SUSE, the OpenSUSe-based variants are more fully tested. These images can be found at Github Release (github.com/rancher-san…

SUSE Rancher is already using COs-Toolkit to build emerging products such as Harvester OS and RancherOS V2. The former provides a base OS for Harvester clusters, allowing harvesters to manage nodes at a deeper level. The latter will focus on providing a base OS for Rancher as well as K3s/RKE2 to further simplify product deployment and maintenance. RancherOS V2 is also actively incubating open source: github.com/rancher-san…

COS – toolkit early experience

The public cloud friendly experience is already a standard for emerging operating systems. The Green (Based on openSUSE) variant built by COS-Toolkit fully supports public clouds such as AWS/Azure/GCP. We take AWS version as an example for initial experience. AMI information can be released at Github (github.com/rancher-san…

As these amis are started in UEFI mode by default when they are built, some instance types cannot be supported. T3.large is used in this experience.

By default, the system initially enters the Recovery mode. Users can customize the system installation in this mode by using the built-in installation tool and predefined YAML configurations. However, since we are using the AWS cloud environment, we can simplify this process with cloud-init mechanisms. When starting an instance based on cOS AMI (root disk 16GiB), configure the following cloud-init:

name: "Default deployment" stages: rootfs.after: - name: "Repart image" layout: # It will partition a device including the given filesystem label or part label (filesystem label matches first) device:  label: COS_RECOVERY add_partitions: - fsLabel: COS_STATE # 10Gb for COS_STATE, so the disk should have at least 16Gb size: 10240 pLabel: state - fsLabel: COS_PERSISTENT # unset size or 0 size means all available space pLabel: persistent initramfs: - if: '[ -f "/run/cos/recovery_mode" ]' name: "Set sshd to wait for deployment" files: - path: "/etc/systemd/system/sshd.service.d/override.conf" content: | [Unit] After=cos-setup-network.service network: - if: '[ -f "/run/cos/recovery_mode" ]' name: "Deploy cos-system" commands: - | cos-deploy --no-verify --no-cosign && shutdown -r nowCopy the code

This cloud-init userdata will help us plan the root partition table information and initialize the installation. Note that if the version is not specified when cos-deploy is deployed, the latest version is installed. This means that although we were using the V0.7.4 AMI, we were looking for the newer version in the source to deploy to the root disk after cos-deploy, which is actually part of the OTA update mechanism.

After the system is installed and initialized, it can be accessed through SSH. Since AWS metadata has not been fully adapted, user/password is still used for access for the time being, and we have not changed the password in userdata. Therefore, you can use the default user name and password root and cos. The system will be exceptionally clean and clean, and since the Green variant is based on the openSUSE distribution, the kernel version is the same as openSUSE 15.3, and the system version is also traced back to V0.8.2.

In addition to using the cloud-init mechanism to install the system, you can also manually install the system using cos-deploy. During the execution, you can clearly see that a relatively new system image of V0.8.2 is pulled.

Of course, we can also make a custom image and push it to the image repository, system image construction can rely on the Dockerfile mechanism. Take the standard example in Github Repo (github.com/rancher-san… After openSUSE, software packages are installed by executing Zypper in Dockerfile, and these packages rely on a strong community to ensure quality:

IO /luet/base:$LUET_VERSION AS luet FROM OpenSUSE/LEAP :15.3 ENV COSIGN_EXPERIMENTAL=1 ENV COSIGN_REPOSITORY=raccos/releases-green ARG ARCH=amd64 ENV ARCH=${ARCH} RUN zypper in -y \ bash-completion \ conntrack-tools \ coreutils \ curl \ device-mapper \ dosfstools \ dracut \ kernel-default \ kernel-firmware-bnx2 \ kernel-firmware-i915 \ kernel-firmware-intel \ kernel-firmware-iwlwifi \ kernel-firmware-mellanox \ Kernel -firmware-network kernel-firmware-platform kernel-firmware-realtek less ...Copy the code

We put the Dockerfile build image into a container (such as: niusmallnan/containeros: dev), and push to Dockerhub. On the AWS VM you just started: Cos – upgrade – no – verify – reboot – d niusmallnan/containeros: dev, automatic restart after, we can get a new version of the OS, this is the foundation of the OTA update based on the pattern of container. If you visit the VM, you can see that the OS version, built-in software package, and kernel version are all changed:

Cos-toolkit is a work in progress, and there is room for improvement in many details. Cos-toolkit does not provide a continuous release. Instead, it maintains a set of tools for building a personalized Container OS, enabling users to build and publish the OS and enjoy container-image-style OTA updates.

For more information, please refer to rancher-sandbox.github. IO/cos-Toolkit…