K8s will not use Docker as a container as early as version 1.23, K8s said in a blog post that it can use CRI runtime such as Containerd instead of Docker. This article takes a closer look at how Docker relates to Containerd and why containerd is a better choice. Here’s an answer to the TKE user’s question: What about our cluster?

What about the TKE cluster

  • TKE has supported the selection of Containerd as a container runtime since May 2019. If you create a cluster, it is recommended that you select Containerd as the container runtime
  • Existing clusters can continue to use Docker as a container runtime until they are upgraded to K8s 1.23 (or possibly 1.24, assuming TKE is the first K8s version that does not support Dockershim)
  • When an existing cluster is upgraded to 1.23 using the TKE Cluster upgrade function, TKE provides the option to switch the runtime to Containerd. Of course, in this case, there is no way to make the Pod not affected, but to reinstall the node to upgrade
  • Existing clusters can also be switched to Containerd at runtime. New nodes will use Containerd. Existing nodes will still use Docker. This will cause the Coexistence of Docker nodes and Containerd nodes in the same cluster. If there are services using Docker in Docker or Docker daemon and Docker. Sock on other dependent nodes, measures should be taken in advance to avoid problems. For example, by scheduling according to node labels, this kind of business is guaranteed to be scheduled to docker nodes. Or run Docker in Docker on containerd cluster as described above)
  • Of course, docker may also implement CRI internally or add a Dockershim process in the future. If Docker makes corresponding adaptations, TKE will also support it in the future.

K8s discard Dockershim

Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called “dockershim” which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#94624, @dims) [SIG Node]

K8s mentioned in the change log of 1.20 that K8s will gradually drop support for Docker from 1.20. The official K8s blog also contains specific statements and some FAQs.

  • Don’t Panic: Kubernetes and Docker
  • Dockershim FAQ

K8s will add the information that docker is not recommended in version 1.20, and dockershim will be removed from Kubelet as early as version 1.23. Then users will not be able to use Docker as the K8s cluster runtime. However, images built with Docker can still be used in K8s clusters without Docker.

“Parasitic” in Kubelet dockershim

The main content of this change is to delete Dockershim in Kubelet, of course, this approach is also in line with expectations. In the early days when RKT and Docker were competing for power, kubelet needed to maintain two batches of code to adapt docker and RKT respectively, which made Kubelet need to consider the adaptation of runtime components every time it released new functions, which seriously slowed down the release of new versions. In addition, virtualization is already a common requirement, and if a type of runtime appears, the Sig-Node team may need to add code to Kubelet that matches the new runtime. This didn’t work for long, so in 2016, SIG-Node introduced a container operation interfaceCRI(Container Runtime Interface). CRI is a set of abstractions for container operations that Kubelet can use to accommodate all runtimes as long as each container runtime implements this set of interfaces. However, Docker did not (and did not intend to) implement this set of interfaces, and Kubelet had to maintain an internal component called “Dockershim” that acted as a CRI adapter for Docker. Kubelet calls Dockershim through the CRI interface when creating the container, and DockerShim passes requests to Docker through HTTP requests. So Kubelet’s architecture looks like this:In the case of the container runtime using components that implement the CRI interface, Kubelet creates the call chain of the container as shown in the red arrow. The ContainerManager in Kubelet can call the container runtime directly through CRI with only one GRPC request. And when using docker, ContainerManager go blue invocation chain in the figure, the request of the CRI by Unix: / / / var/run/dockershim sock to dockershim, Dockershim does the transformation and forwards the request to Docker, and we’ll talk about why there’s a containerd behind Docker later. Implementing the docker adapter in Kubelet is inherently an inelegant implementation, which makes the call chain long and unstable, and also adds extra work to kubelet maintenance. It is only a matter of time to remove this part from Kubelet.

What will happen when Docker is abandoned?

If you’re an end-user of Kubernetes, not a whole lot will be changing for you. This doesn’t mean the death of Docker, and it doesn’t mean you can’t, or shouldn’t, use Docker as a development tool anymore. Docker is still a useful tool for building containers, and the images that result from running docker build can still run in your Kubernetes cluster.

After the news came out, the most important thing that people are concerned about is what will happen after docker is abandoned?

The official reply: Don’t Panic! Then he highlights some of the issues that people are most concerned about. Let’s take a look at the aspects mentioned by officials:

  • Normal K8s users will not have any impact

    Yes, advanced clusters in production only need to switch the runtime from Docker to another runtime such as Containerd. Containerd is a low-level docker component that maintains the container lifecycle and has been with Docker for a long time. At the same time, I will graduate from CNCF at the beginning of 2019 and can be used in clusters as a separate container runtime. TKE has also offered Containerd as a run-time option since 2019, so converting runtime from Docker to Containerd is a largely painless process. Cri-o is another commonly mentioned runtime component, provided by Redhat. It is lighter than Containerd, though it is different from Docker and may have some differences when converted.

  • Images built from docker builds in the development environment can still be used in the cluster

    Mirroring has long been a strength of the container ecosystem, and while it’s often referred to as “Docker mirroring,” mirroring has long been the norm. Refer to image-spec for the specification. Anywhere you build an Image that matches the Image Spec, you can run it on other container runtimes that match the Image Spec.

  • Users who use DinD (Docker in Docker) in Pod will be affected

    Some users will mount docker socket (/run/docker.sock) to Pod, and call DOCker API in Pod to build image or create compilation container, etc. The official recommendation here is to use Kaniko, Img or Buildah. We can use DinD in any runtime by using Docker daemon as a DaemonSet daemon or by adding a Sidecar to the Pod that wants to use Docker. TKE also provides solutions specifically for using DinD in containerd clusters, as described in Using DinD in Containerd.

Containerd’s past life

So what exactly is Containerd? How does it relate to Docker? Some of you might be wondering about containerd and Docker when you read this blog.

The docker and containerd

In 2016, Docker split out the module responsible for the container lifecycle and donated it to the community, now called Containerd. The structure of Docker after splitting is shown in the figure below (of course, Docker company also added part of the choreographed code in Docker).After we call the Docker command to create the container, the Docker Daemon will download the Image through the Image module and save it to the Graph Driver module, and then use the client to call Containerd to create and run the container. We may need to use this when creating containers with Docker--volumeAdd persistent storage to the container; And maybe get through--networkConnect to several containers created by docker command. Of course, these functions are provided by docker Storage module and Networking module. However, K8s provides stronger volume mounting capability and cluster-level network capability. In the cluster, Kubelet will only use the image download and container management functions provided by Docker, while the choreography, network, storage and other functions will not be used. The following figure shows the call chain of each module in the current mode, and the modules highlighted in the red box are the run-time modules that Kubelet relies on to create pods.Once containerd was donated to the CNCF community, the community added an image management module and a CRI module so that it could not only manage the container life cycle, but also directly serve as the K8s runtime. Containerd graduated from the CNCF community in February 2019 and went into production. As you can see in the figure below, running containerd as a container gives Kubelet all the functionality needed to create a Pod, along with purer functional modules and a shorter call chain.As you can see from the comparison above, containerd’s goal has been to be a simple, stable, and reliable container runtime since it was donated to the community; Docker wants to be a complete product. Docker provides a lot of features that developers need in order to give users a better interaction and use experience and more functions. Docker also provides network and volume functions to build the foundation for Swarm. These functions are not available on the K8s; Containerd, on the other hand, provides only the basic functionality kubelet needs to create a Pod, in return for greater robustness and better performance. In some ways, containerd is still a better choice, even though Docker offers CRI after Kubelet 1.23.

Use Containerd in a Kubernetes cluster

Of course, there are many CRI implementors out there, including Containerd and CRI-O. Cri-o is a CRI runtime developed primarily by Red Hat employees and has nothing to do with Docker at all, so migrating from Docker can be difficult. There is no doubt that Containerd is the best candidate for the CRI runtime after Docker is abandoned. For developers, the whole migration process should be unconscious, but for some operations, students may pay more attention to the differences in the details of deployment and operation. Let’s highlight some of the differences between using Containerd and Docker in K8s.

  • Container log comparison items
Compare the item Docker Containerd
The store path If docker as runtime K8s container, the container log trading will be completed by the docker, stored in a similar/var/lib/docker/containers /
C O N T A I N E R I D Directory. k u b e l e t Will be in / v a r / l o g / p o d s and / v a r / l o g / c o n t a i n e r s The soft link below is set up to point to / v a r / l i b / d o c k e r / c o n t a i n e r s / In the CONTAINERID directory. Kubelet in/var/log/pods and/var/log/containers to establish soft links below, pointing to the/var/lib/docker/containers / CONTAINERID Container log file in the directory.		 If Containerd is running as a K8s container, the container logs are dropped by Kubelet, saved to /var/log/Pods /$CONTAINER_NAME, and soft links are created in /var/log/containers. Point to the log file.
Configuration parameters {“max-size”: “100m”,”max-file”: “5”} –container-log-max-files=5 –container-log-max-size=”100Mi” “containerLogMaxSize”: “100Mi”, “containerLogMaxFiles”: 5,
Save the container logs to the data disk Mount the data disk to data-root (/var/lib/docker by default). Create a soft link /var/log/Pods to a directory under the data disk mount point. If you select “Store containers and images on data Disk” in TKE, the soft link /var/log/Pods is automatically created.

  • In the case of Containerd, the built-in Containerd-crI plug-in is used to call the CNI. So about the cni configuration file must be placed on containerd configuration file (/ etc/containerd/containerd toml) :

     [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
Copy the code

  • Stream service differences

    Description:

    Commands such as Kubectl exec/logs require the establishment of a flow channel between apiserver and the container runtime.

    How do I use and configure the Stream service in Containerd?

    Docker API itself provides stream service, docker-shim inside Kubelet will do the flow through Docker API. Containerd’s Stream service needs to be configured separately:

    [plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "0" Enable_tls_streaming = false [plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "0" Stream_server_address = "127.0.0.1" stream_server_port = "0" enable_TLs_streaming = falseCopy the code

What are the differences between K8s 1.11 and K8s 1.11?

Containerd’s Stream service is configured differently in different K8s runtime scenarios.

  • Before K8s 1.11: Kubelet will not do stream proxy, only redirection. Kubelet sends the stream server address exposed by Containerd to Apiserver and gives Apiserver direct access to Containerd’s Stream service. In this case, you need to authenticate the stream forwarder for security protection.
  • After K8s1.11: K8S1.11 introduced the Kubelet Stream Proxy, enabling the Containerd Stream service to listen only for local addresses.

Use Containerd in a TKE cluster

TKE has been supporting Containerd as a container runtime option since May 2019. As TKE continues to support log collection and GPU capabilities in the Containerd cluster, containerd also dropped the Beta label on TKE in September 2020 and is now available in production. Over the long term, we found some containerd issues and fixed them in time, such as:

  • Pod Terminating due to error handling problems
  • The image file is lost due to a kernel version problem

There are three ways to use Containerd as a runtime in a TKE cluster:

  1. When creating a cluster, select K8s 1.12.4 or later and select Containerd as the runtime component

  2. Create a subset of Containerd nodes in an existing Docker cluster by creating a node pool that is containerd at runtime (New Node Pool > More Settings > Runtime Components)

  3. In an existing Docker cluster, change the “runtime Component” property of the cluster or node pool to “containerd”

Note: The latter two methods cause the coexistence of Docker nodes and Containerd nodes in the same cluster. If docker in Docker or docker daemon and Docker. Sock services are used on other dependent nodes, Measures should be taken in advance to avoid problems, such as scheduling by node label to ensure that such business is scheduled to docker node; Or run Docker in Docker in the Containerd cluster as described earlier.

For containerd and Docker options, see here.

reference

[1] Don’t Panic: Kubernetes and Docker

[2] Dockershim FAQ

[3] Dockershim Removal Kubernetes Enhancement Proposal

[4] kubernetes CHANGELOG – 1.20