We tried a variety of different Docker development environments

Many of the company’s colleagues have recently switched to new MacBook Pro models with M1 Pro or M1 Max, and while everyday software such as Chrome, Visual Studio Code and Slack has adapted well, Docker has struggled.

Docker is known to use two Linux features: Namespaces and Cgroups to provide isolation and resource restrictions, so we would have to use Docker through a virtual machine on macOS anyway.

In April 2021, Docker for Mac released experimental support for Apple Silicon, which uses QEMU to run an ARM based Linux VIRTUAL machine, running an ARM based image by default. However, images running x86 are also supported.

QEMU is an open source Virtualizer and Emulator that allows QEMU to simulate a computer without virtualization support from the hardware or operating system, including a CPU architecture different from that of the host computer. For example, simulating x86 computers on the Apple Silicon. In the case of hardware virtualization, QEMU can also be run directly using the host CPU to reduce the performance overhead of emulation, such as the Hypervisor.Framework provided by macOS.

Docker for Mac actually uses the two capabilities of QEMU respectively to run x86 images on ARM virtual machines and ARM virtual machines on Mac.

Docker for Mac is really good. In addition to addressing the problems of the new architecture, Docker maps the file system to the network, and containers can access the file system as if they were running on the local machine or expose network ports to the local machine, with virtually no virtual machine presence. But when LeanCloud joined TapTap, it was no longer a small company, and we each had to pay at least $5 a month for the Docker Desktop’s new pricing plan, which came out in August 2021. It’s not that we don’t want to pay for it, it’s just that I want to find an open source solution.

Previously, on Intel Macs, we used Vagrant or Minikube to create virtual machines, which underneath used VirtualBox or HyperKit for actual virtualization. But neither VirtualBox nor HyperKit has plans to support Apple Silicon. In fact, only QEMU has good support for Apple Silicon among the current open source virtualization schemes. QEMU itself only provides the command line interface. For example, the command line parameter when Docker for Mac calls QEMU is like this:

/Applications/Docker.app/Contents/MacOS/qemu-system-aarch64 -accel hvf \ -cpu host -machine virt,highmem=off -m 2048 -smp 5 \ -kernel /Applications/Docker.app/Contents/Resources/linuxkit/kernel \ -append linuxkit.unified_cgroup_hierarchy=1 page_poison=1 vsyscall=emulate \ panic=1 nospec_store_bypass_disable noibrs noibpb no_stf_barrier mitigations=off \ Vpnkit. Connect = TCP + the bootstrap + client: / / 192.168.65.2:61473 / f1c4db329a4a520d73a79eaa1360de7be7d09948a1ac348b04c8e01f6f6eb2c 9 \ console=ttyAMA0 -initrd /Applications/Docker.app/Contents/Resources/linuxkit/initrd.img \ -serial pipe:/var/folders/12/_bbrd4692hv8r9bx_ggw5kp80000gn/T/qemu-console1367481183/fifo \ -drive if=none,file=/Users/ziting/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw,format=raw,id=hd0 \ -device virtio-blk-pci,drive=hd0,serial=dummyserial -netdev socket,id=net1,fd=3 -device virtio-net-device,netdev=net1,mac=02:50:00:00:00:01 \ -vga none -nographic -monitor noneCopy the code

To actually develop with QEMU, we need a more user-friendly package that automatically configures Docker and Kubernetes (or at least makes writing scripts like Vagrantfile easy), It provides network mapping and file mapping similar to Docker for Mac, so I found Lima.

Lima, which calls itself the macOS Subsystem for Linux, uses QEMU to run a Linux VIRTUAL machine with containerd installed in rootless mode, File mapping and automatic port forwarding are also provided over SSH.

But why Containerd and not Docker? The Container Runtime Interface (CRI) was created by Kubernetes, a Container programming platform, to standardize the critical part of running containers and make it easier to introduce runtimes other than Docker. Containerd is a CRI implementation split out of Docker, which is much more streamlined than the Docker ontology and is now maintained by the community.

So newer open source software such as Lima may prefer Containerd to run containers because of its leaner components, better performance, and less vulnerability to Docker production-level changes. Nerdctl is a command line client that comes with Containerd (nerd is the last four letters of Containerd). Nerdctl is similar to, but not fully compatible with, docker or docker-compose.

Rootless refers to replacing some components so that both containerd and containerd are run by non-root users. Each user has its own containerD, so that most operations do not need to be switched to root and the vulnerability is reduced.

But we want to be able to run a full Rootful dockerd and Kubernetes locally to simulate as much of an online environment as possible, and Lima offers a lot of customizations. I implemented our requirements based on scripts from the community (docker.yaml and minikube.yaml), and the custom logic was written into a yamL description file as a script that created the same virtual machine with a single command.

~ ❯ limactl start docker.yaml? Creating an instance "docker" Proceed with the default configuration INFO[0005] Attempting to download the image from "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img" INFO[0005] Using cache "/Users/ziting/Library/Caches/lima/download/by-url-sha256/ae20df823d41d1dd300f8866889804ab25fb8689c1a68da6b13dd60a8c5c9e 35/data" INFO[0006] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/ziting/.lima/docker/serial.log") INFO[0006] SSH Local Port: 55942 INFO[0006] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" INFO[0039] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh" INFO[0039] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed" INFO[0048] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\"" INFO[0051] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running" INFO[0051] [hostagent] Mounting "/Users/ziting" INFO[0051] [hostagent] Mounting "/tmp/lima" INFO[0052] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/ziting/.lima/docker/ga.sock" (host) INFO[0092] [hostagent] Waiting for the optional requirement 1 of 1: INFO[0154] [hostagent] Forwarding TCP from [::]:2376 to 127.0.0.1:2376 INFO[0304] [hostagent] Forwarding TCP from [::]:8443 to 127.0.0.1:8443 INFO[0332] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished" INFO[0351] READY. Run `limactl shell docker` to open the shell. INFO[0351] To run `docker` on the host (assumes docker-cli is installed): INFO[0351] $export DOCKER_HOST= TCP ://127.0.0.1:2376 INFO[0351] To run 'kubectl' on the host (assumes Kubernetes -cli is installed): INFO[0351] $ mkdir -p .kube && limactl cp minikube:.kube/config .kube/configCopy the code

I also found another Lima-based package, Colima, which provides Rootful Dockerd and Kubernetes by default, but Colima doesn’t show Lima’s strong customizations, so we didn’t use it. But it’s also an easier option for developers who don’t have as many requirements.

By default, Docker in Lima can only run images of the ARM architecture on Apple Silicon, but as mentioned earlier, we can use QEMU’s emulation capabilities to run containers of other architectures (such as x86). Qemu-user-static is a process-level emulator that runs other architectures’ executables like an interpreter, We can use a Linux Binfmt_misc feature to make Linux automatically call qemu-user-static when it encounters architecture-specific executables. This capability also applies to executables in containers.

There are programs like QUS in the community that encapsulate these capabilities, Just one line of Docker run –rm — Privileged aptman/ qus-s — -p x86_64 will magically enable your ARM VIRTUAL machine to run x86 images.

/usr/bin/containerd-shim-runc-v2
 \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
Copy the code

As shown above, all processes (including created child processes) are automatically run through QEMU emulation.

Because Docker relies on the features of the Linux kernel, it must be run on a Mac using a virtual machine. As Apple Silicon is a new architecture, the choice of virtual machines is relatively limited, because some images do not provide images of ARM architecture, so sometimes there are requirements to simulate running x86 images; Docker Desktop, as a commercial product, had plenty of energy to do the dirty work, but it chose not to allow everyone to use it for free at this point; New projects in the open source community want to de-docker-ize and replace Dockerd with Containerd, but this introduces a change in usage habits and may be inconsistent with the online environment. For these reasons, installing Docker on Apple Silicon currently takes some background work, but there are still excellent open source projects to choose from.

While the cloud engine is also built on container technologies like Docker, the cloud engine aims to provide users with an out-of-the-box experience without having to configure the container environment, write build scripts, and collect logs and statistics themselves. If you want the smooth deployment, fast rollback, and automatic expansion benefits of containerization, but don’t want to spend time configuring it, try a cloud engine.

Other references:

  • www.tutorialworks.com/difference-…

Photo by Sigmund on Unsplash.