Author: Mi Qianyang Jiyan, Fang Tian
Cloud native is taking over the software world. Containers have changed the traditional application development model. Now developers not only build applications, but also use Dockerfile to complete the containerization of applications, packaging applications and their dependencies, so as to obtain more reliable products and improve the efficiency of research and development.
As the project iterates and reaches a certain scale, there is a need for collaboration between the operations and r&d teams. The operations team has a different perspective from the R&D team, and their needs for mirroring are safety and standardization. Such as:
- Which base image should different applications choose?
- What versions of the application’s dependencies are there?
- What ports does the application need to expose?
In order to optimize operation and maintenance efficiency and improve application security, developers need to constantly update Dockerfile to achieve the above goals. The o&M team also interferes with the build of the image. If a CVE is fixed in the base image, the O&M team needs to update the Dockerfile to use a newer version of the base image. In short, both o&M and R&D need to intervene in Dockerfile, which cannot be decoupled.
To address this set of issues, better products have emerged to Buildpacks, including Cloud Native Buildpacks (с NB). CNB provides a faster, safer, and more reliable way to build OCI compliant images based on modularization, enabling decoupling between R&D and operations teams.
Before introducing CNB, let’s explain some basic concepts.
Images that conform to the OCI specification
Container runtime is no longer the sole domain of Docker today. To ensure that all container runtimes can run images generated by any build tool, the Linux Foundation, along with Google, Huawei, HP, IBM, Docker, Red Hat, VMware, and others, announced the Open Container Project (OCP), later renamed the Open Container Initiative (OCI). OCI defines industry standards around container image formats and runtimes. Given an OCI image, any container runtime that implements the OCI runtime standard can run the container using that image.
If you’re asking what’s the difference between a Docker image and an OCI image, the answer today is: almost nothing. Some old Docker images existed before the OCI specification. They are called Docker V1 specification and are incompatible with Docker V2 specification. The Docker V2 specification was donated to OCI, forming the basis of OCI specification. All of today’s container image repositories, Kubernetes platform, and container runtimes are built around the OCI specification.
What is a Buildpacks
Buildpacks was first launched by Heroku in 2011 and is widely adopted by The PaaS platform represented by Cloud Foundry.
A Buildpack is a program that turns source code into a runnable package for the PaaS platform. Typically, each Buildpack encapsulates the toolchain of a single language ecosystem, Ruby, Go, NodeJs, Java, Python, and so on all have dedicated buildpacks.
You can think of BuildPack as a bundle of scripts that bundle up your application’s executable files and their dependent environments, configuration, startup scripts, etc., and upload it to a repository like Git. The bundle is called Droplet.
Cloud Foundry then uses the scheduler to select a VIRTUAL machine that can run the application, notifies the Agent on that machine to download the application zip, and launches the application according to the buildPack startup command.
By January 2018, Pivotal and Heroku had launched the Cloud Native Buildpakcs(CNB) project, and by October of that year, the project had made its way to CNCF.
In November 2020, THE CNCF Technical Oversight Committee (TOC) voted to upgrade CNB from a sandbox project to an incubator project. It’s time to take a good look at CNBS.
Why do WE need Cloud Native Buildpacks
Cloud Native Buildpacks(CNB) can be viewed as cloud-native Buildpacks. It supports a modern language ecosystem, shielding developers from the details of application building and deployment, such as which OS to choose, writing processing scripts for mirrored OS, optimizing image sizes, and so on. It also produces OCI container images that can run in any cluster that is compliant with the OCI image standard. CNB also embraces many more cloud-native features, such as BLOB mounts across mirrored repositories and mirrored tier Reps.
Buildpacks provides a higher level of abstraction for building applications than Dockerfile. Buildpacks provides a higher level of abstraction for BUILDING OCI images. Similar to Helm’s abstraction of Deployment orchestration.
In October 2020, Google Cloud began announcing full support for Buildpacks, including Cloud Run, Anthos, and Google Kubernetes Engine (GKE). Companies like IBM Cloud, Heroku, and Pivital have all adopted Buildpacks, and if nothing goes wrong, other Cloud vendors will soon follow suit.
Buildpacks benefits:
- There is no need to write build files repeatedly for applications with the same build purpose (just use one Builder).
- Do not rely on Dockerfile.
- You can easily check what each layer (BuildPacks) is working on based on rich metadata information (buildpack.toml).
- After changing the underlying operating system, there is no need to rewrite the image build process.
- Secure and compliant application builds without developer intervention.
The Buildpacks community also provides a table to compare comparable app packs:
Buildpacks supports more than any other packaging tool, including caching, source code detection, plugins, support for Rebase, reuse, CI/CD ecology.
How Cloud Native Buildpacks work
Cloud Native Buildpacks consists of three components: Builder, Buildpack, and Stack.
Buildpack
Buildpack is essentially a collection of executable units, including checking application source code, building code, generating images, and so on. A typical Buildpack will include the following three files:
- Buildpack.toml – Provides buildPack metadata information.
- Bin /detect – Detects whether this buildpack should be executed.
- Bin /build – Executes the build logic of buildpack and eventually builds the image.
Builder
Buildpacks completes a build logic with three actions: Detect, build, and export. Typically, we use multiple Buildpacks to build an application, so the Builder is a set of build logic that contains a mirror image of all the components and runtime environments needed for the build.
Let’s try to understand how Builder works by going through a hypothetical pipeline:
- Initially, as the application developer, we prepared a copy of the application source code, which we identified here as “0”.
- Then apply “0” to the first step, which we’ll process with Buildpacks1. In this process, Buildpacks1 checks to see if the application has a “0” identifier, and if it does, it starts the build process by adding a “1” to the application identifier and changing it to “01”.
- Similarly, the second and third processes will also determine whether they need to execute their own build logic based on their own access conditions.
In this example, the application meets the entry criteria for all three processes, so the final output OCI image will have the identifier “01234”.
In the concept of Buildpacks, Builders are an ordered combination of Buildpacks, including a base image called Build Image, a lifecycle, and an application to another base image called Run Image. The Builders are responsible for building the application source code into an app image.
Build Image provides the base environment for Builders (such as an Ubuntu Bionic OS image with a build tool), while Run Image provides the base environment for an App image at runtime. The combination of build image and run image is called a Stack.
Stack
As mentioned above, the combination of build image and Run Image is called a Stack, that is, it defines Buildpacks’ execution environment and the base image of the final application.
You can takebuild imageUnderstood as the base image of the first phase of the Dockerfile multi-phase build, willrun imageUnderstood as the base image of stage 2.
All three of the above components are in the form of Docker images and provide very flexible configuration options, as well as the ability to control each layer of images generated. Combined with its powerful caching and rebasing capabilities, custom component images can be reused by multiple applications and each layer can be individually updated as needed.
Lifecycle is the most important concept in Builder, it abstracts the build steps from applying source code to an image, orchestrates the entire process, and ultimately produces an application image. Lifecycle is described in a separate chapter.
Build Lifecyle
Lifecycle takes all of Buildpacks’ detection and Build process and aggregates it into two large steps: Detect and Build. This reduces the architectural complexity of Lifecycle and makes it easier to implement custom Builders.
In addition to the two main steps of Detect and Build, Lifecycle includes some additional steps that we will examine together.
Detect
As we mentioned earlier, as Buildpack includes a /bin/detect file for probes, Lifecycle guides all Buildpacks of /bin/detects in order during detect and takes the results from that execution.
How will Lifecycle maintain the relationship between Detect and Build once they are separated?
Buildpacks, in both the Detect and Build phases, usually tells itself what prerequisites are required in the process and what results it provides.
Within Lifecycle, a structure called a Build Plan is provided to hold what is needed and what is produced for each Buildpack.
type BuildPlanEntry struct {
Providers ` ` toml: "will"
Requires `toml:"requires"`
Copy the codeLifecycle also dictates that Buildpacks can only be assembled into a Builder if all outputs match with a corresponding need.
Analysis
Buildpacks creates directories at runtime that are called layer in Lifecycle. So for these layers, some will be available as a cache for the next Buildpacks, some will need to work at runtime, and some will need to be cleaned up. How can I control these layers more flexibly?
Lifecycle provides three switch parameters that represent the expected processing for each layer:
- Launch indicates whether the layer will be activated when the application runs.
- Build indicates whether this layer will be accessed during subsequent builds.
- Cache indicates whether this layer will be used as a cache.
Lifecycle then determines the final destination of the layer according to a relational matrix. We can also simply say that the Analysis phase provides a cache for building and running applications.
Build
The Build phase uses the Build plan produced in the Detect phase and the metadata information in the environment, along with layers retained to this phase, to execute the Build logic in Buildpacks on the application source code. The result is a runnable application artifact.
Export
The Export phase is easier to understand. After completing the above build, we need to produce the final build as an OCI image so that the App artifact can run in any OCI-compliant cluster.
Rebase
In the design of CNB, the app artifact actually runs on the Stack’s Run Image at last. The artifact above the Run Image is a whole. It is connected to the Run image in the form of an ABI(Application Binary Interface), which allows the artifact to be flexibly switched to another Run image.
This action is actually part of Lifecycle and is called rebase. There is also a rebase during image building, which occurs when the App artifact switches from Build Image to Run Image.
This mechanism is also where CNB has the greatest advantage over Dockerfile. For example, in a large production environment, if there is a problem with the OS layer of the container image and the OS layer of the image needs to be changed, then the image for different types of application needs to rewrite their dockerfile and verify whether the new dockerfile is feasible, and whether there is a conflict between the new layer and the existing layer. And so on. With CNB, only one rebase is required, simplifying the upgrade of images in mass production.
The above is the analysis of the process of CNB building the image. In summary:
- Buildpacks are minimal build units that perform specific build operations;
- Lifecycle is the mirror build Lifecycle interface provided by CNB;
- A Builder is a set of Buildpacks plus Lifecycle and stack built for a specific purpose.
Further refinement:
- build image + run image = stack
- stack(build image) + buildpacks + lifecycle = builder
- stack(run image) + app artifacts = app
So now the question is, how do you use this tool?
Platform
There is a need for a Platform and Platform is the executor of Lifecycle. What it does is apply the Builder to a given source code to complete Lifecycle.
During this process, the Builder builds the source code into the app, which is in the Build Image. At this time Lifecycle will have the underlying logic to convert app artifacts from build Image to Run Image using the ABI(Application Binary Interface) according to the Rebase interface in Lifecycle. This is the final OCI image.
Common platforms include Tekton and CNB’s Pack. Next, we’ll use Pack to see how to build mirrors using Buildpacks.
Install the Pack CLI tool
Currently, the Pack CLI supports Linux, MacOS, and Windows. Ubuntu is used as an example.
$ sudo add-apt-repository ppa:cncf-buildpacks/pack-cli
$ sudo apt-get update
$ sudo apt-get install pack-cli
Copy the codeView version:
$pack version 0.22.0 + git - 26 d8c5c. Build - 2970Copy the codeNote: Docker needs to be installed and running before you can use Pack.
Currently, Pack CLI only supports Docker, not other container runtimes (such as Containerd, etc.). However, Podman can use a few hacks to support this, and in Ubuntu’s case, the steps are as follows:
Start by installing Podman.
$ . /etc/os-release
$ echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
$ curl -L "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get -y upgrade
$ sudo apt-get -y install podman
Copy the codeThen enable the Podman Socket.
$ systemctl enable --user podman.socket
$ systemctl start --user podman.socket
Copy the codeSpecify the DOCKER_HOST environment variable.
$ export DOCKER_HOST="unix://$(podman info -f "{{.Host.RemoteSocket.Path}}")"
Copy the codeFinally, you can use Pack to build the image in the Podman container runtime. Detailed configuration steps can be found in the Buildpacks official documentation.
Build the OCI image with Pack
Once Pack is installed, we can learn more about Buildpacks through samples provided by CNB. This is a Java example. There’s no need to install the JDK, run Maven, or any other build environment during the build process. Buildpacks takes care of that for us.
First clone the sample repository:
$ git clone https://github.com/buildpacks/samples.git
Copy the codeWe will use the Bionic Builder to build the image. First let’s look at the configuration of the Builder:
$ cat samples/builders/bionic/builder.toml
# Buildpacks to include in builder
[[buildpacks]]
id = "samples/java-maven"
version = "0.0.1"
uri = ".. /.. /buildpacks/java-maven"
[[buildpacks]]
id = "samples/kotlin-gradle"
version = "0.0.1"
uri = ".. /.. /buildpacks/kotlin-gradle"
[[buildpacks]]
id = "samples/ruby-bundler"
version = "0.0.1"
uri = ".. /.. /buildpacks/ruby-bundler"
[[buildpacks]]
uri = "docker://cnbs/sample-package:hello-universe"
# Order used for detection
[[order]]
[[order.group]]
id = "samples/java-maven"
version = "0.0.1"
[[order]]
[[order.group]]
id = "samples/kotlin-gradle"
version = "0.0.1"
[[order]]
[[order.group]]
id = "samples/ruby-bundler"
version = "0.0.1"
[[order]]
[[order.group]]
id = "samples/hello-universe"
version = "0.0.1"
# Stack that will be used by the builder
[stack]
id = "io.buildpacks.samples.stacks.bionic"
run-image = "cnbs/sample-stack-run:bionic"
build-image = "cnbs/sample-stack-build:bionic"
Copy the codeThe builder is defined in the builder.toml file. The configuration structure can be divided into three parts:
- The [[BuildPacks] syntax identifier is used to define buildPacks included with the Builder.
- [[order]] is used to define the execution order of Buildpacks included with Builder.
- [[stack]] is used to define which base environment the Builder will run on.
We can use this Builder. toml to build our own Builder image:
$ cd samples/builders/bionic
$ pack builder create cnbs/sample-builder:bionic --config builder.toml
284055322776: Already exists
5b7c18d5e17c: Already exists
8a0af02bbad1: Already exists
0aa0fb9222a5: Download complete
3d56f4bc2c9a: Already exists
5b7c18d5e17c: Already exists
284055322776: Already exists
8a0af02bbad1: Already exists
a967314b5694: Already exists
a00d148009e5: Already exists
dbb2c49b44e3: Download complete
53a52c7f9926: Download complete
0cceee8a8cb0: Download complete
c238db6a02a5: Download complete
e925caa83f18: Download complete
Successfully created builder image cnbs/sample-builder:bionic
Tip: Run pack build <image-name> --builder cnbs/sample-builder:bionic to use this builder
Copy the codeNext, go to the Samples /apps directory and use the Pack tool and the Builder image to finish building the application. When the build is successful, an OCI image named sample-app is generated.
$ cd ../..
$ pack build --path apps/java-maven --builder cnbs/sample-builder:bionic sample-app
Copy the codeFinally, run the sample-app image with Docker:
$ docker run -it -p 8080:8080 sample-app
Copy the codeVisit **http://localhost:8080** and if all is well, you can see the following interface in your browser:
Now let’s look at the image we built earlier:
$Docker images REPOSITORY TAG IMAGE ID CREATED SIZE CNBS/Sample-Package Hell-universe e925CAa83f18 42 years ago 4.65kB sample-app latest 7867e21a60cd 42 years ago 300MB cnbs/sample-builder bionic 83509780fa67 42 years ago 181MB Buildpacksio/Lifecycle 0.13.1 76412e6be4e1 42 years ago 16.4MBCopy the codeThe image was created with a fixed timestamp: 42 years ago. Why is that? If the timestamp is not fixed, the hash value will be different each time the image is built. Once the hash value is different, it is not easy to determine whether the contents of the image are the same. With fixed timestamps, you can reuse layers created during the previous build process.
conclusion
Cloud Native Buildpacks represent a major step forward in modern software development, and the benefits over Dockerfile are immediate in most scenarios. While large organizations need to put in the effort to re-tune CI/CD processes or write custom Builders, significant time and maintenance costs can be saved in the long run.
This article introduced the origins of Cloud Native Buildpacks(CNB) and its advantages over other tools, explained how CNB works in detail, and finally gave a simple example to experience how to build images using CNB. Subsequent articles will show how to create custom Builder, Buildpack, Stack, and functional computing platforms (e.g., OpenFunction, Google Cloud Functions) that can take advantage of THE S2I capabilities provided by CNB, Implement the conversion process from the user’s function code to the final application.
This article is published by OpenWrite!