Docker automatically builds the image by reading instructions in a Dockerfile, a text file that in turn contains all the commands needed to build a given image.

The use of Dockerfile is very important because it is our blueprint, the record of the layers we add to the Docker image.

In this article, we’ll learn how to take advantage of BuildKit, a set of enhancements introduced in Docker V18.09. Integration with BuildKit will give us better performance, storage management, and security.

In this paper, the target

  • Reduce build time;

  • Reduce the mirror size;

  • Maintainability;

  • Obtain repeatability;

  • Understand multistage Dockerfile;

  • Learn about BuildKit functionality.

A prerequisite for

  • Docker concept knowledge

  • Docker installed (currently using V19.03)

  • A Java application (in this article, I used a Jenkins Maven sample application)

Let’s get started!

Simple Dockerfile example

The following is an example of an unoptimized Dockerfile containing a Java application. We’re going to do some optimization over time.

FROM debian COPY. /app RUN apt-get update RUN apt-get -y install openjdk-11 SSH emacs CMD [" Java ", "-jar", "/ app/target/my - app - 1.0 - the SNAPSHOT. Jar"]

At this point, we might ask ourselves: How long will build take? To answer this question, let’s create the Dockerfile on a local development environment and have Docker build the image.

# enter your Java app folder
cd simple-java-maven-app-master
# create a Dockerfile
vim Dockerfile
# write content, save and exit
docker pull debian:latest # pull the source image
time docker build --no-cache -t docker-class . # overwrite previous layers
# notice the build time
0,21s user 0,23s system 0% cpu 1:55,17 total
Copy the code

At this point, our build needs 1M55s.

Would it make a difference if we just enabled BuildKit and no other changes were made?

Enable BuildKit

BuildKit can be enabled in two ways:

Set the DOCKER_BUILDKIT = 1 environment variable when the Docker build command is called, for example:

time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class

To set Docker BuildKit to default, do the following in /etc/docker/daemon.json and restart:

{ "features": { "buildkit": true } }

BuildKit initial effect

DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class. 0,54s user 0,93s system 1% CPU 1:43,00 totalCopy the code

At this point, our build needs 1m43s. On the same hardware, the build took about 12 seconds less than before. This means that the build is almost effortless and saves about 10% of the time.

Now let’s see if there are some additional steps we can take to further improve.

The order of change from smallest to most frequent

Because order is important for caching, we move the COPY command closer to the end of the Dockerfile.

FROM debian RUN apt-get update RUN apt-get -y install openjdk-11-jdk SSH emacs RUN COPY. /app CMD [" Java ", "-jar", "/ app/target/my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

Avoid “COPY.”

Select a more specific COPY parameter to avoid cache interrupts. Copy only what you need.

FROM debian RUN apt-get update RUN apt-get -y install openjdK-11-jdk SSH vim COPY target/ my-app-1.0-snapshot.jar /app CMD [" Java ", "- jar", "/ app/my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

Apt-get update is used with the install command

This prevents the use of stale package caches.

Search the backend architect of the public account to reply “clean architecture” and get a surprise gift package.

FROM debian RUN apt-get update && \ apt-get -y install openJDK-11-jdk SSH vim COPY target/ my-app-1.0-snapshot.jar /app CMD [" Java ", "- jar", "/ app/my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

Remove unnecessary dependencies

Do not install debugging and editing tools at first; you can install them later if you need them.

FROM debian RUN apt-get update && \ apt-get -y install --no-install-recommends \ openjdk-11-jdk COPY Target/my-app-1.0-snapshot.jar /app CMD [" Java ", "-jar", "/app/ my-app-1.0-snapshot.jar"]Copy the code

Delete the package manager cache

Your image does not need this cached data. Take the opportunity to free up some space.

FROM debian RUN apt-get update && \ apt-get -y install --no-install-recommends \ openjdk-11-jdk && \ rm -rf /var/lib/apt/lists/* COPY target/ my-app-1.0-snapshot.jar /app CMD [" Java ", "-jar", "/app/ my-app-1.0-snapshot.jar"]Copy the code

Use official mirrors whenever possible

There are many reasons to use official images, such as reducing image maintenance time and image size, and pre-configuring images for use by containers.

FROM openJDK COPY target/ my-app-1.0-snapshot.jar /app CMD [" Java ", "-jar", "/app/ my-app-1.0-snapshot.jar"]Copy the code

Use specific tags

Do not use the latest label.

FROM openJDK :8 COPY target/ my-app-1.0-snapshot.jar /app CMD [" Java ", "-jar", "/app/ my-app-1.0-snapshot.jar"]Copy the code

Find the smallest mirror image

The following is a list of OpenJDK images. Choose the lightest mirror that works best for you.

REPOSITORY TAG SIZE openjdk 8 634MB openjdk 8-jre 443MB openjdk 8-jre-slim 204MB openjdk 8-jre-alpine 83MBCopy the code

Build from the source in a consistent environment

If you don’t need the entire JDK, you can use Maven Docker images as a base to build on.

Maven :3.6-jdk-8-alpine WORKDIR /app COPY pop.xml. COPY src. / SRC RUN MVN -e -b package CMD [" Java ", "-jar", "/ app/my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

Get the dependencies in a separate step

You can cache – the Dockerfile command used to get dependencies. Caching this step will speed up the build.

Maven: 3.6-jdK-8-alpine WORKDIR /app COPY pam.xml. RUN mvn-e -b dependency:resolve COPY src. / SRC RUN mvn-e-b dependency:resolve COPY src. / SRC RUN mvn-e-b Package CMD [" Java ", "- jar", "/ app/my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

Multi-phase build: Remove build dependencies

Why use multi-phase builds?

  • Separate the build from the runtime environment

  • DRY way

  • Different details of development, test, and other environments

  • Linearize dependencies

  • Have platform-specific stages

    Maven: 3.6-jdK-8 -alpine AS builder WORKDIR /app COPY pam.xml. RUN MVN -e -b dependency:resolve COPY src. / SRC RUN mvn -e -B package

    FROM openJDK :8-jre-alpine COPY — FROM =builder /app/target/ my-app-1.0-snapshot. jar/CMD [” Java “, “-jar”, “/ my – app – 1.0 – the SNAPSHOT. Jar”]

If you build our application right now,

Time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class. 0,41s user 0,54s system 2% CPU 35,656 totalCopy the code

You’ll notice that our application build took about 35.66 seconds. This is a pleasant development.

Below, we describe the functionality of other scenarios.

Multi-stage builds: Different mirroring styles

The Dockerfile below shows the different stages of debian-based and Alpine based mirroring.

Maven: 3.6-JDK-8-Alpine AS Builder... FROM openJDK :8-jre-jessie AS release-Jessie COPY -- FROM =builder /app/target/ my-app-1.0-snapshot. jar/CMD "- the jar." "/ my-app-1.0-snapshot.jar"] FROM openJDK :8-jre-alpine AS release-alpine COPY -- FROM = Builder / app/target/my - app - 1.0 - the SNAPSHOT. Jar/CMD [" Java ", "- jar", "/ my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

To build a specific image, we can use the -target argument:

time docker build --no-cache --target release-jessie .

Different mirroring styles (DRY/global ARG)

ARG flavor=alpine FROM Maven: 3.6-jdK-8-alpine AS Builder... $flavor AS release COPY -- FROM =builder /app/target/ my-app-1.0-snapshot. jar/CMD [" Java ", "-jar", "/ my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

The ARG command specifies the image to build. In the example above, we specified Alpine as the default image, but we can also specify the image in the docker build command with the -build-arg flavor= parameter.

time docker build --no-cache --target release --build-arg flavor=jessie .

Concurrency is important when building Docker images because it takes full advantage of available CPU threads. In a linear Dockerfile, all phases are executed sequentially. With a multi-phase build, we can get the smaller dependency phases ready for use by the main phase.

BuildKit even brings another performance benefit. If this phase is not used in future builds, it will be skipped directly at the end rather than processed and discarded.

Here is a sample Dockerfile where the site’s assets are built in an Assets phase:

Maven: 3.6-JDK-8-Alpine AS Builder... FROM Tiborvass/Whalesay AS assets RUN Whalesay "Hello DockerCon! > out/assets. HTML FROM openJDK :8-jre-alpine AS release COPY -- FROM =builder /app/ my-app-1.0-snapshot.jar/COPY --from=assets /out /assets CMD [" Java ", "-jar", "/ my-app-1.0-snapshot.jar"]Copy the code

This is another Dockerfile where the C and C ++ libraries are compiled, respectively, and used later in the Builder.

Maven: 3.6-JDK-8-alpine AS Builder-base... maven: 3.6-JDk-8-alpine AS Builder-base... FROM GCC :8-alpine AS Builder -someClib... RUN the git clone... /configure --prefix=/out && make && make install FROM g++:8-alpine AS builder-some CPPlib... RUN the git clone... && cmake... FROM builder-base AS builder COPY --from=builder-someClib /out / COPY --from=builder-someCpplib /out /Copy the code

BuildKit application cache

BuildKit has special features for package manager caching. Here are some examples of cache folder locations:

Package manager path

apt /var/lib/apt/lists
go ~/.cache/go-build
go-modules $GOPATH/pkg/mod
npm ~/.npm
pip ~/.cache/pip
Copy the code

We can compare this Dockerfile to the Dockerfile introduced above in a consistent environment from source build. This older Dockerfile has no special cache handling. We can do this using -mount =type=cache.

Maven: 3.6-jdK-8-alpine AS builder WORKDIR /app RUN --mount=target=. --mount=type=cache,target /root/.m2 \ &&mvn Package-doutputdirectory =/ FROM openJDK :8-jre-alpine COPY -- FROM = Builder /app/target/ my-app-1.0-snapshot.jar/CMD [" Java ", "- jar", "/ my - app - 1.0 - the SNAPSHOT. Jar"]Copy the code

BuildKit’s security features

BuildKit has security features. In the following example, we use -mount =type=secret to hide some confidential files, such as ~/. Aws /credentials.

The FROM < baseimage > RUN... RUN --mount=type=secret,id=aws,target=/root/.aws/credentials,required \ ./fetch-assets-from-s3.sh RUN ./build-scripts.shCopy the code

To build this Dockerfile, use the -secret argument:

docker build --secret id=aws,src=~/.aws/credentials

/keys/private.pem/root.ssh /private.pem/root.ssh /private.pem/root.ssh /private.pem/root.ssh /private.pem/root.ssh /private.pem

FROM alpine
RUN apk add --no-cache openssh-client
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
ARG REPO_REF=19ba7bcd9976ef8a9bd086187df19ba7bcd997f2
RUN --mount=type=ssh,required git clone [email protected]:org/repo /work && cd /work && git checkout -b $REPO_REF
Copy the code

To build this Dockerfile, you need to load your SSH private key in ssh-agent.

eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa # this is the SSH key default location
docker build --ssh=default .
Copy the code

conclusion

In this article, we showed you how to use Docker BuildKit to optimize Dockerfiles and thus speed up image build times. These speed increases can help us improve efficiency and save computing power.