This article extracts code snippets from the corresponding full sample source code project:

  • Gitee.com/xautlx/pack…
  • Github.com/xautlx/pack…

Relevant codes and configurations have actually been tested. If any problems are found in the verification process, please Issue feedback for timely correction. Thanks for your support!

overview

With the popularity of Spring Boot, you can experience the simplicity of building an output JAR file and then deploying and running your application with a java-jar command. It is common for individual applications to grow in size as the project grows, with individual JAR files often being two or three hundred MB. If the microservices architecture is introduced, at least ten or twenty microservices, all the module JARS add up to one or two GIGABytes of optical deployment files for the entire system.

Once a system is running online, no matter new requirements iteration or Bug repair, it is inevitable to do deployment update. Especially for some delivery projects, the first deployment or remote update, often need to transfer hundreds of MB or several GB of deployment files, which is really a headache.

Imagine that an urgent and serious Bug is found in the online system and reported to the supervisor, who is told to fix it immediately. The research and development colleagues rush to analyze and troubleshoot and submit the code in a minute, complete the construction and package, and deliver it to the operation and maintenance. After a while the leadership anxious angry to ask questions updated solved? Operation and maintenance can only awkwardly reply: not yet, the deployment package file is quite large and uploading is a bit slow…

I changed a few lines of code. Why upload hundreds of MB files to deploy updates? Isn’t there a way to optimize it?

Encounter such a situation, I suggest you look down, may be able to find the answer you want.

The contents of this paper include:

  • How w to separate a single Spring Boot JAR file of one or two hundred MB into a dependent component lib directory and a business JAR for deployment, optimizing the size of a single JAR file to one or two hundred KB.
  • How to combine twenty highly overlapping dependencies of microservices into a single lib directory and multiple business Jars of one or two hundred KB for deployment, optimizing the overall project deployment file size from one or two GB to two or three hundred MB.

This article does not include:

  • Apart from the separation of Spring Boot configuration files, it is generally simple to overwrite the configuration in the JAR file from the external YAML configuration file by specifying active profile or to configure the service mode such as Nacos.
  • Not including Maven best practices, included in the sample project for demonstration purposes such as putting some of the specific configuration declarations that should be placed in each Boot module directly into the top-level parent definition, please be careful to optimize the use of the actual situation.
  • The implementation method in this paper is mainly oriented towards the Java-JAR runtime mode.

Slimming monster upgrade process

Level 0: General Fat Jar build reference project directory: package-optimize-Level0

Main configuration:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just a demonstration of the convenience of configuration, and does the plugin configuration and run definition directly in the parent build section. But in a real project, you need to put these definitions only in the Spring Boot module project (optimized to use the pluginManagement form), <plugins> <plugins> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build>Copy the code

Configuration output:

cd package-optimize-level0
mvn clean install

ls -lh package-optimize-app1/target/package-optimize-app1.jar
-rw-r--r--  1 lixia  wheel    16M Feb 24 21:06 package-optimize-app1/target/package-optimize-app1.jar

java -jar package-optimize-app1/target/package-optimize-app1.jar
Copy the code

Key notes:

  • (The demo application currently relies on only a few spring-boot-starter-Web components, and all the build output is only about ten MB.) The actual situation is that the output jar of a single build is usually tens of MB to one or two hundred MB or more, depending on the number of components the project depends on.
  • If you have a dozen or so microservices to deploy, that means transferring one or two GIGABytes of files, which takes as long as you can imagine. Even single updates to individual microservices require a transfer of one or two hundred MEgabytes.

Level 1: Common dependent JAR separation builds

Refer to the project directory: package-optimize-level1

Solve the problem:

  • Reduce the file size of a single microservice JAR so that the deployment process uploads files in seconds.

Main configuration:

For details about key configurations, see the following notes:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just a demonstration of the convenience of configuration, and does the plugin configuration and run definition directly in the parent build section. In real projects, however, you need to put these definitions only in the Spring Boot module project (which can be optimized to use the pluginManagement form) to avoid interfering with other util, common, etc module projects. <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <outputDirectory>${project.build.directory}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>true</silent> </configuration> </execution> </executions> </plugin> <! <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- include references that do not exist </artifactId> null</artifactId> </include> </include> </includes> <layout>ZIP</layout> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>Copy the code

Configuration output:

cd package-optimize-level1
mvn clean install

ls -lh package-optimize-app1/target/package-optimize-app1.jar
-rw-r--r--  1 lixia  wheel   149K Feb 24 20:56 package-optimize-app1/target/package-optimize-app1.jar

java -jar -Djava.ext.dirs=lib package-optimize-app1/target/package-optimize-app1.jar
Copy the code

Effect:

  • The output jar of a single build is usually only one or two hundred KILobytes based on the number of project-dependent components, which can be transmitted in seconds.
  • This is one of the most common optimizations available on the web, and it’s worth going further: if you have a dozen microservices, each with a JAR and a lib directory file, you’ll probably need to transfer one or two GB files for your first deployment.

Level 2: Merge all module dependent jars into the same lib directory

Refer to the project directory: package-optimize- Level2

Solve a problem:

  • Merge all module dependent jars into the same lib directory. Generally, due to the high degree of overlap between module project dependent Jars, the total size of all service deployment files merged is about two or three hundred MB
  • However, if -djava.ext. dirs=lib is used to load all jars to each JVM, on the one hand, all jars are completely loaded for each JVM, and on the other hand, version conflicts may occur due to different versions of microservice components

Main configuration:

For details about key configurations, see the following notes:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just a demonstration of the convenience of configuration, and does the plugin configuration and run definition directly in the parent build section. In real projects, however, you need to put these definitions only in the Spring Boot module project (which can be optimized to use the pluginManagement form) to avoid interfering with other util, common, etc module projects. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <useUniqueVersions>false</useUniqueVersions> </manifest> </archive> </configuration> </plugin> <! <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <! -- Each sub-module defines the corresponding attribute value of each module according to the actual level. > <outputDirectory>${boot-jar-output}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>false</silent> </configuration> </execution> </executions> </plugin> <! <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- include references that do not exist </artifactId> null</artifactId> </include> </include> </includes> <layout>ZIP</layout> <! -- Maven-jar-plugin is used to export jar files of microservices to the same directory as lib. See the boot-jar-output attribute definition in each submodule for co-referencing the same lib directory. -- --> <outputDirectory>${boot-jar-output}</outputDirectory> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>Copy the code

All lib directory files and each microservice build JAR are aggregated into the Devops common directory.

The meta-info /MANIFEST file in the microservice JAR file generates the class-path attribute based on the list of modules dependent components, thus avoiding different versions of the JAR:

Class-Path: Lib/spring - the boot - starter - web - 2.4.3. Jar lib/spring - the boot - starte r - 2.4.3. Jar lib/spring - the boot - 2.4.3. Jar Jar lib/spring-boot-starter-logging-2.4.3.jar lib/spring-boot-starter-logging-2.4.3.jar lib/ logback-classic-1.2.3. jar Jar lib/slf4j-api-1.7.30.jar lib/ slf4j-2.13.3.jar lib/log4j-api-2.13.3.jar Jar lib/ Jakarta. Annotation - API -1.3.5.jar lib/spring-core-5.3.4.jar lib/ spr-ing -jcl-5.3.4.jar Lib/snakeyaml - 1.27. Jar lib/spring - the boot - starter - json - 2. 4.3 the jar lib/Jackson - databind - 2.11.4. Jar Annotations lib/jackson-core-2.11.4.jar lib/jackson-datatype-jdk8-2.11.4.jar l The ib/Jackson - datatype - jsr310-2.11.4. Jar lib/Jackson - module - parameter - the name s - 2.11.4. Jar Lib/spring - the boot - starter - tomcat - 2.4.3. Jar lib/tomcat embed - core - 9.0.43. Jar lib/Jakarta. El - 3.0.3. Jar Jar lib/spring-web-5.3.4.jar lib/spring-beans-5.3.4.jar lib/ spring-webmvc-5.3.4. jar Lib/spring aop -- 5.3.4. Jar lib/spring - the context - 5.3.4. Jar lib/spring - expression - 5.3.4. JarCopy the code

Configuration output:

CD package optimize- Level2 MVN clean Install ls - LH Devops/Total 912 drwxr-xr-x 34 Lixia Wheel 1.1K Feb 24 22:27 Lib -rw-r--r-- 1 lixia wheel 150K Feb 24 22:31 package-optimize-app1.jar -rw-r--r-- 1 lixia wheel 149K Feb 24 22:31 package-optimize-app2.jar -rw-r--r-- 1 lixia wheel 149K Feb 24 22:31 package-optimize-app3.jar java -jar devops/package-optimize-app1.jarCopy the code

Effect:

  • The startup process no longer requires the -djava.ext.dirs =lib parameter definition.
  • All microservice jars refer to the common directory where all projects merge dependent components, and the total size of the deployment files is typically in the range of two or three hundred MB.
  • By customizing the class-path in the meta-info /MANIFEST file of each microservice JAR file to clearly indicate the dependent version component classes, the conflicts of different microservices component versions can be solved.

Level 3: Support for unofficial tripartite dependent components introduced by system

Refer to the project directory: package-optimize- Level3

Solve a problem:

  • There are some unofficial third parties such as the SDK Jar. One way to do this is to submit it to a Maven local server for reference, which is the same as the normal dependency JAR handling. But in the absence of maven private servers, a common simplification is to place dependent jars directly in the project and define them in the POM as System Scope.
  • The Maven-jar-plugin component has no direct parameter to declare components that contain the specified scope. If the meta-info /MANIFEST is not specified, the components defined by these scopes will not be found at runtime.

Main configuration:

For details about key configurations, see the following notes:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just a demonstration of the convenience of configuration, and does the plugin configuration and run definition directly in the parent build section. In real projects, however, you need to put these definitions only in the Spring Boot module project (which can be optimized to use the pluginManagement form) to avoid interfering with other util, common, etc module projects. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <useUniqueVersions>false</useUniqueVersions> </manifest> <manifestEntries> <! The maven-jar-plugin component has no direct parameter to declare components that contain specified scope. Append the list of specified dependent components with an additional class-path value. You can specify the jar-manifestens-classpath value in the submodule as you wish. For example (note the preceding dot character and each space delimiter) : .lib /xxx-1.0.0.jar lib/yyy-2.0.0.jar <Class-Path>${jar-manifestEntries-classpath}</Class-Path> </manifestEntries> </archive> </configuration> </plugin> <! <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <! -- Each sub-module defines the corresponding attribute value of each module according to the actual level. > <outputDirectory>${boot-jar-output}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>false</silent> </configuration> </execution> </executions> </plugin> <! <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- include references that do not exist </artifactId> null</artifactId> </include> </include> </includes> <layout>ZIP</layout> <! -- Maven-jar-plugin is used to export jar files of microservices to the same directory as lib. See the boot-jar-output attribute definition in each submodule for co-referencing the same lib directory. -- --> <outputDirectory>${boot-jar-output}</outputDirectory> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>Copy the code

The main configurations of the sub-module are as follows:

<properties> <! <boot-jar-output>... <boot-jar-output>... /devops</boot-jar-output> <! The maven-jar-plugin component does not have a direct parameter to declare the component containing the specified scope. The maven-jar-plugin component uses an additional Class Path value to append the specified dependent component list. You can specify the jar-manifestens-classpath value as you wish. For example, (note the preceding dot character and space delimitors, and the artifactid-version. jar format following lib instead of the actual file name) : .lib /xxx-1.0.0.jar lib/yyy-2.0.0.jar --> <jar-manifestEntries-classpath>. Lib/hik - SDK - 1.0.0. Jar < / jar - manifestEntries - classpath > < / properties > < dependencies > <! Hik </groupId> <artifactId>hik- SDK </artifactId> The < version > 1.0.0 < / version > < scope > system < / scope > < systemPath > ${project. The basedir} / lib/hik - SDK - 1.0.0. Jar < systemPath > </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies>Copy the code

The meta-info /MANIFEST file in the microservice output JAR file generates the class-path property based on the list of module-dependent components, which is prefixed with the jar-manifestens-classpath property definition:

Class-Path: Jar lib/spring-boot-starter-web-2.4.3. Ja r lib/spring-boot-starter-2.4.3. Jar Jar lib/sprin g-boot-autoconfigure-2.4.3.jar lib/sprin boot-starter-logging-2.4.3.ja r Jar lib/ slf4j-api-1.7.30. jar lib/log4j-to-slf4j-2.13.3.jar Jar lib/ jul-to-slf4j-1.7.30.jar lib/ Jakarta. Annotation -api-1.3.5.jar lib/sprin g-core-5.3.4.jar Lib /spring-jcl-5.3.4.jar lib/snakeyaml-1.27.jar lib/ spring-boot-starter-json 2.4.3.jar lib/jackson-databind-2.11.4.jar Annotations lib/ja ckson-annotations-2.11.4.jar lib/jackson-core-2.11.4.jar lib/jackson-da tatype-jdk8-2.11.4.jar Lib/Jackson - datatype - jsr310-2.11.4. Jar lib/jacks on the module - parameter - names - 2.11.4. Jar lib/spring - the boot - starter - tomcat 2.4 Jar lib/tomcat-embed-core-9.0.43. Jar lib/ Jakarta. El-3.0.3. jar lib/to McAt-embed - websock-9.0.43 Jar lib/ spring-webvc-5.3.4. jar lib/ spring-webvc-5.3.4. jar lib/ spring-webvc-5.3.4. jar lib/spring-aop-5.3.4.jar lib/ spring-webvc-5.3.4. jar lib/spring-aop-5.3.4.jar lib/ spring-webvc-5.3.4. jar lib/spring-aop-5.3.4.jar lib/spring-aop-5.3.4.jar lib Pring - context - 5.3.4. Jar lib/spring - expression - 5.3.4. JarCopy the code

Configuration output:

CD Package optimize- Level3 MVN Clean Install ls - LH Devops/Total 912 DRwxr-xr-x 36 Lixia Wheel 1.1K Feb 24 23:14 Lib -rw-r--r--@ 1 lixia wheel 150K Feb 24 23:14 package-optimize-app1.jar -rw-r--r-- 1 lixia wheel 150K Feb 24 23:14 package-optimize-app2.jar -rw-r--r-- 1 lixia wheel 150K Feb 24 23:14 package-optimize-app3.jar java -jar devops/package-optimize-app1.jarCopy the code

Final effect

  • All the dependent components of the services are consolidated into a directory with a total size of two or three hundred MB, which significantly speeds up the first deployment.
  • The size of each micro-service JAR is one or two hundred KB, and the daily emergency Bug repair and update of individual JAR is basically instantaneous transmission.
  • The respective definitions in each microservice JAR depend on the component list of the specified version, and there will be no loading conflict between different versions of components.
  • Unofficial tripartite dependent components can also be referenced properly.

Special note

After the above separation of deployment components, daily updates only need to transfer one or two hundred KB of business JAR files. However, if a project’s Maven-dependent components are configured with changes, care needs to be taken to synchronize the jar files to the common lib directory.

Tips for minimizing changes to JAR files: You can commit the build deployment resource directory to the GIT repository and commit it to the GIT repository at the same time of each subsequent release. Through the commit view, you can clearly identify the list of change files in the lib directory and the business JAR for this release, including the microservice JAR and the dependent JAR change files, so as to minimize the transfer of files.