I took over a set of system with a sense of time, and planned to write a series of articles on the reconstruction and problems encountered, so that the old trees would grow new branches and review some actual combat techniques to share with everyone. [Reconstruction 01], tell you about Jar package conflict and principle.

background

The current market project management is either based on Maven or Gradle, and I recently took on a set of purely manual jar adding projects.

Adding JARS by hand has been done for years in a way that a technician working three or five years now might not have experienced. Find the jar packages needed in the project one by one, add them to a lib directory, and add them manually in the IDE.

Adding JAR dependencies in this way is not only cumbersome, but also prone to JAR conflicts, and analyzing the means of conflicts is a matter of experience.

I recently encountered A situation where A project could start normally in developer A’s environment but not in DEVELOPER B’s, and the exception message was that classes were not found.

Anyone with a bit of development experience can immediately conclude that jar package conflicts are the cause. See how to solve and extend the knowledge point that come out below.

Temporary solution

Due to the temporary inability to carry out large-scale reconstruction of the project, it is not easy to replace and upgrade the Jar package. It can only be solved by AD hoc means.

Here are a few steps to take in case you need them, and often tips for resolving Jar dependencies.

First: Look for classes in the IDE that are not found in the exception. For example, on the IDEA MAC operating system, THE shortcut keys I use are Command + Shift + N.

Take the Assert class as an example. You can see that there are many packages that contain Assert, but the launcher can’t find a method of that class. The problem is basically Jar package conflicts.

Second, after the Jar conflicts are located, find the Jar packages the system should have used.

For example, you need to use the classes in Spring-core instead of the classes in spring.jar. You can then use the JVM’s class-loading order mechanism to make the JVM load the Spring-Core JAR packages first.

Jar packages in the same directory are loaded by the JVM in the order in which the jar packages are loaded. Once a class with the same full path name is loaded, it is not loaded again.

Therefore, a temporary solution is to tweak the order in which the JVM compiles (loads) Jar packages. This is supported in Eclipse and Idea and can be adjusted manually.

Adjustments in Eclipse:

Adjustment mode in Idea:

The jar that needs to be loaded first is adjusted so that it can be loaded first.

An extension of class loading

The above is only a temporary solution limited by the current situation of the project. Eventually, it will have to be upgraded to manage Jar packages based on Maven or Gradle and solve the Jar conflicts at the same time.

In this interim solution, a key aspect of the JVM was addressed: the isolation of the JVM’s classloader and the parent delegate mechanism. Without knowledge of the JVM class loading mechanism, you might not even be able to come up with the above temporary solutions.

Class loader isolation issues

Each class loader has its own namespace to hold loaded classes. When a Class loader loads a Class, it detects that the Class has been loaded by searching for the Fully Qualified Class Name stored in the namespace.

The JVM’s only recognition of a class is the ClassLoader id + PackageName + ClassName, so it is possible to have two classes in a running program with exactly the same PackageName and ClassName. And if the two classes are not loaded by the same ClassLoader, there is no way to force an instance of one class into another class, which is ClassLoader isolation.

To address class loader isolation, the JVM introduced parental delegation.

Parent delegation mechanism

At the heart of the parent delegate mechanism are two things: first, a bottom-up check to see if the class is loaded; Second, you try to load classes from the top down.

There are generally four types of class loaders: startup class loaders, extension class loaders, application class loaders, and custom class loaders.

Regardless of the custom class loader, the JDK’s built-in class loader executes as follows:

When the AppClassLoader loads a class, it delegates the request to the parent class loader, ExtClassLoader.

When the ExtClassLoader loads a class, it delegates the request to the BootStrapClassLoader.

Third, if the BootStrapClassLoader fails to load (for example, the class cannot be found in %JAVA_HOME%/jre/lib), the ExtClassLoader will be used to try loading.

Fourth: if the ExtClassLoader also fails to load, AppClassLoader will be used to load. If the AppClassLoader also fails to load, ClassNotFoundException will be reported.

Parent delegate implementation of ClassLoader

ClassLoader implements the parent delegate mechanism through the loadClass() method for dynamic loading of classes.

The source code for this method is as follows:

protected Class<? > loadClass(String name, boolean resolve) throws ClassNotFoundException{ synchronized (getClassLoadingLock(name)) { // First, check if the class has already been loaded Class<? > c = findLoadedClass(name); if (c == null) { long t0 = System.nanoTime(); try { if (parent ! = null) { c = parent.loadClass(name, false); } else { c = findBootstrapClassOrNull(name); } } catch (ClassNotFoundException e) { // ClassNotFoundException thrown if class not found // from the non-null parent class loader } if (c == null) { // If still not found, then invoke findClass in order // to find the class. long t1 = System.nanoTime(); c = findClass(name); // this is the defining class loader; record the stats sun.misc.PerfCounter.getParentDelegationTime().addTime(t1 - t0); sun.misc.PerfCounter.getFindClassTime().addElapsedTimeFrom(t1); sun.misc.PerfCounter.getFindClasses().increment(); } } if (resolve) { resolveClass(c); } return c; }}Copy the code

The loadClass method itself is a recursive upward call, as you can see from the parent. LoadClass call in the code above.

Before doing anything else, the findLoadedClass method first checks to see if the specified class has been loaded, starting with the bottom-most classloader. If it is loaded, the resolve parameter determines whether to perform the join procedure and returns a Class object.

Jar conflicts usually occur here, when the first class with the same name is loaded, it will be returned directly during this step check, and the class that is really needed will not be loaded. A program that uses the class will throw an exception that the class can’t be found, or that its methods can’t be found.

The loading order of Jar packages

As you saw above, once a class is loaded, classes with the same globally qualified name may not be loaded. The order in which the Jar packages are loaded directly determines the order in which the classes are loaded.

The following factors usually determine the order in which Jar packages are loaded:

  • First, the loading path of the Jar package. That is, the level in the JVM classloader tree at which the Jar package is loaded. The paths of the Jar packages loaded by the four class loaders mentioned above have different priorities.
  • Second, the file loading order of the file system. The ClassLoader of Tomcat, Resin and other containers does not sort the file list under the loading path, which depends on the order returned by the underlying file system. When the file systems of different environments are inconsistent, there will be no problem in some environments and conflicts in some environments.

The problem I encountered was a branch of the second factor, where different Jar packages in the same directory were loaded in different order. Therefore, the problem is solved temporarily by tweaking the loading order of the Jar packages.

A common representation of Jar package conflicts

Jar conflicts are often weird and difficult to troubleshoot, but there are some common manifestations.

  • Throw Java. Lang. ClassNotFoundException: typical abnormalities, mainly rely on there is no class. There are two reasons for this: first, it is true that the class was not introduced; Second, Maven arbitration chose the wrong version due to Jar conflicts, causing the class to not be included in the loaded Jar.
  • Throw a Java. Lang. NoSuchMethodError: can’t find the specific method. Jar package conflict, resulting in the selection of the wrong dependency version, the class pair in the dependency version does not exist, or the method has been upgraded.
  • Throw a Java. Lang. NoClassDefFoundError, Java. Lang. LinkageError etc, the above reasons.
  • No exception but different expected results: The wrong version is loaded. The underlying implementation of different versions is different, resulting in inconsistent expected results.

The loading sequence of Jar packages and classes during Tomcat startup

Finally, take a look at the order in which Jar packages and classes are loaded when Tomcat starts, including the default directories for the different classes mentioned above:

  • Java core API in $java_HOME /lib;
  • $java_home/lib/ext Java extension JAR package;
  • Java-classpath / -djava.class. path class and JAR packages in the specified directory;
  • The $CATALINA_HOME/common directory is loaded from top to bottom.
  • The $CATALINA_HOME/server directory is loaded from top to bottom.
  • The $CATALINA_BASE/shared directory is loaded from top to bottom.
  • Class file in project path /WEB-INF/classes;
  • Jar file in project path /WEB-INF/lib;

Jar packages in the same folder in the preceding directory are loaded from top to next time. If a class file is already loaded into the JVM, subsequent identical class files will not be loaded.

summary

Jar conflicts are a very common problem in our daily development. A good understanding of the causes and underlying mechanisms of conflicts can greatly improve the ability to solve problems and the impact of the team. As a result, this question comes up in many interviews.

This article focuses on the causes and solutions of Jar package conflicts when manually adding dependencies. Maven’s strategies for Jar conflict management, such as dependency transfer, shortest path first, and first declaration, are often designed to address this issue, which we’ll discuss in more detail in the next article.

About the blogger: Author of the technology book SpringBoot Inside Technology, loves to delve into technology and writes technical articles.

Public account: “program new vision”, the blogger’s public account, welcome to follow ~

Technical exchange: Please contact the weibo user at Zhuan2quan