The premise
I have an idea: long before the reference of existing mainstream ORM framework design, build an ORM wheels, on the premise of basic don’t change the experience the reflection design rely on the framework of a large number of are removed, the reflection API to build components used “dynamic compilation” loaded instance to replace, thus can get close to the performance of the direct use native JDBC. So with this in mind, in-depth study of Java dynamic compilation. JDK11 was used for this article.
The basic principle of
The following diagram, which looks familiar from the section on front-end compilation and optimization in Understanding the Java Virtual Machine, describes the compilation process:
The image above looks like there are only three steps, but each step is actually a large number of steps. The image below attempts to describe the specific steps in relative detail.
In fact, it is not necessary for developers or users to fully understand the details of the compilation process alone. The JDK provides a toolkit called Javax.tools that allows users to compile using a simple API (in most cases, developers develop for business functions, Details like compilation and packaging are handled directly by development tools, Maven, Gradle, etc.) :
Specific use process includes:
- To get a
javax.tools.JavaCompiler
Instance. - Based on the
Java
The file object initializes a compilation taskjavax.tools.JavaCompiler$CompilationTask
Instance. CompilationTask
The result of instance execution represents the success of the compilation process.
❝
We know the javac compiler is actually JavaCompiler interface implementation, in JDK1.6 +, the corresponding implementation classes for the com. Sun. View javac. API. JavacTool.
❞
Because classes in the JVM are isolated based on ClassLoader, a custom Class loader can be used to load the corresponding Class instance after successful compilation, and then the reflection API can be applied for instantiation and subsequent calls.
JDK dynamic compilation
The steps for dynamic compilation of the JDK were clearly explained in the previous section, so here’s a simple scenario. Suppose there is an interface as follows:
package club.throwable.compile;
public interface HelloService {
void sayHello(String name);
} // Default implementation package club.throwable.compile; public class DefaultHelloService implements HelloService { @Override public void sayHello(String name) { System.out.println(String.format("%s say hello [by default]", name)); } } Copy the code
We can define a class with the string SOURCE_CODE:
static String SOURCE_CODE = "package club.throwable.compile; \n" +
"\n" +
"public class JdkDynamicCompileHelloService implements HelloService{\n" +
"\n" +
" @Override\n" +
" public void sayHello(String name) {\n" + " System.out.println(String.format(\"%s say hello [by jdk dynamic compile]\", name)); \n" + " }\n" + "}"; // There is no need to define the class file package club.throwable.compile; public class JdkDynamicCompileHelloService implements HelloService{ @Override public void sayHello(String name) { System.out.println(String.format("%s say hello [by jdk dynamic compile]", name)); } } Copy the code
There are a few more things that need to be done before assembling the compile task instance:
- The built-in
JavaFileObject
Standards implementationSimpleJavaFileObject
Is the class source file, because the dynamic compilation time input is the content of the class source file string, need to achieveJavaFileObject
. - The built-in
JavaFileManager
It’s classpath orientedJava
Source file to load, here also need to implementJavaFileManager
. - You need to customize one
ClassLoader
Instance to load the compiled dynamic class.
Implement JavaFileObject
SimpleJavaFileObject (SimpleJavaFileObject);
public class CharSequenceJavaFileObject extends SimpleJavaFileObject {
public static final String CLASS_EXTENSION = ".class";
public static final String JAVA_EXTENSION = ".java";
private static URI fromClassName(String className) { try { return new URI(className); } catch (URISyntaxException e) { throw new IllegalArgumentException(className, e); } } private ByteArrayOutputStream byteCode; private final CharSequence sourceCode; public CharSequenceJavaFileObject(String className, CharSequence sourceCode) { super(fromClassName(className + JAVA_EXTENSION), Kind.SOURCE); this.sourceCode = sourceCode; } public CharSequenceJavaFileObject(String fullClassName, Kind kind) { super(fromClassName(fullClassName), kind); this.sourceCode = null; } public CharSequenceJavaFileObject(URI uri, Kind kind) { super(uri, kind); this.sourceCode = null; } @Override public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException { return sourceCode; } @Override public InputStream openInputStream(a) { return new ByteArrayInputStream(getByteCode()); } // Note that this method is the OutputStream of the compile-result callback, which will then fetch the compiled bytecode byte array of the target class via the getByteCode() method below @Override public OutputStream openOutputStream(a) { return byteCode = new ByteArrayOutputStream(); } public byte[] getByteCode() { return byteCode.toByteArray(); } } Copy the code
If after successful compilation, directly through to add CharSequenceJavaFileObject# getByteCode () method can obtain the target class compiled bytecode corresponding byte array (binary content). Here the CharSequenceJavaFileObject reserved multiple constructors for compatibility with the original way of compilation.
To achieve this
The key is to override the ClassLoader#findClass() method used to search for custom JavaFileObject instances and extract the corresponding bytecode byte arrays for loading. To do this, you can add a hash table as cache. The key-value is the alias of the full class name (in the form xx.yy.myclass rather than the URI schema) and the corresponding JavaFileObject instance of the target class.
public class JdkDynamicCompileClassLoader extends ClassLoader {
public static final String CLASS_EXTENSION = ".class";
private final Map<String, JavaFileObject> javaFileObjectMap = Maps.newConcurrentMap();
public JdkDynamicCompileClassLoader(ClassLoader parentClassLoader) { super(parentClassLoader); } @Override protectedClass<? > findClass(String name)throws ClassNotFoundException { JavaFileObject javaFileObject = javaFileObjectMap.get(name); if (null! = javaFileObject) { CharSequenceJavaFileObject charSequenceJavaFileObject = (CharSequenceJavaFileObject) javaFileObject; byte[] byteCode = charSequenceJavaFileObject.getByteCode(); return defineClass(name, byteCode, 0, byteCode.length); } return super.findClass(name); } @Nullable @Override public InputStream getResourceAsStream(String name) { if (name.endsWith(CLASS_EXTENSION)) { String qualifiedClassName = name.substring(0, name.length() - CLASS_EXTENSION.length()).replace('/'.'. '); CharSequenceJavaFileObject javaFileObject = (CharSequenceJavaFileObject) javaFileObjectMap.get(qualifiedClassName); if (null! = javaFileObject &&null! = javaFileObject.getByteCode()) { return new ByteArrayInputStream(javaFileObject.getByteCode()); } } return super.getResourceAsStream(name); } / / compile the temporary storage of the source file object, the key to all alias name of the class (not a URI pattern), such as club.throwable.com. Running the HelloService void addJavaFileObject(String qualifiedClassName, JavaFileObject javaFileObject) { javaFileObjectMap.put(qualifiedClassName, javaFileObject); } Collection<JavaFileObject> listJavaFileObject(a) { return Collections.unmodifiableCollection(javaFileObjectMap.values()); } } Copy the code
Implement JavaFileManager
A JavaFileManager is an abstract manager of Java files that is used to manage regular Java files, but not just files, but also Java class file data from other sources. Below by implementing a custom JavaFileManager used to manage the string type of source code. For the sake of simplicity, can be directly inherited the already existing ForwardingJavaFileManager:
public class JdkDynamicCompileJavaFileManager extends ForwardingJavaFileManager<JavaFileManager> {
private final JdkDynamicCompileClassLoader classLoader;
private final Map<URI, JavaFileObject> javaFileObjectMap = Maps.newConcurrentMap();
public JdkDynamicCompileJavaFileManager(JavaFileManager fileManager, JdkDynamicCompileClassLoader classLoader) { super(fileManager); this.classLoader = classLoader; } private static URI fromLocation(Location location, String packageName, String relativeName) { try { return new URI(location.getName() + '/' + packageName + '/' + relativeName); } catch (URISyntaxException e) { throw new IllegalArgumentException(e); } } @Override public FileObject getFileForInput(Location location, String packageName, String relativeName) throws IOException { JavaFileObject javaFileObject = javaFileObjectMap.get(fromLocation(location, packageName, relativeName)); if (null! = javaFileObject) { return javaFileObject; } return super.getFileForInput(location, packageName, relativeName); } / / this is a compiler to return to the same (source) of Java file object, replace with CharSequenceJavaFileObject implementation @Override public JavaFileObject getJavaFileForOutput(Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException { JavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, kind); classLoader.addJavaFileObject(className, javaFileObject); return javaFileObject; } // Override the original classloader @Override public ClassLoader getClassLoader(Location location) { return classLoader; } @Override public String inferBinaryName(Location location, JavaFileObject file) { if (file instanceof CharSequenceJavaFileObject) { return file.getName(); } return super.inferBinaryName(location, file); } @Override public Iterable<JavaFileObject> list(Location location, String packageName, Set<JavaFileObject.Kind> kinds, boolean recurse) throws IOException { Iterable<JavaFileObject> superResult = super.list(location, packageName, kinds, recurse); List<JavaFileObject> result = Lists.newArrayList(); // There is a distinction between compiled Location and compiled Kind if (location == StandardLocation.CLASS_PATH && kinds.contains(JavaFileObject.Kind.CLASS)) { //.class file and classPath for (JavaFileObject file : javaFileObjectMap.values()) { if (file.getKind() == JavaFileObject.Kind.CLASS && file.getName().startsWith(packageName)) { result.add(file); } } // All the Java file objects loaded by the class loader need to be added result.addAll(classLoader.listJavaFileObject()); } else if (location == StandardLocation.SOURCE_PATH && kinds.contains(JavaFileObject.Kind.SOURCE)) { //.java file and compile path for (JavaFileObject file : javaFileObjectMap.values()) { if (file.getKind() == JavaFileObject.Kind.SOURCE && file.getName().startsWith(packageName)) { result.add(file); } } } for (JavaFileObject javaFileObject : superResult) { result.add(javaFileObject); } return result; } // Custom method to add and cache source file objects to be compiled public void addJavaFileObject(Location location, String packageName, String relativeName, JavaFileObject javaFileObject) { javaFileObjectMap.put(fromLocation(location, packageName, relativeName), javaFileObject); } } Copy the code
Note in this class introduces the custom class loader JdkDynamicCompileClassLoader, the purpose is to achieve JavaFileObject instances sharing and provide file manager class loader instance.
Dynamically compile and run
We can compile the previously mentioned string using JavaCompiler. For better bytecode compatibility, we can specify a lower JDK version such as 1.6 at compile time:
public class Client {
static String SOURCE_CODE = "package club.throwable.compile; \n" +
"\n" +
"public class JdkDynamicCompileHelloService implements HelloService{\n" +
"\n" + " @Override\n" + " public void sayHello(String name) {\n" + " System.out.println(String.format(\"%s say hello [by jdk dynamic compile]\", name)); \n" + " }\n" + "}"; // Compile diagnostic collector static DiagnosticCollector<JavaFileObject> DIAGNOSTIC_COLLECTOR = new DiagnosticCollector<>(); public static void main(String[] args) throws Exception { // Get the system compiler instance JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); // Set build parameters - specify the build version as JDK1.6 to improve compatibility List<String> options = new ArrayList<>(); options.add("-source"); options.add("1.6"); options.add("-target"); options.add("1.6"); // Get a standard Java file manager instance StandardJavaFileManager manager = compiler.getStandardFileManager(DIAGNOSTIC_COLLECTOR, null.null); // Initializes the custom class loader JdkDynamicCompileClassLoader classLoader = new JdkDynamicCompileClassLoader(Thread.currentThread().getContextClassLoader()); // Initializes a custom Java file manager instance JdkDynamicCompileJavaFileManager fileManager = new JdkDynamicCompileJavaFileManager(manager, classLoader); String packageName = "club.throwable.compile"; String className = "JdkDynamicCompileHelloService"; String qualifiedName = packageName + "." + className; // Build the Java source file instance CharSequenceJavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, SOURCE_CODE); // Add a Java source file instance to a custom Java file manager instance fileManager.addJavaFileObject( StandardLocation.SOURCE_PATH, packageName, className + CharSequenceJavaFileObject.JAVA_EXTENSION, javaFileObject ); // Initializes a compilation task instance JavaCompiler.CompilationTask compilationTask = compiler.getTask( null. fileManager, DIAGNOSTIC_COLLECTOR, options, null. Lists.newArrayList(javaFileObject) ); // Perform a compilation task Boolean result = compilationTask.call(); System.out.println(String.format("Compile [%s] result :%s", qualifiedName, result)); Class<? > klass = classLoader.loadClass(qualifiedName); HelloService instance = (HelloService) klass.getDeclaredConstructor().newInstance(); instance.sayHello("throwable"); } } Copy the code
The following output is displayed:
Compile [club.throwable.com running JdkDynamicCompileHelloService] results: truethrowable say hello [by jdk dynamic compile]
Copy the code
Visible through the string class source code, the implementation of dynamic compilation, class loading, reflection instantiation and the final method call. In addition, diagnostic information for the build process is available through a DiagnosticCollector instance. For reuse, we can extract the JDK dynamic compilation process into a method:
public final class JdkCompiler {
static DiagnosticCollector<JavaFileObject> DIAGNOSTIC_COLLECTOR = new DiagnosticCollector<>();
@SuppressWarnings("unchecked")
public static <T> T compile(String packageName, String className, String sourceCode, Class<? >[] constructorParamTypes, Object[] constructorParams) throws Exception { // Get the system compiler instance JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); // Set the compile parameters List<String> options = new ArrayList<>(); options.add("-source"); options.add("1.6"); options.add("-target"); options.add("1.6"); // Get a standard Java file manager instance StandardJavaFileManager manager = compiler.getStandardFileManager(DIAGNOSTIC_COLLECTOR, null.null); // Initializes the custom class loader JdkDynamicCompileClassLoader classLoader = new JdkDynamicCompileClassLoader(Thread.currentThread().getContextClassLoader()); // Initializes a custom Java file manager instance JdkDynamicCompileJavaFileManager fileManager = new JdkDynamicCompileJavaFileManager(manager, classLoader); String qualifiedName = packageName + "." + className; // Build the Java source file instance CharSequenceJavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, sourceCode); // Add a Java source file instance to a custom Java file manager instance fileManager.addJavaFileObject( StandardLocation.SOURCE_PATH, packageName, className + CharSequenceJavaFileObject.JAVA_EXTENSION, javaFileObject ); // Initializes a compilation task instance JavaCompiler.CompilationTask compilationTask = compiler.getTask( null. fileManager, DIAGNOSTIC_COLLECTOR, options, null. Lists.newArrayList(javaFileObject) ); Boolean result = compilationTask.call(); System.out.println(String.format("Compile [%s] result :%s", qualifiedName, result)); Class<? > klass = classLoader.loadClass(qualifiedName); return (T) klass.getDeclaredConstructor(constructorParamTypes).newInstance(constructorParams); } } Copy the code
Javassist builds dynamically
Why do bytecode enhancement tools like Javassist exist when there is dynamic compilation of the JDK? Performance or efficiency aside, there are major limitations to dynamic compilation in the JDK. One obvious one is that bytecode piling cannot be done, in other words, it cannot be modified or enhanced based on existing classes and methods, but Javassist can. Furthermore, Javassist provides an API that is very similar to the JDK’s reflection API, making it easier to get started with Javassist if reflection is more familiar. Here is just one example of enhancing the DefaultHelloService mentioned earlier, starting with the introduction of dependencies:
<dependency>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
<version>3.27.0 - GA</version>
</dependency>
Copy the code
The encoding is as follows:
public class JavassistClient {
public static void main(String[] args) throws Exception {
ClassPool pool = ClassPool.getDefault();
CtClass cc = pool.get("club.throwable.compile.DefaultHelloService");
CtMethod ctMethod = cc.getDeclaredMethod("sayHello".new CtClass[]{pool.get("java.lang.String")}); ctMethod.insertBefore("System.out.println(\"insert before by Javassist\");"); ctMethod.insertAfter("System.out.println(\"insert after by Javassist\");"); Class<? > klass = cc.toClass(); System.out.println(klass.getName()); HelloService helloService = (HelloService) klass.getDeclaredConstructor().newInstance(); helloService.sayHello("throwable"); } } Copy the code
The following output is displayed:
club.throwable.compile.DefaultHelloService
insert before by Javassist
throwable say hello [by default]
insert after by Javassist
Copy the code
Javaassist is a combination of Java and Assist, which stands for Java assistant and is a Java bytecode enhanced class library:
- Bytecode enhancements can be made based on existing classes, such as modifying existing methods, variables, or even adding new methods directly to existing classes.
- You can dynamically create a whole new class, just like building blocks.
Unlike ASM, which has a steep learning curve and is a relatively low-level bytecode manipulation class library, and of course is far more efficient at bytecode enhancement in terms of performance than other higher-level encapsulated frameworks, Javaassist makes bytecode enhancement easy to get started with.
Advanced example
MySQL > SELECT Host,User FROM mysql.user; MySQL > SELECT Host,User FROM mysql.user;
@Data
public class MysqlUser {
private String host;
private String user;
} public interface MysqlInfoMapper { List<MysqlUser> selectAllMysqlUsers(a); } Copy the code
MySQL: MysqlInfoMapper MySQL: MysqlInfoMapper MySQL: MysqlInfoMapper MySQL: MysqlInfoMapper MysqlInfoMapper
- You need a connection manager to manage it
MySQL
The connection. - The need for a
SQL
Executors are used to execute queriesSQL
. - A result handler is required to extract and transform query results.
For the sake of simplicity, I define these three component interfaces through singletons in the interface (part of the configuration is completely written down) :
// Connection manager
public interface ConnectionManager {
String USER_NAME = "root";
String PASS_WORD = "root"; String URL = "jdbc:mysql://localhost:3306/mysql? useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT%2B8&useSSL=false"; Connection newConnection(a) throws SQLException; void closeConnection(Connection connection); ConnectionManager X = new ConnectionManager() { @Override public Connection newConnection(a) throws SQLException { return DriverManager.getConnection(URL, USER_NAME, PASS_WORD); } @Override public void closeConnection(Connection connection) { try { connection.close(); } catch (Exception ignore) { } } }; } / / actuator public interface SqlExecutor { ResultSet execute(Connection connection, String sql) throws SQLException; SqlExecutor X = new SqlExecutor() { @Override public ResultSet execute(Connection connection, String sql) throws SQLException { Statement statement = connection.createStatement(); statement.execute(sql); return statement.getResultSet(); } }; } // Result handler public interface ResultHandler<T> { T handleResultSet(ResultSet resultSet) throws SQLException; ResultHandler<List<MysqlUser>> X = new ResultHandler<List<MysqlUser>>() { @Override public List<MysqlUser> handleResultSet(ResultSet resultSet) throws SQLException { try { List<MysqlUser> result = Lists.newArrayList(); while (resultSet.next()) { MysqlUser item = new MysqlUser(); item.setHost(resultSet.getString("Host")); item.setUser(resultSet.getString("User")); result.add(item); } return result; } finally { resultSet.close(); } } }; } Copy the code
Then we need to dynamically compile the MysqlInfoMapper implementation class, whose source file contains the following string content (be careful not to create the DefaultMysqlInfoMapper class in the classpath) :
package club.throwable.compile;
import java.sql.Connection;
import java.sql.ResultSet;
import java.util.List;
public class DefaultMysqlInfoMapper implements MysqlInfoMapper { private final ConnectionManager connectionManager; private final SqlExecutor sqlExecutor; private final ResultHandler resultHandler; private final String sql; public DefaultMysqlInfoMapper(ConnectionManager connectionManager, SqlExecutor sqlExecutor, ResultHandler resultHandler, String sql) { this.connectionManager = connectionManager; this.sqlExecutor = sqlExecutor; this.resultHandler = resultHandler; this.sql = sql; } @Override public List<MysqlUser> selectAllMysqlUsers(a) { try { Connection connection = connectionManager.newConnection(); try { ResultSet resultSet = sqlExecutor.execute(connection, sql); return (List<MysqlUser>) resultHandler.handleResultSet(resultSet); } finally { connectionManager.closeConnection(connection); } } catch (Exception e) { // Ignore exception handling for now and encapsulate as IllegalStateException throw new IllegalStateException(e); } } } Copy the code
Then write a client to dynamically compile and execute:
public class MysqlInfoClient {
static String SOURCE_CODE = "package club.throwable.compile; \n" +
"import java.sql.Connection; \n" +
"import java.sql.ResultSet; \n" +
"import java.util.List; \n" + "\n" + "public class DefaultMysqlInfoMapper implements MysqlInfoMapper {\n" + "\n" + " private final ConnectionManager connectionManager; \n" + " private final SqlExecutor sqlExecutor; \n" + " private final ResultHandler resultHandler; \n" + " private final String sql; \n" + "\n" + " public DefaultMysqlInfoMapper(ConnectionManager connectionManager,\n" + " SqlExecutor sqlExecutor,\n" + " ResultHandler resultHandler,\n" + " String sql) {\n" + " this.connectionManager = connectionManager; \n" + " this.sqlExecutor = sqlExecutor; \n" + " this.resultHandler = resultHandler; \n" + " this.sql = sql; \n" + " }\n" + "\n" + " @Override\n" + " public List<MysqlUser> selectAllMysqlUsers() {\n" + " try {\n" + " Connection connection = connectionManager.newConnection(); \n" + " try {\n" + " ResultSet resultSet = sqlExecutor.execute(connection, sql); \n" + " return (List
) resultHandler.handleResultSet(resultSet); \n"
+ " } finally {\n" + " connectionManager.closeConnection(connection); \n" + " }\n" + " } catch (Exception e) {\n" + "// Ignore exception handling for now and wrap as IllegalStateException\n" + " throw new IllegalStateException(e); \n" + " }\n" + " }\n" + "}\n"; static String SQL = "SELECT Host,User FROM mysql.user"; public static void main(String[] args) throws Exception { MysqlInfoMapper mysqlInfoMapper = JdkCompiler.compile( "club.throwable.compile". "DefaultMysqlInfoMapper". SOURCE_CODE, new Class[]{ConnectionManager.class, SqlExecutor.class, ResultHandler.class, String.class}, new Object[]{ConnectionManager.X, SqlExecutor.X, ResultHandler.X, SQL}); System.out.println(JSON.toJSONString(mysqlInfoMapper.selectAllMysqlUsers())); } } Copy the code
The final output is:
Compile [club.throwable.com running DefaultMysqlInfoMapper] results: true[{
"host": "%",
"user": "canal"
}, {
"host": "%", "user": "doge" }, { "host": "localhost", "user": "mysql.infoschema" }, { "host": "localhost", "user": "mysql.session" }, { "host": "localhost", "user": "mysql.sys" }, { "host": "localhost", "user": "root" }] Copy the code
Then I check the results of the locally installed MySQL to verify that the query results are correct.
To simplify the whole example, I don’t add query parameters to MysqlInfoMapper#selectAllMysqlUsers(). SELECT Host,User FROM mysql. User WHERE User = ‘XXX’
❝
If the dynamically implemented DefaultMysqlInfoMapper is registered in the IOC container, MysqlInfoMapper can be automatically assembled by type. If SQL and parameter processing can be separated into separate files and a corresponding file parser is implemented, you can isolate class files from SQL, as both Mybatis and Hibernate do.
❞
summary
Dynamic compilation or more at the bottom of the bytecode aspect oriented programming, is a very challenging but can create infinite possible areas, this paper simply analyzes the process of the Java source code to compile, and dynamically compiled by some simple examples of simulation, used in practical application is still some distance, Later, I need to spend more time to analyze the knowledge of related fields.
References:
JDK11
Part of the source- Understanding The Java VIRTUAL Machine – 3rd
- Javassist
(C-4-D E-A-20200606 0:23 R-A-20200718)
This article is formatted using MDNICE