New features in Java SE 6

The Instrumentation features

Series contents:

This is part of the series:New features in Java SE 6

Introduction of Instrumentation

Using Java code, java.lang. Instrument to do dynamic Instrumentation is a new feature of Java SE 5, it frees Java instrument functions from native code. Make it possible to solve problems in Java code. With Instrumentation, developers can build an application-independent Agent to monitor and assist programs running on the JVM, and even replace and modify the definition of certain classes. With this capability, developers can implement more flexible runtime virtual machine monitoring and Java class manipulation, which in effect provides a virtual-machine-supported implementation of AOP that allows developers to implement some of THE functionality of AOP without making any upgrades or changes to the JDK.

In Java SE 6, the Instrumentation package provides more powerful capabilities: instrument after startup, native code instrument, dynamic classpath changes, and more. These changes mean that Java has more dynamic control and interpretation capabilities, which makes the Java language more flexible.

In Java SE6, the biggest changes make runtime Instrumentation possible. In Java SE 5, Instrument requires that the agent classes be set with command line arguments or system arguments before execution. In actual execution, instrumentation Settings are started at the time of virtual machine initialization (before most Java libraries have been loaded). A callback function is set up in the virtual machine to detect the loading of specific classes and complete the actual work. However, in many cases, it is not possible to create an agent for the vm when it is started, which limits the application of instrument. The new features of Java SE 6 change this situation, through the Attach method in the Java Tool API, we can easily set the dynamic loading proxy class in the process of running, in order to achieve the purpose of instrumentation.

In addition, Instrumentation for native interfaces is a new feature of Java SE 6, which enables functionality that could not be accomplished before — Instrumentation for native interfaces can be implemented in Java SE 6. This is accomplished through one or a series of prefix additions.

Finally, Instrumentation in Java SE 6 adds the ability to dynamically add class Paths. All of these new features enrich the instrument package, which in turn makes the Java language itself more powerful.

The basic functions and usage of Instrumentation

The implementation of the java.lang.instrument package depends on JVMTI. The Java Virtual Machine Tool Interface (JVMTI) is a set of local programming interfaces provided by Java Virtual machines for jVM-related tools. JVMTI was introduced with Java SE 5, Integration and replacement of the Java Virtual Machine Profiler Interface (JVMPI) and the Java Virtual Machine Debug Interface (JVMDI), In Java SE 6, JVMPI and JVMDI have disappeared. JVMTI provides a “proxy” program mechanism that enables third-party tools to connect to and access the JVM in a proxy manner, and to perform many JVMTI-related functions using the rich programming interfaces provided by JVMTI. In fact, the java.lang. Instrument package is implemented based on this mechanism: In the implementation of Instrumentation, there is a JVMTI agent, by calling JVMTI Java class related functions to complete the dynamic operation of Java classes. In addition to Instrumentation, JVMTI also provides a number of valuable functions for virtual machine memory management, thread control, methods and variable manipulation, and so on. For more information on JVMTI, refer to the Introduction in the Java SE 6 documentation (see Resources).

The best use of Instrumentation is to define dynamic changes and operations for classes. In Java SE 5 and later, developers can run a normal Java program (a Java class with the main function), Start the Instrumentation agent by specifying a specific JAR file that contains the Instrumentation agent with the -JavaAgent parameter.

In Java SE 5, developers can have Instrumentation agents run before main. Briefly speaking, it is the following steps:

1. Write the premain function

   Write a Java class that contains either of the following two methods

   public static void premain(String agentArgs, Instrumentation inst);  [1]
   public static void premain(String agentArgs); [2]Copy the code

   [1] has a higher priority than [2] and will be executed first (if [1] and [2] exist together, [2] will be ignored).

   In this premain function, the developer can perform various operations on the class.

   AgentArgs is a program parameter derived from the premain function, passed along with “-JavaAgent”. Unlike the main function, this argument is a string rather than an array of strings, and the program will parse the string itself if it has more than one argument.

   Inst is a Java. Lang. Instrument. The instance of Instrumentation and automatic passed by the JVM. Java. Lang. Instrument. Instrumentation is instrument package defined in an interface, also is the core part of the packet, focused almost all functions of methods, such as the class definition of transformation and operation and so on.

2. Jar file packaging

   Package the Java Class into a JAR file and add “premain-class” to its manifest attribute to specify the Java Class with Premain written in Step 1. (You may need to specify additional properties to enable more features.)

3. run

   Run a Java program with Instrumentation as follows:

   Java-javaagent: Location of jar file [= parameters passed to premain]Copy the code

An operation on a Java class file can be interpreted as an operation on a byte array (reading the binary stream of the class file into a byte array). The developer can retrieve the definition of a class (a byte array) in the Transform method of “ClassFileTransformer.” Apache’s BCEL open source project provides strong support for this, and readers can see an example of BCEL and Instrumentation combined in the reference article “Java SE 5 feature Instrumentation Practices.” The specific bytecode manipulation is not the focus of this article, so the examples in this article demonstrate the use of Instrumentation using a simple class file substitution.

Below, we through a simple example, to illustrate the basic use of Instrumentation.

First, we have a simple class, TransClass, that returns an integer 1 through a static method.

public class TransClass { public int getNumber() { return 1; }}Copy the code

We run the following class to get the output “1”.

public class TestMainInJar { public static void main(String[] args) { System.out.println(new TransClass().getNumber()); }}Copy the code

We then change the TransClass getNumber method to the following:

public int getNumber() { 
       return 2; 
}Copy the code

The Java file that returns 2 is then compiled into a class file. To distinguish the original class that returns 1, we name the class file that returns 2 TransClass2.

Next, we create a Transformer class:

import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.lang.instrument.ClassFileTransformer; import java.lang.instrument.IllegalClassFormatException; import java.security.ProtectionDomain; class Transformer implements ClassFileTransformer { public static final String classNumberReturns2 = "TransClass.class.2"; public static byte[] getBytesFromFile(String fileName) { try { // precondition File file = new File(fileName); InputStream is = new FileInputStream(file); long length = file.length(); byte[] bytes = new byte[(int) length]; // Read in the bytes int offset = 0; int numRead = 0; while (offset <bytes.length && (numRead = is.read(bytes, offset, bytes.length - offset)) >= 0) { offset += numRead; } if (offset < bytes.length) { throw new IOException("Could not completely read file " + file.getName()); } is.close(); return bytes; } catch (Exception e) { System.out.println("error occurs in _ClassTransformer!" + e.getClass().getName()); return null; } } public byte[] transform(ClassLoader l, String className, Class<? > c, ProtectionDomain pd, byte[] b) throws IllegalClassFormatException { if (! className.equals("TransClass")) { return null; } return getBytesFromFile(classNumberReturns2); }}Copy the code

This class implements the ClassFileTransformer interface. The getBytesFromFile method reads the binary character stream based on the file name, while the Transform method specified in ClassFileTransformer does the substitution conversion of the class definition.

Finally, we create a Premain class and write the Instrumentation proxy method Premain:

Public class Premain {public static void Premain (String agentArgs, Instrumentation inst) throws ClassNotFoundException,  UnmodifiableClassException { inst.addTransformer(new Transformer()); }}Copy the code

As you can see, the addTransformer method does not specify which class to convert. The transform occurs after premain and before main, and the transform method executes once for every class loaded to see if a transform is needed, so in the transform (Transformer class) method, The program uses classname.equals (“TransClass”) to determine whether the current class needs to be converted.

When the code is complete, we package them as testinstrument1.jar. The TransClass file that returns 1 remains in the JAR package, while the transclass.class. 2 that returns 2 is placed outside the JAR. Add the following attributes to the manifest to specify which class premain belongs to:

The Manifest - Version: 1.0 Premain - Class: PremainCopy the code

When running this program, if we run the main function in the JAR the normal way, we get the output “1”. If run as follows:

Java - javaagent: TestInstrument1 jar - cp TestInstrument1. Jar TestMainInJarCopy the code

You get the output “2”.

Of course, the main function that the program runs does not have to be in the same JAR file that Premain is in; it is put together for the convenience of the sample program package.

In addition to using addTransformer, another method in Instrumentation is “redefineClasses” to implement the transformations specified in Premain. The usage is similar as follows:

Public class Premain {public static void Premain (String agentArgs, Instrumentation inst) throws ClassNotFoundException, UnmodifiableClassException {ClassDefinition def = new ClassDefinition (TransClass. Class,  Transformer .getBytesFromFile(Transformer.classNumberReturns2)); inst.redefineClasses(new ClassDefinition[] { def }); System.out.println("success"); }}Copy the code

RedefineClasses is powerful enough to convert many classes in bulk.

New in Java SE 6: Dynamic Instrument after VMS are started

In Java SE 5, developers can only use their imagination in Premain, and Instrumentation can only be used before the main function is executed, which has some limitations.

Based on Java SE 5, Java SE 6 improves this situation. Developers can start their Instrumentation after the main function starts executing.

In Instrumentation for Java SE 6, there is a parallel “agentMain” method to premain, which can be run after main has started.

As with premain, developers can write a Java class with an “AgentMain” function:

public static void agentmain (String agentArgs, Instrumentation inst);          [1] 
public static void agentmain (String agentArgs);            [2]Copy the code

Also, [1] has a higher priority than [2] and will be executed first.

As with preMain, developers can perform various operations on classes in AgentMain. AgentArgs and Inst are used the same as premain.

Similar to “premain-class”, developers must set “agent-class” in the manifest file to specify classes that contain agentMain functions.

However, unlike Premain, agentMain needs to start after main has started. How can this be determined and implemented?

In the Java SE 6 documentation, developers may not find a clear introduction to the java.lang.instrument package in the documentation section, let alone specific examples of using AgnetMain. However, one of the less notable new features in Java SE 6 reveals the use of AgentMain. This is the Attach API provided in Java SE 6.

The Attach API is not a standard Java API, but an extended set of APIS provided by Sun for “attaching” the proxy application to the target JVM. With it, developers can easily monitor a JVM and run an additional agent.

The Attach API is simple, with only two main classes, both in the com.sun.tools. Attach package: VirtualMachine represents a Java VirtualMachine, the target VirtualMachine that the program needs to monitor, providing JVM enumerations, Attach action and Detach action (the reverse of Attach action, removing an agent from the JVM), and so on; VirtualMachineDescriptor is a container class that describes virtual machines, and implements various functions with the VirtualMachine class.

For the sake of simplicity, let’s simplify the examples as follows: Attach API is written in a thread. Use sleep wait to check all Java virtual machines every half second. When a new vm appears, Attach function is called. Then load the Jar file as described in the Attach API documentation. Wait until 5 seconds, attach automatically ends. In main, the program outputs the return value every half second (showing that the return value changes from 1 to 2).

The code for TransClass and Transformer classes remains the same, as described in the previous section. The code for TestMainInJar with main is:

public class TestMainInJar { public static void main(String[] args) throws InterruptedException { System.out.println(new  TransClass().getNumber()); int count = 0; while (true) { Thread.sleep(500); count++; int number = new TransClass().getNumber(); System.out.println(number); if (3 == number || count >= 10) { break; }}}}Copy the code

The code for the AgentMain class that contains AgentMain is:

import java.lang.instrument.ClassDefinition; import java.lang.instrument.Instrumentation; import java.lang.instrument.UnmodifiableClassException; public class AgentMain { public static void agentmain(String agentArgs, Instrumentation inst) throws ClassNotFoundException, UnmodifiableClassException, InterruptedException { inst.addTransformer(new Transformer (), true); inst.retransformClasses(TransClass.class); System.out.println("Agent Main Done"); }}Copy the code

RetransformClasses is a new method in Java SE 6 that, like redefineClasses, can transform class definitions in bulk and is mostly used in AgentMain.

The testinstrument1. Jar file is the same as the Jar file in the Premain example. The Jar file is the same as the Jar file in the Premain example. The Jar file is the same as the Jar file in the Premain example. The Manifest file in the Jar file is:

The Manifest - Version: 1.0 Agent - Class: AgentMainCopy the code

In addition, to run the Attach API, we can write another control program to simulate the monitoring process :(snippet)

import com.sun.tools.attach.VirtualMachine; import com.sun.tools.attach.VirtualMachineDescriptor; ... Static class AttachThread extends Thread {private Final List<VirtualMachineDescriptor> listBefore; private final String jar; AttachThread(String attachJar, List<VirtualMachineDescriptor> vms) { listBefore = vms; Jar = attachJar; } public void run() { VirtualMachine vm = null; List<VirtualMachineDescriptor> listAfter = null; try { int count = 0; while (true) { listAfter = VirtualMachine.list(); for (VirtualMachineDescriptor vmd : listAfter) { if (! Listbefore. contains(VMD)) {// If the VM is added, we think that the VM is started. break; } } Thread.sleep(500); count++; if (null ! = vm || count >= 10) { break; } } vm.loadAgent(jar); vm.detach(); } catch (Exception e) {ignore}}}...... public static void main(String[] args) throws InterruptedException { new AttachThread("TestInstrument1.jar", VirtualMachine.list()).start(); }Copy the code

At run time, you can start the test Jar file by running the main function above, which starts a new thread, and then, within 5 seconds (simply mimicking the JVM’s monitoring process), run the following command:

Java - javaagent: TestInstrument2 jar - cp TestInstrument2. Jar TestMainInJarCopy the code

If the timing isn’t too bad, the program will first type a 1 on the screen, which is the output of the class before the change, and then some 2’s, indicating that the AgentMain has been attached to the JVM successfully by the Attach API, and the agent is working. Of course, You can also see the output of Agent Main Done.

The above examples are just a simple example to illustrate this feature. Real-world examples tend to be complex and may run in multiple JVMS in a distributed environment.

New in Java SE 6: Instrumentation for native methods

As of version 1.5, there is no processing of Java Native methods, and there is no way to change the Method signature under the Java standard JVMTI, This makes it very difficult to replace local methods. A fairly straightforward and simple idea is to replace the dynamically linked library where the native code resides at startup — but this is essentially a static replacement, not a dynamic Instrumentation. Furthermore, this may require compiling a large number of dynamically linked libraries — for example, if we have three native functions, each of which requires a replacement, and each of which may require different combinations in different applications, if we compile all three functions in the same dynamically linked library, We need up to eight different dynamic link libraries to meet our needs. Of course, we could have compiled it independently, which would have required six dynamic-linked libraries — in any case, this cumbersome approach was unacceptable.

In Java SE 6, the new Native Instrumentation proposed a new Native code parsing method, as a supplement to the original Native method parsing method, to solve some problems well. This is why, in the new java.lang.instrument package, we now have the instrument method for native code – setting prefix.

Let’s say we have a native function named nativeMethod, and we need to point it to another function at runtime. (Note that under JVMTI, the signature must be consistent with the native function name.) For example, our Java code is:

package nativeTester; The class nativePrefixTester {... native int nativeMethod(int input); ... }Copy the code

So the native code we have implemented is:

jint Java_nativeTester_nativeMethod(jclass thiz, jobject thisObj, jint input);Copy the code

Now we need to point to another function when we call this function. In accordance with J2SE, we can use its naming method and add a prefix as the new function name. For example, if we use “another_” as prefix, then our new function is:

jint Java_nativeTester_another_nativePrefixTester(jclass thiz, jobject thisObj, 
jint input);Copy the code

It is then codified into a dynamically linked library.

Now that we have our new local functions, it’s time to set up the instrument. As mentioned above, we can use Premain to complete instrument agent setup by loading PreMain upon vm startup. You can also use agentMain to attach the virtual machine to start the agent. Setting up native functions is also fairly simple:

Premain (){// You can also use agentMain... if (! isNativeMethodPrefixSupported()){ return; } setNativeMethodPrefix(transformer,"another_");} setNativeMethodPrefix(transformer,"another_"); // Set the prefix for native functions. Note that this underline must be specified by the user... }Copy the code

There are two things to note here. Firstly, is it possible to set the prefix of native function in any case? First, notice the Manifest feature in the agent package:

Can-Set-Native-Method-PrefixCopy the code

Note that this parameter can affect whether or not native prefix can be set, and that, by default, this parameter is false, so we need to set it to true (BTW, the Manifest attributes are case insensitive, of course, If given a value other than “true”, it will be treated as a false value).

Of course, we also need to verify that the virtual machine itself supports setNativePrefix. In the Java API, the Instrumentation class provides a function, isNativePrefix, through which we can know whether the function is implemented.

Second, we can add its own NativePrefix to each ClassTransformer. Also, each ClassTransformer can transform for the same class, so a native function might have a different prefix for a class, so it might have several ways to parse for that function.

In Java SE 6, Native prefix is explained as follows: For a native method in a class of a package, first of all, suppose we set the native prefix “another” to the transformer of this function, which interpreates the interface of this function as:

Function interface by Java

native void method()Copy the code

And prefix”another” above to find functions in native code

void Java_package_class_another_method(jclass theClass, jobject thiz); // Notice where prefix appears in the function name!Copy the code

Once found, the function is called and the parsing process ends; If not, the virtual machine will do further parsing. We will use the most basic parsing method of the Java Native interface to find functions in native code:

void Java_package_class_method(jclass theClass, jobject thiz);Copy the code

If found, it is executed. Otherwise, the process fails because there is no proper way to parse.

What if there are multiple Transformers, each with its own prefix? In fact, virtual machines are parsed in the order transformer is added to the Instrumentation (remember our basic addTransformer method?). .

Suppose we have three transformers to be added in their order and the corresponding prefix: Transformer1 and Prefix1_, Transformer2 and Prefix2_, and Transformer3 and Prefix3_. So, the first thing the virtual machine does is resolve the interface to:

native void prefix1_prefix2_prefix3_native_method()Copy the code

Then find the corresponding native code.

However, if the second transformer (transformer2) does not have a prefix set, then it is very simple and we get the following resolution:

native void prefix1_prefix3_native_method()Copy the code

This approach is simple and natural.

Of course, in the case of multiple prefixes, there are some complications to be aware of. For example, suppose we have a native function interface that is:

native void native_method()Copy the code

We then set two prefixes to it, such as “wrapped_” and “wrapped2_”, so what do we get? is

void Java_package_class_wrapped_wrapped2_method(jclass theClass, jobject thiz); // Is the function name correct?Copy the code

? The answer is no, because in fact, there are a number of rules for the interface to native mapping of Java native functions, so there may be some special characters that have to be substituted. In practice, the correct name for this function is:

void Java_package_class_wrapped_1wrapped2_1method(jclass theClass, jobject thiz); // Only the function name will be foundCopy the code

Isn’t it interesting? So if we want to do something similar, a good suggestion is to first write a native interface with prefix in Java, use the Javah tool to generate a C header-file, and see what it actually resolves to the function name. In this way we can avoid some unnecessary trouble.

Another fact is that the virtual machine does not do much parsing for two or more prefixes, as might be expected; It does not attempt to remove a prefix and assemble the function interface. It does and only does two parses.

In a word, the new native prefix-instrumentation approach changes the shortcoming of the previous Java native code that cannot be dynamically changed. At present, using JNI to write native code is also a very important link in Java application, so its dynamic means that the whole Java can be dynamically changed — now our code can use prefix to dynamically change the pointing of native function, As mentioned above, the vm will try to do standard parsing if it can’t find it. This gives us a way to dynamically replace native code. We can compile many functions with different prefixes into a dynamic link library. Make native functions dynamically changed and replaced like Java functions.

Of course, there are still some restrictions on the current native instrumentation. For example, different transformers will have their own native prefix, that is to say, Each Transformer is responsible for all classes it replaces, not the prefix of a particular class — so this granularity may not be precise enough.

New in Java SE 6: Dynamic add-on to BootClassPath/SystemClassPath

Boot Class (-xbootCLASspath) and System Class (-cp) can be set to bootclass (-xbootclasspath) and system class (-cp). Of course, we cannot replace it after we run it. However, there may be times when we need to load some JARS into the bootCLASspath and we cannot apply the above two methods; Or we need to load some JARS into bootClasspath after the virtual machine starts. In Java SE 6, we can do just that.

This is easy to do, first of all, we still need to confirm that the virtual machine already supports this functionality, and then add the required classpath to preMain/AgantMain. We can use in our Transformer appendToBootstrapClassLoaderSearch/appendToSystemClassLoaderSearch to complete the task.

You can also add your own Boot Class Path to the agent manifest by adding boot-class-path. In Java Code it can be done more dynamically and intelligently — we can easily add judgment and selection components.

There are a few things to note here. First of all, we should add jar files to the classpath with no system namesake classes related to the instrumentation of the system, otherwise everything will fall into unpredictability – not what an engineer wants, right?

Second, notice how the virtual machine’s ClassLoader works, which logs the parsing results. For example, if we asked to read some someclass and failed, the ClassLoader will remember that. Even if we later dynamically add a JAR containing the class, the ClassLoader will still assume that we cannot parse the class, and the same error as the last one will be reported.

Again, we know that in the Java language there is a system parameter “java.class.path” that records our current classpath. However, using these two functions actually changes the actual classpath. It doesn’t have any effect on the property itself.

One of the interesting things we found in the public JavaDoc was that the Sun designers told us, The function in fact depends on this appendtoClassPathForInstrumentation method – this is a private function, so we do not recommend direct use it (using reflection, etc.), in fact, The two functions in the instrument package already solve our problem very well.

conclusion

From the above introduction, we can conclude that in Java SE 6, instrumentation package new features – dynamic instrument after virtual machine startup, native code instrumentation, Adding classpath dynamically, and more, gives Java more dynamic control and interpretation, making the Language more flexible.

These capabilities, in a sense, begin to change the Java language itself. In the past long period of time, the emergence and rapid development of dynamic scripting languages has played a very important role in improving the productivity of the entire software and network industry. Against this background, Java is also slowly changing. The new capabilities of Instrument and the emergence of the Script platform, which will be covered in a later article in this series, have greatly enhanced the dynamics of languages and their integration with dynamic languages, which is a significant new trend in the evolution of Java.

On the topic

• Read the full list of other important Java SE 6 enhancements in the Java SE 6 new feature series.
• Java SE 6 documentation: The Java SE 6 specification documentation provides official descriptions of most of the new features.
• Apache BCEL: Apache BCEL project, can help developers to operate class files, the development of powerful instrumentation agents.
• Read the article “Java 5 features Instrumentation practice” : my colleague wrote an article, introduced in the Java SE 5 environment, using BCEL to complete a timing program.
• DeveloperWorks Java Technology zone: There are hundreds of articles on all aspects of Java programming.