The author | | sparrow source alibaba cloud native public number

Arthas submitted this article in March 2021.

Arthas was initially used for two main purposes:

Arthas solves the problem of implementing performance analysis tools for test environments, performance test environments, and production environments.
Through the combination of JAD, MC and Re-define functions to achieve the ability of hot updating of some node codes in production environment.

Technology selection related

Because the company has not yet established a relatively unified production micro-service configuration and state management ability, the research and development operation and maintenance of their respective systems are relatively independent. Now the project uses Spring Cloud and Eureka framework structure, which matches the basic support capacity of SBA. At the same time, SBA can provide service awareness, log level configuration management, and many management plug-ins based on THE JVM and Spring container. Can meet the needs of basic use.

At the time of the study, Arthas was released in 3.4.5 as a whole and provided a WebConsole-based Tunner Server model that was already integrated with SBA, as we’ve seen in linked articles. Because the project itself had no historical baggage, version 2.0 of SBA was adopted during the actual integration process to provide more administrative functions and graphical interface capabilities. Other advantages:

The Arthas Web Console interface integrates SBA password login and web page permission management so that users can use arthas Web Console functions only after they log in to the SBA.
The JMX management of JOLokia-Core open target service process based on SBA client can reduce the requirement of front-end interface development by realizing the reuse of SBA related operation interface by JMX interface.

The overall structure

A few key points, using the Arthas Spring Boot plug-in built into the JVM, following the ICBC model to establish a complete client download and modify the script to achieve remote control. The built-in solution has a small amount of development work and only needs to integrate relevant open source components to achieve the relevant remote use mode and take into account security. Icbc’s plan is large and suitable for the city with a dedicated R&D team after the overall architecture planning. The built-in solution also includes start-stop operation through JMX (Spring Boot plug-in based on 3.4.5 cannot obtain the relevant handle, so it cannot be implemented for the time being), and does not start by default. After remote JMX is enabled, 8 related threads are added to the JVM, and about 30MB of vm memory is added, which is the same as SBA1.0 scheme referred to in this article. It is necessary to consider whether the JVM memory can be supported before online startup.

Implementation effect

One of the biggest benefits of SBA 2.0 is the ability to configure links to external web pages, as well as local permission management for Spring-Security if the pages are implemented in the current JVM process, and the associated integrated arthas functionality can only be used in a production environment after logging into the SBA.

Inline connection position

The use of JMX

Jump to arthas Web Console

Retrofit scheme

SpringBoot Admin integrates several steps implemented in Arthas practice.

1. Overall engineering structure

The overall project is modified from the example project of SBA open source project, and the specific project link using Custom-UI is as follows: [spring-boot-admin-sample-custom-ui]_, _ The part in red box is arthas Web Console static files. Implementation of SBA startup custom loading. Maven Resource configuration:

<resource>
                <directory>static</directory>
                <targetPath>${project.build.directory}/classes/META-INF/spring-boot-admin-server-ui/extensions/arthas
                </targetPath>
                <filtering>false</filtering>
            </resource>
Copy the code

The meta-info in the final jar can be loaded into the associated static resources after the SBA tomcat starts, and the final URL corresponds to the external URL of the arthas Console configuration for the custom implementation.

2. Configure external links

SBA 2.0 has been using vUE bucket, expansion integration is more convenient. The official document provides the configuration mode of Embedding connection: [Linking/Embedding External Pages].

Sba Example application. Yml

# tag::customization-external-views[] spring: boot: admin: ui: external-views: - label: "Arthas Console" url: http://21.129.49.153:8080/ order: 1900 # end: : customization - external - views []Copy the code

3. Corresponding to Spring MVC Controller implementation

Refer to the SBA integration part of the original implementation, which is mainly modified to achieve the following functions:

Refresh the loaded instance list of the Tunnel Server and display it in the AgentID box for selecting and clicking the link.
Refresh user-defined IP addresses (to resolve the inconsistency between two production IP addresses and o&M IP addresses in the production environment).

4. Arthas Spring Boot plugin modification and configuration

Reference the plugin modifications and client configuration application.yml in the SBA integration of the original implementation.

The main change to the original Spring Boot plugin is that it is automatically loaded via Spring’s @conditionalonmissingBean.

The modification is mainly achieved by modifying this part of the configuration file default does not start, and then used by remote start related agent threads.

5. JMX implementation based on Spring Actuator

Jolokia-core-.jar is imported by maven by default. If you don’t have jolokia-core-.jar, the SBA client will import it by default. The operating capability of HTTP-based JMX can be seamlessly combined with SBA console functions through the Actuator.

Management configurations are open in application.yml, and Spring Security authentication can be enabled on the client side according to its own environment. SBA also supports access of endpoints with password protection through service discovery.

Include: "*" include: "*" exclude: Env endpoint: health: # Display details to all users. Show-details: ALWAYS health: status: HTTP-mapping: # Customize the HTTP status code of the health check. FATAL: 503Copy the code

The JMX reference implementation EnvironmentChangeListener in the original implementation approach, based on the Spring JMX annotations to realize.

@Component @ManagedResource(objectName = "com.ArthasAgentManageMbean:name=ArthasMbean", Description = "ArthasMbeanImpl ") public class ArthasMbeanImpl {@autoWired private Map<String, String> arthasConfigMap; @Autowired private ArthasProperties arthasProperties; @Autowired private ApplicationContext applicationContext; Private ArthasAgent arthasAgentInit() {arthasConfigMap = StringUtils.removeDashKey(arthasConfigMap); Map<String, String> mapWithPrefix = new HashMap<String, String>(arthasConfigmap.size ()); for (Map.Entry<String, String> entry : arthasConfigMap.entrySet()) { mapWithPrefix.put("arthas." + entry.getKey(), entry.getValue()); } final ArthasAgent arthasAgent = new ArthasAgent(mapWithPrefix, arthasProperties.getHome(), arthasProperties.isSlientInit(), null); arthasAgent.init(); return arthasAgent; } @managedOperation (description = "Obtain ArthasTunnelServer address ") Public String getArthasTunnelServerUrl() {return arthasProperties.getTunnelServer(); } @managedOperation (description = "Arthas Tunnel Server address, To come into force after the attach ") @ ManagedOperationParameter (name = "tunnelServer", Description = "example: ws: / / 127.0.0.1:7777 / ws") public Boolean setArthasTunnelServerUrl (String tunnelServer) {if (tunnelServer == null || tunnelServer.trim().equals("") || tunnelServer.indexOf("ws://") < 0) { return false; } arthasProperties.setTunnelServer(tunnelServer); return true; } @ ManagedOperation (description = "obtain AgentID") public String getAgentId () {return arthasProperties. GetAgentId (); } @ ManagedOperation (description = "access application name") public String getAppName () {return arthasProperties. GetAppName (); } @managedOperation (description = "get ArthasConfigMap") public HashMap<String, String> getArthasConfigMap() { return (HashMap) arthasConfigMap; } @ ManagedOperation (description = "return is already loaded Arthas agent") public Boolean isArthasAttched () {DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory(); String bean = "arthasAgent"; if (defaultListableBeanFactory.containsBean(bean)) { return true; } return false; } @ ManagedOperation (description = "start Arthas agent") public Boolean startArthasAgent () {DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory(); String bean = "arthasAgent"; if (defaultListableBeanFactory.containsBean(bean)) { ((ArthasAgent) defaultListableBeanFactory.getBean(bean)).init(); return true; } defaultListableBeanFactory.registerSingleton(bean, arthasAgentInit()); return true; } @managedOperation (description = "Close Arthas Agent, ") public Boolean stopArthasAgent() {// TODO can't get classLoader for custom TMP folder, So we can not get to com. Taobao. Arthas. Core. The server ArthasBootstrap class and call the destroy method DefaultListableBeanFactory DefaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory(); String bean = "arthasAgent"; if (defaultListableBeanFactory.containsBean(bean)) { defaultListableBeanFactory.destroySingleton(bean); return true; } else { return false; }}}Copy the code

The actual use

After the management project was put into operation, it was used for troubleshooting and code hot repair in production environment for many times. Performance problems are mainly used for online verification and debugging of performance flow control components and configuration parameters related to gray release.

In the early stage of code hot loading, jad+ MC was used to carry out operations. Later, it was found that some codes in JAD were inconsistent with decompiled codes due to environment configuration and JVM problems. Later, Maven was used to package and deploy application source compressed package to solve the problem. It is more reliable to make changes directly using the source built with the same version of the jar applied. The overall solution provides effective performance analysis and thermal repair capabilities in tightly managed production environments.

legacy

Existing official com. Taobao. Arthas. Agent. Attach. ArthasAgent started arthas arthasClassLoader and bootstrapClass agent client to use All are temporary variables within the method, and the relevant handles cannot be obtained externally to realize the function of shutting down arthas Agent through bootstrapClass. The temporary solution is to shut down the arthas Agent in the target process using the stop command after starting through JMX and connecting to the Web Console.

The existing bytecode loading tool can implement the hot deployment of internal classes and private classes and is compatible with skywalk8.x javaAgent plug-in. However, the jacoco coverage acquisition plug-in and Arthas bytecode were not compatible in the test environment. In some environments, you need to disable the agent before you can use arthas.

Arthas 57 hands-on experiments at start.aliyun.com: Arthas Experimental preview

Arthas Award 9 will run from 1 April to 30 April click here for details.

SpringBoot Admin2.0 integrates the Arthas practice for Java diagnostics