Java monitoring tool Arthas

[TOC]

Cactus:

Arthas is an open source Java diagnostic tool for Alibaba that developers love.

Arthas can help you when you are stuck with a problem like the following:

1. From which JAR is this class loaded? Why are all kinds of class-related exceptions reported?

2. Why didn’t the code I changed execute? Did I not commit? Got the branch wrong?

3. If you encounter a problem, you cannot debug it online. Can you only re-publish it by logging?

4. There is a problem with a user’s data processing online, but it cannot be debugged online, and it cannot be reproduced offline!

5. Is there a global view of the health of the system?

Is there any way to monitor the real-time health of the JVM?

Arthas supports JDK 6+, Linux/Mac/ Windows, command line interaction, and rich Tab auto-completion to further locate and diagnose problems.

Ii. Practice tutorial: Cake:

wget https://code.aliyun.com/middleware-container/handsonLabExternedFiles/raw/master/demo-arthas-spring-boot.jar; java -jar demo-arthas-spring-boot.jarCopy the code

1.1 start arthas – the boot

wget https://arthas.aliyun.com/arthas-boot.jar; Java -jar arthas-boot.jar --target-ip 0.0.0.0Copy the code

Arthas-boot is the arthas launcher. Once arthas is started, all Java processes are listed and the user can select the target process to diagnose.

Select the first process, type 1, then Enter/ Press Enter:

[INFO] arthas-boot version: 3.5.5 [INFO] Found Existing Java process, please choose one and input the serial number of the process, eg: 1. Then hit ENTER. * [1]: 1265 demo-arthas-spring-boot.jarCopy the code

1
Copy the code

After Attach is successful, Arthas LOGO will be printed. Type help to obtain more help information.

Factory:

Here are the commands for viewing JVM information in Arthas.

1.1 sysprop

Sysprop can print all System Properties information.

$ syspropKEY VALUE ------------------------------------------------------------------------------------------------------------------------ ----- awt.toolkit sun.awt.x11.xToolkit file.encoding. PKG sun.io java.specification. Vers 1.8 ion sun.cpu.isalist sun.jnu.encoding UTF-8 java.class.path demo-arthas-spring-boot.jar java.vm.vendor Private Build sun.arch.data.model 64 java.vendor.url http://java.oracle.com/ catalina.useNaming false user.timezone Asia/Shanghai org.jboss.logging.provi Slf4j der OS.name Linux java.vm.specification.v 1.8 ersion user.country US sun.java. Launcher SUN_STANDARD sun.boot.library.path /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64 sun.java.command demo-arthas-spring-boot.jar sun.cpu.endian little user.home /home/shell user.language en java.specification.vend Oracle Corporation or java.home /usr/lib/jvm/java-8-openjdk-amd64/jre file.separator / line.separator java.vm.specification.v Oracle Corporation endor java.specification.name Java Platform API Specification java.awt.graphicsenv sun.awt.X11GraphicsEnvironment java.awt.headless true sun.boot.class.path /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/ jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd 64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-open jdk-amd64/jre/classes  java.protocol.handler.p org.springframework.boot.loader kgs sun.management.compiler HotSpot 64-Bit Tiered Compilers Separator: java.runtime.version 1.8.0_292-8u292-b10-0Ubuntu1 ~18.04-b10 user.name shell path.separator: OS, version 3.10.0-957.21.3. El7. X86_64 Java. Endorsed inside. The dirs/usr/lib/JVM/Java - 8 - its - amd64 / jre/lib/endorsed inside java.runtime.name OpenJDK Runtime Environment file.encoding UTF-8 spring.beaninfo.ignore true sun.nio.ch.bugLevel java.vm.name OpenJDK 64-Bit Server VM java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.io.tmpdir /tmp Catalina. Home/TMP/tomcat 8558577891545713433.61000 Java version 1.8.0 comes with _292 user. Dir/home/shell OS. The arch amd64 PID 1265 java.vm.specification.n Java Virtual Machine Specification ame java.awt.printerjob sun.print.PSPrinterJob Sun. OS. Patch. The level of unknown catalina. The base/TMP/tomcat 8558577891545713433.61000 Java library. The path /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-l Inux - gnu: / usr/lib/jni: / lib: / usr/lib. Java vm. The info mixed mode Java. Vendor Private Build Java vm. Version 25.292 - b10 java.ext.dirs /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext sun.io.unicode.encoding UnicodeLittle Java. Class. Version 52.0Copy the code

You can also specify a single key: sysProp java.version

$ sysprop java.versionKEY VALUE ------------------------------------------------------------------------------------------------------------------------ . -- -- -- -- -- the Java version 1.8.0 comes with _292Copy the code

Can also through the grep to filter: sysprop | grep user

$ sysprop | grep user user.timezone Asia/Shanghai user.country US user.home /home/shell user.language en user.name shell  user.dir /home/shellCopy the code

A new value can be set: sysprop testKey testValue

$ sysprop testKey testValueSuccessfully changed the system property. KEY VALUE ------------------------------------------------------------------------------------------------------------------------ ----- testKey testValueCopy the code

1.2 sysenv

The sysenv command is used to obtain environment variables. This is similar to the sysprop command.

sysenv
 KEY                      VALUE                                                                                              
-----------------------------------------------------------------------------------------------------------------------------
 PATH                     /usr/local/nvm/versions/node/v12.13.1/bin:/home/shell/.local/bin:/home/shell/bin:/usr/shell/bin:/u 
                          sr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                                         
 USE_CLOUD_STORAGE        false                                                                                              
 ACCOUNT_ID               1030655505651579                                                                                   
 TZ                       Asia/Shanghai                                                                                      
 ALIBABA_CLOUD_VENDOR     CloudShell/0.2                                                                                     
 CLOUD_SHELL_ROOT_KEY_ID  LTAI4G8C8WqQihqKWH6L54Cr                                                                           
 ALIYUN_ACCESS_KEY_SECRE  J4TnqPfEhKWLpmuAk5ohhfqqNVHhPPP4YMwUmgXWCqfy                                                       
 T                                                                                                                           
 USER_ID                  1030655505651579                                                                                   
 ALIBABA_CLOUD_ACCESS_KE  J4TnqPfEhKWLpmuAk5ohhfqqNVHhPPP4YMwUmgXWCqfy                                                       
 Y_SECRET                                                                                                                    
 GOCACHE                  /go/.cache/go-build                                                                                
 DEFAULT_REGION           cn-hangzhou                                                                                        
 ALICLOUD_REGION          cn-hangzhou                                                                                        
 LOGNAME                  shell                                                                                              
 CLOUDSHELL_CONNECT_KEY   7zgfkrzvimgbwpun                                                                                   
 PWD                      /home/shell                                                                                        
 NVM_CD_FLAGS                                                                                                                
 CHANNEL_ID               OFFICIAL                                                                                           
 ALICLOUD_ACCOUNT_ID      1030655505651579                                                                                   
 ACCESS_KEY_ID            TMP.3KfX6ZiSosuHVWXdnmkZsJfhE7YtEQpzzhES4zuLz4BfQh1jAWHjaqE9gtygQycXhj9qyhSWVhgwdsh2hm7Rs4BaUTu4Ez 
 GOPATH                   /go                                                                                                
 OLDPWD                   /                                                                                                  
 ALIYUN_ACCESS_KEY_ID     TMP.3KfX6ZiSosuHVWXdnmkZsJfhE7YtEQpzzhES4zuLz4BfQh1jAWHjaqE9gtygQycXhj9qyhSWVhgwdsh2hm7Rs4BaUTu4Ez 
 GEM_HOME                 /home/shell/.gems                                                                                  
 REGION                   cn-hangzhou                                                                                        
 CLOUDSHELL_TENANT_CERT   -----BEGIN CERTIFICATE-----\nMIIDazCCAlOgAwIBAgIQWntjiFQ5tgjmatJi2Ol7iDANBgkqhkiG9w0BAQsFADBC\nMRY 
                          wFAYDVQQKDA1jbG91ZHNoZWxsIENBMRMwEQYDVQQLDApjbG91ZHNoZWxsMRMw\nEQYDVQQDDApjbG91ZHNoZWxsMB4XDTIxMTA 
                          yNjAwNDgwOVoXDTMxMTAyNjAwNDgw\nOVowQjEWMBQGA1UECgwNY2xvdWRzaGVsbCBDQTETMBEGA1UECwwKY2xvdWRzaGVs\nb 
                          DETMBEGA1UEAwwKY2xvdWRzaGVsbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\nAQoCggEBAKz3d8k3+Ak9+c81EDbRTI2Ls 
                          AWhXnrArh2E1c2dFAYQk5UyndT80wUv\nNt1uvqk23KOVE4rQOZ2KZ6mQczxlH62H0nX6gSMeZsCqGNtjRtb7j3PNDXGXoYTF\ 
                          n2WxXhYLgzszcUaJ+/iAiedQx0uIc9J5+sG3Do5LjOsFYGc8UYYBA+UpRJFq71oYs\nZXPJiNxLcOfF3k0JnKMqgNXmIoqEZFm 
                          OizpcVGMQZfycgqgid1yDqRVHJRwQHwHv\nclnUzxYTYVa5DsqYDr+OXmx++uHmQDtdli5eIA7rATqKt6BhFY/6VbdSklA8m5s 
                          Y\nt1xOZcE/tVSfNyAtsNpwirqAF2U6EeECAwEAAaNdMFswHwYDVR0jBBgwFoAUOC1+\nGQ6yPqW7TzKjy2IBHaqj190wHQYDV 
                          R0OBBYEFDgtfhkOsj6lu08yo8tiAR2qo9fd\nMAsGA1UdDwQEAwICBDAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBA 
                          QAb\nI0APX3asjhpxGkGxQrrBrtAzdyFuQNSBPP6uIdhK+BskYaK3VLh2RwRxiHdoUGO1\nzXK9VFvW2jb/1Xkcdlbm8ZkUefa 
                          P03D7E8+1HYuIx1uNo24TcOLZgOCeNvd3sFkz\nLvbluvF4PC44q9M067qlQOk5XvlkhJILYDXdfZLUKY1kC5o6u4f0sv0f6yO 
                          yRY55\nUE9WAhZdq1qNvaWD2bqR21x1Gc7fCIXv6rkHmETzcjdrEAaclShs6DmsYi3p148H\nXWfHNEqbIYzZJtzErvjqcgMOC 
                          xOHQ8mo6FmripT7w+9YhwbjOfIukIifPg3XdUj3\nUd3M38VSncE805iPPkGM\n-----END CERTIFICATE-----\n         
 ALIBABA_CLOUD_ACCOUNT_I  1030655505651579                                                                                   
 D                                                                                                                           
 ALIBABA_CLOUD_ACCESS_KE  TMP.3KfX6ZiSosuHVWXdnmkZsJfhE7YtEQpzzhES4zuLz4BfQh1jAWHjaqE9gtygQycXhj9qyhSWVhgwdsh2hm7Rs4BaUTu4Ez 
 Y_ID                                                                                                                        
 LS_COLORS                rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00: 
                          su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31 
                          :*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31: 
                          *.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01; 
                          31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31 
                          :*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*. 
                          alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm 
                          =01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.b 
                          mp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tif 
                          f=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpe 
                          g=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vo 
                          b=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=0 
                          1;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35 
                          :*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:* 
                          .midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.o 
                          pus=00;36:*.spx=00;36:*.xspf=00;36:                                                                
 CLOUD_SHELL_ROOT_KEY_SE  cf4x2LqNAyQXdSqzR5dLUkDrNQQ5VZ                                                                     
 CRET                                                                                                                        
 SHLVL                    3                                                                                                  
 CLOUDSHELL_PROXY_ACCESS  96924942-e575-484f-8fc5-a1cbee03a437                                                               
 _KEY                                                                                                                        
 default_serverless_devs  {"AccountID":"1030655505651579","AccessKeyID":"TMP.3KfX6ZiSosuHVWXdnmkZsJfhE7YtEQpzzhES4zuLz4BfQh1 
 _access                  jAWHjaqE9gtygQycXhj9qyhSWVhgwdsh2hm7Rs4BaUTu4Ez","AccessKeySecret":"J4TnqPfEhKWLpmuAk5ohhfqqNVHhPP 
                          P4YMwUmgXWCqfy"}                                                                                   
 ALIYUN_DEFAULT_REGION    cn-hangzhou                                                                                        
 TERM                     xterm-256color                                                                                     
 LANG                     en_US.UTF-8                                                                                        
 GOPROXY                  https://mirrors.aliyun.com/goproxy/                                                                
 ALICLOUD_SECRET_KEY      J4TnqPfEhKWLpmuAk5ohhfqqNVHhPPP4YMwUmgXWCqfy                                                       
 ACCESS_KEY_SECRET        J4TnqPfEhKWLpmuAk5ohhfqqNVHhPPP4YMwUmgXWCqfy                                                       
 CLOUD_SHELL_LANG         zh                                                                                                 
 _                        /usr/bin/java                                                                                      
 ALIBABA_CLOUD_DEFAULT_R  cn-hangzhou                                                                                        
 EGION                                                                                                                       
 CLOUDSHELL_DOWNLOAD_MAX  5368709120                                                                                         
 _SIZE                                                                                                                       
 NVM_DIR                  /usr/local/nvm                                                                                     
 CLOUD_SHELL              true                                                                                               
 FC_NO_ISOLATION          y                                                                                                  
 USER                     shell                                                                                              
 GEM_PATH                 /home/shell/.gems:/var/lib/gems/2.5.0                                                              
 ALICLOUD_ACCESS_KEY      TMP.3KfX6ZiSosuHVWXdnmkZsJfhE7YtEQpzzhES4zuLz4BfQh1jAWHjaqE9gtygQycXhj9qyhSWVhgwdsh2hm7Rs4BaUTu4Ez 
 ALIYUN_USER_AGENT        CloudShell/0.2                                                                                     
 HOSTNAME                 68ea4d9747b3                                                                                       
 MOUNT_FAILED             false                                                                                              
 NVM_BIN                  /usr/local/nvm/versions/node/v12.13.1/bin                                                          
 TF_PLUGIN_CACHE_DIR      /home/shell/.terraform.d/plugin-cache                                                              
 CLOUDSHELL_PROXY_ACCESS  c2b141e2-0c63-4959-a267-2faf34d2b8be                                                               
 _SECRET                                                                                                                     
 HOME                     /home/shell                            
Copy the code

1.3 the JVM

JVM commands print out various details about the JVM.

 $ jvmRUNTIME ------------------------------------------------------------------------------------------------------------------------ ----- machine-name 1265@68ea4d9747b3 jVM-start-time 2022-01-27 09:39:11 management-spec-version 1.2 spec-name Java Virtual Machine Specification spec-vendor Oracle Corporation spec-version 1.8 VM-name OpenJDK 64-bit Server VM Vm-vendor Private Build vm-version 25.292-b10 input-arguments [] class-path demo-arthas-spring-boot.jar boot-class-path /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd 64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm /java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.ja r:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-am d64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes LIBRARY-PATH /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/l ib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib ------------------------------------------------------------------------------------------------------------------------ ----- CLASS-LOADING ------------------------------------------------------------------------------------------------------------------------ ----- LOADED-CLASS-COUNT 9071 TOTAL-LOADED-CLASS-COUNT 9071 UNLOADED-CLASS-COUNT 0 IS-VERBOSE false ------------------------------------------------------------------------------------------------------------------------ ----- COMPILATION ------------------------------------------------------------------------------------------------------------------------ ----- NAME HotSpot 64-Bit Tiered Compilers TOTAL-COMPILE-TIME 13882 [time (ms)] ------------------------------------------------------------------------------------------------------------------------ ----- GARBAGE-COLLECTORS ------------------------------------------------------------------------------------------------------------------------ ----- Copy name : Copy [count/time (ms)] collectionCount : 97 collectionTime : 602 MarkSweepCompact name : MarkSweepCompact [count/time (ms)] collectionCount : 3 collectionTime : 208 ------------------------------------------------------------------------------------------------------------------------ ----- MEMORY-MANAGERS ------------------------------------------------------------------------------------------------------------------------ ----- CodeCacheManager Code Cache Metaspace Manager Metaspace Compressed Class Space Copy Eden Space Survivor Space MarkSweepCompact Eden Space Survivor Space Tenured Gen ------------------------------------------------------------------------------------------------------------------------ ----- MEMORY ------------------------------------------------------------------------------------------------------------------------ ----- heap-memory-usage init: 31457280(30.0 MiB) [MEMORY in bytes] Used: 42087040(40.1 MiB) Committed: 81309696(77.5 MiB) Max: 466288640(444.7 MiB) No-heap-memory-usage init: 2555904(2.4 MiB) [MEMORY in bytes] Used: 67432512(64.3 MiB) COMMITTED: 68878336(65.7 MiB) Max: -1(-1 B) PENDING-FINALIZE-COUNT 0 ------------------------------------------------------------------------------------------------------------------------ ----- OPERATING-SYSTEM ------------------------------------------------------------------------------------------------------------------------ ----- OS Linux ARCH amd64 PROCESSORS-COUNT 1 Load-Average 0.02 VERSION 3.10.0-957.21.3.el7.x86_64 ------------------------------------------------------------------------------------------------------------------------ ----- THREAD ------------------------------------------------------------------------------------------------------------------------ ----- COUNT 30 DAEMON-COUNT 28 PEAK-COUNT 30 STARTED-COUNT 38 DEADLOCK-COUNT 0 ------------------------------------------------------------------------------------------------------------------------ ----- FILE-DESCRIPTOR ------------------------------------------------------------------------------------------------------------------------ ----- MAX-FILE-DESCRIPTOR-COUNT 1048576 OPEN-FILE-DESCRIPTOR-COUNT 67Copy the code

1.4 dashboard

The dashboard command displays the real-time data panel of the current system.

To exit the Dashboard command, type Q or Ctrl+C.

Arthas use tips:

1.1 help

Arthas has detailed help information for each command. You can view it using -h. There are EXAMPLES and WIKI links in the help information.

sysprop -h
Copy the code

1.2 Automatic completion

Arthas supports rich auto-completion features that allow you to type Tab for more information when in doubt. For example, if you type sysprop Java. And then you type Tab, the corresponding key will be completed:Copy the code

$ sysprop java.
java.runtime.name             java.protocol.handler.pkgs    java.vm.version
java.vm.vendor                java.vendor.url               java.vm.name
...
Copy the code

1.3 Readline shortcut key support

Arthas supports common command-line shortcuts such as Ctrl + A to the beginning of A line and Ctrl + E to the end of A line.

More shortcuts can be viewed using the keymap command.

[arthas@1265]$ keymapShortcut Description Name ------------------------------------------------------------------------------------------------------------------------ ----- "\C-a" Ctrl + a beginning-of-line "\C-e" Ctrl + e end-of-line "\C-f" Ctrl + f forward-word "\C-b" Ctrl + b backward-word "\e[D" Left arrow backward-char "\e[C" Right arrow forward-char "\e[A" Up arrow history-search-backward "\e[B" Down arrow history-search-forward "\C-h" Ctrl + h backward-delete-char "\C-?" Ctrl + ? backward-delete-char "\C-u" Ctrl + u undo "\C-d" Ctrl + d delete-char "\C-k" Ctrl + k kill-line "\C-i" Ctrl + i complete "\C-j" Ctrl + j accept-line "\C-m" Ctrl + m accept-line "\C-w" Ctrl + w backward-delete-word "\C-x\e[3~" "\C-x\e[3~" backward-kill-line "\e\C-?" "\e\C-?" backward-kill-word "\e[1~" "\e[1~" beginning-of-line "\e[4~" "\e[4~" end-of-line "\e[5~" "\e[5~" beginning-of-history "\e[6~" "\e[6~" end-of-history "\e[3~" "\e[3~" delete-char "\e[2~" "\e[2~" quoted-insert "\e[7~" "\e[7~" beginning-of-line "\e[8~" "\e[8~" end-of-line "\eOH" "\eOH" beginning-of-line "\eOF" "\eOF" end-of-line "\e[H" "\e[H" beginning-of-line "\e[F" "\e[F" end-of-lineCopy the code

1.4 Completion of historical Commands

If you want to run the previous command again, press Up/↑ or Ddown/↓ to match the previous command.

For example, if you have run sysprop java.version before, after entering sysprop JA, you can enter Up/↑ to automatically complete sysprop java.version.

You can also use the history command to view all the history commands.

history
Copy the code

1.5 pipeline

Arthas supports executing simple commands after pipeline, such as:

sysprop | grep java
Copy the code

sysprop | wc -l
Copy the code

Boxing_glove:

1.1 sc

The sc command finds all classes that have been loaded by the JVM. If the search is for an interface, all implementation classes are also searched. For example, look at all the Filter implementation classes:

sc javax.servlet.Filter
Copy the code

Using the -d parameter, you can print detailed information about class loading, which helps you locate class loading problems.

sc -d javax.servlet.Filter
Copy the code

Sc supports wildcards, such as searching all StringUtils:

sc *StringUtils
Copy the code

1.2 sm

The sm command is a specific function to find a class. Such as:

sm java.math.RoundingMode
Copy the code

The -d argument prints the specific properties of the function:

sm -d java.math.RoundingMode
Copy the code

You can also find specific functions, such as finding constructors:

sm java.math.RoundingMode <init>
Copy the code

Six, Jad

Decompilating code can be done using jad:

[arthas@1268]$ jad com.example.demo.arthas.user.UserController ClassLoader: +-org.springframework.boot.loader.LaunchedURLClassLoader@19469ea2 +-sun.misc.Launcher$AppClassLoader@1b6d3586 +-sun.misc.Launcher$ExtClassLoader@7d0587f1 Location: file:/home/shell/demo-arthas-spring-boot.jar! /BOOT-INF/classes! //* * Decompiled with CFR. * * Could not load the following classes: * com.example.demo.arthas.user.User * org.slf4j.Logger * org.slf4j.LoggerFactory * org.springframework.web.bind.annotation.GetMapping * org.springframework.web.bind.annotation.PathVariable * org.springframework.web.bind.annotation.RestController */
       package com.example.demo.arthas.user;
       
       import com.example.demo.arthas.user.User;
       import org.slf4j.Logger;
       import org.slf4j.LoggerFactory;
       import org.springframework.web.bind.annotation.GetMapping;
       import org.springframework.web.bind.annotation.PathVariable;
       import org.springframework.web.bind.annotation.RestController;
       
       @RestController
       public class UserController {
           private static final Logger logger = LoggerFactory.getLogger(UserController.class);
       
           @GetMapping(value={"/user/{id}"})
           public User findUserById(@PathVariable Integer id) {
/ * * / 15         logger.info("id: {}", (Object)id);
/ * 17 * /         if(id ! =null && id < 1) {
                   throw new IllegalArgumentException("id < 1");
               }
               return new User(id.intValue(), "name" + id);
           }
       }

Affect(row-cnt:1) cost in 1262 ms.
Copy the code

With the –source-only argument you can print only the source code in the decompilation:

[arthas@1268]$ jad --source-only com.example.demo.arthas.user.UserController
       /* * Decompiled with CFR. * * Could not load the following classes: * com.example.demo.arthas.user.User * org.slf4j.Logger * org.slf4j.LoggerFactory * org.springframework.web.bind.annotation.GetMapping * org.springframework.web.bind.annotation.PathVariable * org.springframework.web.bind.annotation.RestController */
       package com.example.demo.arthas.user;
       
       import com.example.demo.arthas.user.User;
       import org.slf4j.Logger;
       import org.slf4j.LoggerFactory;
       import org.springframework.web.bind.annotation.GetMapping;
       import org.springframework.web.bind.annotation.PathVariable;
       import org.springframework.web.bind.annotation.RestController;
       
       @RestController
       public class UserController {
           private static final Logger logger = LoggerFactory.getLogger(UserController.class);
       
           @GetMapping(value={"/user/{id}"})
           public User findUserById(@PathVariable Integer id) {
/ * * / 15         logger.info("id: {}", (Object)id);
/ * 17 * /         if(id ! =null && id < 1) {
                   throw new IllegalArgumentException("id < 1");
               }
               return new User(id.intValue(), "name"+ id); }}Copy the code

Seven, Ognl: kick_scooter:

In Arthas, there is a separate OGNL command that executes code dynamically.

1.1 Calling the static function

[arthas@1268]$ ognl '@[email protected]("hello ognl")'
Copy the code

Hello ogNL is printed from the process output in Terminal 1.

1.2 Finding the ClassLoader of UserController

sc -d com.example.demo.arthas.user.UserController | grep classLoaderHash
Copy the code

Note that hashcode changes. You need to view the current ClassLoader information and extract the hashcode of the corresponding ClassLoader.

If you use -c, you need to manually type hashcode: -c

Example: Check for abnormal function calls :rabbit:

1.1 the phenomenon

At present, visit http://localhost:61000/user/0, abnormal returns 500:

curl http://localhost:61000/user/0
Copy the code

{"timestamp":1550223186170."status":500."error":"Internal Server Error"."exception":"java.lang.IllegalArgumentException"."message":"id < 1"."path":"/user/0"}
Copy the code

But what is the exception stack for the request parameters?

1.2 Viewing UserController Parameters/Exceptions

watch com.example.demo.arthas.user.UserController * '{params, throwExp}'
Copy the code

1. The first argument is the class name, which supports wildcards
2. The second argument is the function name, which supports wildcard accesscurl http://localhost:61000/user/0 ,watchThe command prints the arguments and exceptions to the call

curl http://localhost:61000/user/0
Copy the code

Abnormal results are displayed:

[arthas@1268]$ watch com.example.demo.arthas.user.UserController * '{params, throwExp}'
Press Q or Ctrl+C to abort.
Affect(class count: 1,method count: 2) cost in 227 ms.listenerId: 1.method=com.example.demo.arthas.user.UserController.findUserById location=AtExceptionExit
ts=2022-01-27 10:58:41; [cost=2.370685ms] result=@ArrayList[
    @Object[][isEmpty=false; size=1].@IllegalArgumentException[java.lang.IllegalArgumentException: id < 1]]Copy the code

You can see that the exception actually thrown is IllegalArgumentException.

You can exit the watch command by typing Q or Ctrl+C.

To expand the obtained result, use the -x argument:

[arthas@1268]$ watch com.example.demo.arthas.user.UserController * '{params, throwExp}' -x 2
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 2) cost in 59 ms, listenerId: 2
method=com.example.demo.arthas.user.UserController.findUserById location=AtExceptionExit
ts=2022-01-27 11:00:38; [cost=0.475683ms] result=@ArrayList[
    @Object[][
        @Integer[0],
    ],
    java.lang.IllegalArgumentException: id < 1
        at com.example.demo.arthas.user.UserController.findUserById(UserController.java:18)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
,
]
Copy the code

Return value expression

In the above example, the third argument is the return value expression, which is actually an OGNL expression that supports some built-in objects:

• loader
• clazz
• method
• target
• params
• returnObj
• throwExp
• isBefore
• isThrow
• isReturn

You can use these built-in objects to form different expressions. For example, return an array:

watch com.example.demo.arthas.user.UserController * '{params[0], target, returnObj}'
Copy the code

More reference: arthas.aliyun.com/doc/advice-…

Haircut_woman :haircut_woman:

1.1 watchThe command supports conditional expressions in the fourth argument, such as:

watch com.example.demo.arthas.user.UserController * returnObj 'params[0] > 100'
Copy the code

The watch command has no output when user/1 is accessed

When user/101 is accessed, Watch prints out the results.

$ watch com.example.demo.arthas.user.UserController * returnObj 'params[0] > 100'Press Q or Ctrl+C to abort. Affect(class-cnt:1 , method-cnt:2) cost in 47 ms. ts=2019-02-13 19:42:12; [cost=0.821443ms] result= @user [ID = @INTEGER [101], name= @string [name101],]Copy the code

1.2 Catch exceptions

The watch command supports the -e option, which only catches requests when an exception is thrown:

watch com.example.demo.arthas.user.UserController * "{params[0],throwExp}" -e
Copy the code

1.3 Filtering By Time

The watch command supports filtering by request time, for example:

watch com.example.demo.arthas.user.UserController * '{params, returnObj}' '#cost>200'
Copy the code

X. Case: Hot update code: Fire:

The following describes how to dynamically update the code using the jad/ MC/Re-define command.

At present, visit http://localhost:61000/user/0, abnormal returns 500:

{"timestamp":1550223186170."status":500."error":"Internal Server Error"."exception":"java.lang.IllegalArgumentException"."message":"id < 1"."path":"/user/0"}
Copy the code

Let’s change this logic by hot updating the code.

1.1 Jad Decompiles UserController

jad --source-only com.example.demo.arthas.user.UserController > /tmp/UserController.java
Copy the code

The jad decompile is saved in the/TMP/userController.java file.

Use vim to edit/TMP/userController.java:

## New window opens
vim /tmp/UserController.java
Copy the code

For example, if the user id is less than 1, it will return normally without raising an exception:

    @GetMapping(value={"/user/{id}"})
    public User findUserById(@PathVariable Integer id) {
        logger.info("id: {}", (Object)id);
        if(id ! =null && id < 1) {
            return new User(id, "name" + id);
            // throw new IllegalArgumentException("id < 1");
        }
        return new User(id.intValue(), "name" + id);
    }
Copy the code

1.2 SC Searches for the ClassLoader to load the UserController

sc -d *UserController | grep classLoaderHash

classLoaderHash   19469ea2
Copy the code

It can be found that spring Boot LaunchedURLClassLoader@19469ea2 loaded.

Make a note of your classLoaderHash, which you’ll need to use later. In this case, it’s 19469eA2.

1.3 MC

After saving/TMP/userController.java, use MC (Memory Compiler) and specify ClassLoader with -c or –classLoaderClass:

[arthas@2167]$ mc --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader /tmp/UserController.java -d /tmp
Memory compiler output:
/tmp/com/example/demo/arthas/user/UserController.class
Affect(row-cnt:1) cost in 2972 ms.
Copy the code

Java -d/TMP, using -c to specify classLoaderHash:

$ mc -c 1be6f5c3 /tmp/UserController.java -d /tmp
Copy the code

1.4 redefine

Re-define userController.class to reload the newly compiled userController.class:

[arthas@2167]$ redefine /tmp/com/example/demo/arthas/user/UserController.class
redefine success, size: 1, classes:
com.example.demo.arthas.user.UserController
Copy the code

Define success, re-visit user/0, and the result is:

{
  "id": 0,
  "name": "name0"
}
Copy the code

11. Case: Dynamic update application Logger Level: Hotdog:

In this case, change the Logger Level of the application dynamically.

1.1 Finding the ClassLoader of UserController

[arthas@2167]$ sc -d com.example.demo.arthas.user.UserController | grep classLoaderHash
 classLoaderHash   19469ea2
Copy the code

1.2 Obtaining the Logger using OGNL

[arthas@2167]$ ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '@com.example.demo.arthas.user.UserController@logger'
@Logger[
    serialVersionUID=@Long[5454405123156820674],
    FQCN=@String[ch.qos.logback.classic.Logger],
    name=@String[com.example.demo.arthas.user.UserController],
    level=null,
    effectiveLevelInt=@Integer[20000],
    parent=@Logger[Logger[com.example.demo.arthas.user]],
    childrenList=null,
    aai=null,
    additive=@Boolean[true],
    loggerContext=@LoggerContext[ch.qos.logback.classic.LoggerContext[default]],
]
Copy the code

You can see that UserController@logger actually uses logback. You can see that level=null, indicating that the actual final level came from the root Logger.

1.3 Set the Logger level of the UserController separately

[arthas@2167]$ ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '@[email protected](@ch.qos.logback.classic.Level@DEBUG)'
Copy the code

Get UserController@logger again and you can see that it is already DEBUG:

[arthas@2167]$ ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '@com.example.demo.arthas.user.UserController@logger'
@Logger[
    serialVersionUID=@Long[5454405123156820674],
    FQCN=@String[ch.qos.logback.classic.Logger],
    name=@String[com.example.demo.arthas.user.UserController],
    level=@Level[DEBUG],
    effectiveLevelInt=@Integer[10000],
    parent=@Logger[Logger[com.example.demo.arthas.user]],
    childrenList=null,
    aai=null,
    additive=@Boolean[true],
    loggerContext=@LoggerContext[ch.qos.logback.classic.LoggerContext[default]],
]
Copy the code

1.4 Changing the Global Logger Level for LogBack

By obtaining the root Logger, you can change the global logger level:

ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '@org.slf4j.LoggerFactory@getLogger("root").setLevel(@ch.qos.logback.classic.Level@DEBUG)'
Copy the code

Xii. Case: Troubleshooting logger conflict problems: Calendar:

In this case, show how to troubleshoot logger conflicts.

1.1 Determine the Logger system used by the application

In the case of UserController, it uses the SLF4J API, but the actual logger system is Logback.

[arthas@2167]$ ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '@com.example.demo.arthas.user.UserController@logger'
@Logger[
    serialVersionUID=@Long[5454405123156820674],
    FQCN=@String[ch.qos.logback.classic.Logger],
    name=@String[com.example.demo.arthas.user.UserController],
    level=@Level[DEBUG],
    effectiveLevelInt=@Integer[10000],
    parent=@Logger[Logger[com.example.demo.arthas.user]],
    childrenList=null,
    aai=null,
    additive=@Boolean[true],
    loggerContext=@LoggerContext[ch.qos.logback.classic.LoggerContext[default]],
]
Copy the code

1.2 Obtaining the Actual LogBack Configuration File

[arthas@2167]$ ognl --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader '#[email protected]@getLogger("root").loggerContext.objectMap, #map1.get("CONFIGURATION_WATCH_LIST")'@ConfigurationWatchList[ mainURL=@URL[jar:file:/home/shell/demo-arthas-spring-boot.jar!/BOOT-INF/classes! /logback-spring.xml], fileWatchList=@ArrayList[isEmpty=true;size=0], lastModifiedList=@ArrayList[isEmpty=true; size=0], noContextWarning=@Integer[0], context=@LoggerContext[ch.qos.logback.classic.LoggerContext[default]], declaredOrigin=@ConfigurationWatchList[ch.qos.logback.core.joran.spi.ConfigurationWatchList@3a34751e], ]Copy the code

1.3 Use the classloader command to find possible Logger configuration files

[arthas@2167]$ classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader -r logback-spring.xmljar:file:/home/shell/demo-arthas-spring-boot.jar! /BOOT-INF/classes! /logback-spring.xml Affect(row-cnt:1) cost in 1 ms.Copy the code

You know exactly where the loaded configuration came from.

Try loading conflicting files:

[arthas@2167]$ classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader -r logback.xml

Affect(row-cnt:0) cost in 1 ms.
Copy the code

[arthas@2167]$ classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader -r log4j.properties

Affect(row-cnt:0) cost in 0 ms.
Copy the code

Get Spring Context: Kimono:

In this case, we show you get the Spring Context, get the bean, and then call the function.

1.1 Using the TT command to Obtain the Spring Context

Tt is a TimeTunnel, which records the input and return information of each call to a specified method, and can observe these different time calls.

https://arthas.aliyun.com/doc/tt.html
Copy the code

Access: user / 1

You can see that the tt command caught a request:

[arthas@2167]$ tt -t org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter invokeHandlerMethodPress Q or Ctrl+C to abort. Affect(class count: 1 , method count: 1) cost in 115 ms, listenerId: 5 INDEX TIMESTAMP COST(ms) IS-RET IS-EX OBJECT CLASS METHOD P ------------------------------------------------------------------------------------------------------------------------ -- -- -- -- -- 1000 2022-01-27 11:26:4.8384 true or false 0 x3b265475 RequestMappingHandlerAdapte invokeHandlerMethodCopy the code

1.2 Use the TT command to obtain the Spring Context from the call record

Type Q or Ctrl + C to exit the tt-t command above.

tt -i 1000 -w 'target.getApplicationContext()' @AnnotationConfigEmbeddedWebApplicationContext[ reader=@AnnotatedBeanDefinitionReader[org.springframework.context.annotation.AnnotatedBeanDefinitionReader@2e457641], scanner=@ClassPathBeanDefinitionScanner[org.springframework.context.annotation.ClassPathBeanDefinitionScanner@6eb38026],  annotatedClasses=null, basePackages=null, ] Affect(row-cnt:1) cost in 439 ms.Copy the code

1.3 Get the Spring bean and call the function

[arthas@2167]$ tt -i 1000 -w 'target.getApplicationContext().getBean("helloWorldService").getHelloMessage()'
@String[Hello World]
Affect(row-cnt:1) cost in 7 ms.
Copy the code

HTTP request return 401:baby_chick

In this case, show tips for troubleshooting HTTP 401 problems.

https://61000-dot-a5c4lpbtkr.shell.aliyuncs.com/admin
Copy the code

Access: admin

The result is:

Something went wrong: 401 Unauthorized
Copy the code

We know that 401 is usually blocked by a permission managed Filter, so which Filter processed the request and returned 401?

1.1 Trace all Filter functions

Began to trace:

trace javax.servlet.Filter *
Copy the code

Access: admin

AdminFilterConfig$AdminFilter returns 401 at the bottom of the call tree:

+ - [3.806273] ms javax.mail. Servlet. FilterChain: doFilter () | ` - [3.447472] ms Com. Example. Demo. Arthas. AdminFilterConfig $AdminFilter: doFilter () | ` - [0.17259] ms javax.servlet.http.HttpServletResponse:sendError()Copy the code

1.2 Using the Stack to obtain the call stack

Above is by trace command to obtain information, from the results, we can know through the stack trace HttpServletResponse: sendError (), also can know which Filter is returned to the 401

[arthas@2167]$ stack javax.servlet.http.HttpServletResponse sendError 'params[0]==401'
Press Q or Ctrl+C to abort.
Affect(class count: 3 , method count: 4) cost in 176 ms, listenerId: 7
ts=2022-01-27 11:33:39;thread_name=http-nio-61000-exec-6;id=15;is_daemon=true;priority=5;TCCL=org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@e2dab9b
    @org.apache.catalina.connector.Response.sendError()
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:462)
        at com.example.demo.arthas.AdminFilterConfig$AdminFilter.doFilter(AdminFilterConfig.java:38)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

ts=2022-01-27 11:33:39;thread_name=http-nio-61000-exec-6;id=15;is_daemon=true;priority=5;TCCL=org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@e2dab9b
    @org.apache.catalina.connector.ResponseFacade.sendError()
        at com.example.demo.arthas.AdminFilterConfig$AdminFilter.doFilter(AdminFilterConfig.java:38)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

Copy the code

404:honey_pot:

In this case, show tips for troubleshooting HTTP 404 problems.

Access: a.t xt

The result is:

Something went wrong: 404 Not Found
Copy the code

So which Servlet processed the request and returned 404?

1.1 Trace all Servlet functions

trace javax.servlet.Servlet * > /tmp/servlet.txt
Copy the code

Access: a.t xt

/ TMP /servlet.txt

cat /tmp/servlet.txt 
Copy the code

/ TMP /servlet.txt will have a lot of content, so you need to be patient to find the longest place in the call tree.

You can see that the request is ultimately handled by the Freemarker:

` - [13.974188] ms org. Springframework. Web. Servlet. ViewResolver: resolveViewName (); + - [0.045561] ms javax.mail. Servlet. GenericServlet: < init > (+) - [min = 0.045545 ms, Max = 0.074342 ms, total = 0.119887 ms, count = 2] Org. Springframework. Web. Servlet. The freemarker. FreeMarkerView $GenericServletAdapter: < init > (+) - [0.170895] ms Javax.mail. Servlet. GenericServlet: init () | ` - [0.068578] ms javax.mail. Servlet. GenericServlet: init () | ` - [0.021793] ms Javax.mail. Servlet. GenericServlet: init () ` - [0.164035] ms javax.mail. Servlet. GenericServlet: getServletContext ()Copy the code

Hourglass_flowing_sand:

The following describes the function of the classloader command.

Visit a JSP page and trigger the JSP to load: hello

1.1 List all ClassLoaders

[arthas@2167]$ classloader -lname loadedCount hash parent BootstrapClassLoader 3748 null null com.taobao.arthas.agent.ArthasClassloader@55948442 2628  55948442 sun.misc.Launcher$ExtClassLoader@7 d0587f1 java.net.FactoryURLClassLoader@7de451d6 841 7de451d6 sun.misc.Launcher$AppClassLoader@1 b6d3586 g.apache.jasper.servlet.JasperLoader@a9af914 1 a9af914 TomcatEmbeddedWebappClassLoader context: ROOT delegate: true ----------> Parent Classloader: org.springframework.boot.loader.La unchedURLClassLoader@19469ea2 0 e2dab9b org.springframework.boot.loader.La unchedURLClassLoader@19469ea2 9ea2 org.springframework.boot.loader.LaunchedURLClassLoader@19469ea2 5325 19469ea2 sun.misc.Launcher$AppClassLoader@1 b6d3586 sun.misc.Launcher$AppClassLoader@1b6d3586 45 1b6d3586 sun.misc.Launcher$ExtClassLoader@7 d0587f1 sun.misc.Launcher$ExtClassLoader@7d0587f1 63 7d0587f1 null Affect(row-cnt:8) cost in 14 ms.Copy the code

• TomcatEmbeddedWebappClassLoader loading the class number is zero, so in the spring the boot embedded tomcat, it’s just an empty shell, all class loadingLaunchedURLClassLoaderTo complete the

1.2 List all classes loaded in ClassLoader

Listed above org. Apache. Jasper. Servlet. JasperLoader loaded classes:

classloader -a --classLoaderClass apache.jasper.servlet.JasperLoader
Copy the code

classloader -a --classLoaderClass apache.jasper.servlet.JasperLoader
 hash:1698045338, org.apache.jasper.servlet.JasperLoader@65361d9a
 org.apache.jsp.jsp.hello_jsp
Copy the code

• Note: Same as OGNL, also available-c <hashcode>without--classLoaderClassThe specified

1.3 Decompile JSP code

jad org.apache.jsp.jsp.hello_jsp
Copy the code

$ jad org.apache.jsp.jsp.hello_jsp

ClassLoader:
+-org.apache.jasper.servlet.JasperLoader@65361d9a
  +-TomcatEmbeddedWebappClassLoader
      context: ROOT
...
Copy the code

1.4 Viewing the ClassLoader tree

[arthas@1265]$ classloader -t
+-BootstrapClassLoader                                                                                                       
+-sun.misc.Launcher$ExtClassLoader@7d0587f1                                                                                  
  +-com.taobao.arthas.agent.ArthasClassloader@744a1259                                                                       
  +-sun.misc.Launcher$AppClassLoader@1b6d3586                                                                                
    +-org.springframework.boot.loader.LaunchedURLClassLoader@19469ea2                                                        
                                                                                     
                                                                                                     
                                                                                                    
                                                                                     
                                                     assLoader@19469ea2
        +-org.apache.jasper.servlet.JasperLoader@7ca74939                                                                    
Affect(row-cnt:7) cost in 20 ms.
Copy the code

Note: Please override

with your classLoaderHash value and then manually execute the following commands

1.5 List the ClassLoader urls

Spring LaunchedURLClassLoader’s hashcode is 19469eA2, which lists all urls with the -c or –classLoaderClass argument:

classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader
Copy the code

1.6 Loading resource files in a specified ClassLoader

Find the specified resource file: this — classLoaderClass org. Springframework.. The boot loader. LaunchedURLClassLoader -r logback – spring. XML

[arthas@1265]$ classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader -r logback-spring.xmljar:file:/home/shell/demo-arthas-spring-boot.jar! /BOOT-INF/classes! /logback-spring.xml Affect(row-cnt:1) cost in 1 ms.Copy the code

1.7 Attempting to load the specified class

For example, try loading java.lang.String with the spring LaunchedURLClassLoader above:

[arthas@1265]$ classloader --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader --load java.lang.String
load class success.
 class-info        java.lang.String                                                                                          
 code-source                                                                                                                 
 name              java.lang.String                                                                                          
 isInterface       false                                                                                                     
 isAnnotation      false                                                                                                     
 isEnum            false                                                                                                     
 isAnonymousClass  false                                                                                                     
 isArray           false                                                                                                     
 isLocalClass      false                                                                                                     
 isMemberClass     false                                                                                                     
 isPrimitive       false                                                                                                     
 isSynthetic       false                                                                                                     
 simple-name       String                                                                                                    
 modifier          final,public                                                                                              
 annotation                                                                                                                  
 interfaces        java.io.Serializable,java.lang.Comparable,java.lang.CharSequence                                          
 super-class       +-java.lang.Object                                                                                        
 class-loader                                                                                                                
 classLoaderHash   null
Copy the code

Top N threads hammer_and_pick

1.1 Viewing Information about All Threads

[arthas@1265]$ threadThreads Total: 35, NEW: 0, RUNNABLE: 11, BLOCKED: 0, WAITING: 14, TIMED_WAITING: 5, TERMINATED: 0, Internal threads: 5 ID NAME GROUP PRIORITY STATE %CPU DELTA_TIME TIME INTERRUPTE DAEMON -1 C2 CompilerThread0 - -1-14.07 0.029 0:8.038 False true-1 C1 CompilerThread1 - -1-1.02 0.002 0:2.718 false true 44 arthas-command-execute system 5 RUNNABLE 0.56 0.001 0:0.828 False true-1 VM Periodic Task Thread -1-0.11 0.000 false true-1 VM Periodic Task Thread -1-0.07 0.000 false true-1 VM Periodic Task Thread -1-0.07 0.000 0:0.725 false true 28 HTTP-NiO-61000-AsyncTimeout main 5 TIMED_WAI 0.02 0.000 0:0.035 False true 2 Reference Handler System 10 WAITING 0.0 0.000 0:0.009 false true 3 Finalizer system 8 WAITING 0.0 0.000 0:0.018 false true 4 Signal Dispatcher System 9 RUNNABLE 0.0 0.000 0:0.006 false true 31 Attach Listener System 9 RUNNABLE 0.0 0.000 0:0.006 false True 33 arthas-Timer System 9 WAITING 0.0 0.000 0:0.000 false true 36 arthas-nettyHttpTelNetbootstra system 5 RUNNABLE 0.0 0.000 0:0.024 false true 37 arthas-NettyWebsocketTtyBootst System 5 RUNNABLE 0.0 0.000 0:0.001 false true 38 arthas-NettyWebsocketTtyBootst System 5 RUNNABLE 0.0 0.000 0:0.001 false true 38 arthas-NettyWebsocketTtyBootst System Arthas-nettywebsocketttybootst System 5 RUNNABLE 0.0 0.000 0:0.001 false true 39 arthas-shell-server system 9 TIMED_WAI 0.0 0.000 0:0.002 false true 40 arthas-session-Manager System 9 TIMED_WAI 0.0 0.000 0:0.001 false true 41 arthas-Session-Manager system 9 TIMED_WAI 0.0 0.000 0:0.001 false true 41 Arthas-userstat System 9 WAITING 0.0 0.000 0:0.000 false true 43 arthas-nettyHttpTelNetbootstra system 5 RUNNABLE 0.0 0.000 0-0. 238 false true 13 ContainerBackgroundProcessor [S main 5 TIMED_WAI 0-0 at 0.000 0.0. 007 false true 14 container - 0 The main five TIMED_WAI 0-0 at 0.0 0.000. 002 false false. 15 NioBlockingSelector BlockPolle main 5 RUNNABLE 0-0 at 0.000 0.0. 027 false True 16 http-nio-61000-exec-1 main 5 WAITING 0.0 0.000 0:1.055 false true 17 http-nio-61000-exec-1 main 5 WAITING 0.0 0.000 0:1.055 false true 17 http-nio-61000-exec-2 main 5 WAITING 0.0 0.000 0:0.025 false true 18 HTTP-nio-61000-exec-3 main 5 WAITING 0.0 0.000 0:0.000 false true 19 http-nio-61000-exec-4 Main 5 WAITING 0.0 0.000 0:0.000 false true 20 HTTP-niO-61000-exec-5 main 5 WAITING 0.0 0.000 0:0.000 false true 21 Http-nio-61000-exec-6 main 5 WAITING 0.0 0.000 0:0.000 false true 22 http-nio-61000-exec-7 main 5 WAITING 0.0 0.000 0:0.000 false true 22 HTTP-nio-61000-exec-7 main 5 WAITING 0.0 0.000 0:0.000 false true 23 HTTP-nio-61000-exec-8 main 5 WAITING 0.0 0.000 0:0.000 false true 24 http-nio-61000-exec-9 main 5 WAITING 0.0 0.000 0:0.000 false true 25 HTTP-nio-61000-exec-10 main 5 WAITING 0.0 0.000 0:0.000 false true 26 Http-nio-61000-clientpoller-0 main 5 RUNNABLE 0.0 0.000 0:0.039 false true 27 HTTP-nio-61000-acceptor-0 main 5 RUNNABLE 0.0 0.000 0:0.006 false true 30 DestroyJavaVM main 5 RUNNABLE 0.0 0.000 0:3.569 false false -1 Service Thread - 1-0.0 0.000 the 0-0. 000 false trueCopy the code

1.2 Viewing the stack of a specific thread

Check thread ID 16’s stack:

[arthas@1265]$ thread 16
"http-nio-61000-exec-1" Id=16 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42ab2c2f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42ab2c2f
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Copy the code

1.3 Viewing stacks of top N CPU Usage Threads

[arthas@1265]$ thread -n 3"C1 CompilerThread1" [Internal] cpuUsage=0.8% deltaTime=1ms time=2743ms "C2 CompilerThread0" [Internal] cpuUsage=0.43% DeltaTime =0ms time=8072ms "arthas-command-execute" Id=44 cpuUsage=0.34% deltaTime=0ms time=839ms RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:461) at  com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:206) at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:122) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111) at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108) at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Copy the code

1.4 Viewing top N THREAD Stacks in CPU Usage Within 5 seconds

[arthas@1265]$ thread -n 3 -i 5000"C2 CompilerThread0" [Internal] cpuUsage=0.14% deltaTime=7ms time=8081ms "VM Periodic Task Thread" [Internal] CpuUsage = 0.08%deltatime =4ms time=523ms "C1 CompilerThread1" [Internal] cpuUsage= 0.03%deltatime =1ms time=2748msCopy the code

1.5 Check whether the thread is blocked

[arthas@1265]$ thread -b
No most blocking thread found!
Copy the code

Web Console :hamster

Arthas supports connection through Web sockets.

Arthas supports connection through Web sockets.

Local experience

When starting locally, Arthas can be accessed through a browser by visiting http://127.0.0.1:8563/.

If you put up the hamburger, leave /Stop :hamburger:

1.1 the reset

Arthas actually modifies the application bytecode to insert enhanced code when using commands such as Watch /trace. This enhancement code can be cleared by explicitly executing the reset command.

reset
Copy the code

1.2 out of Arthas

You can exit Arthas using the exit or quit command.

exit
Copy the code

Ctrl+C

After exiting Arthas, you can connect again with java-jar arthas-boot.jar.

java -jar arthas-boot.jar
Copy the code

Ctrl+C

Quit Arthas completely

The exit/quit command simply exits the current session and arthas Server is still running in the target process.

To exit Arthas completely, execute the stop command.

stop
Copy the code

Arthas-boot supports kiwi_fruit:

Arthas-boot. jar supports many parameters. You can run the Java -jar arthas-boot.jar -h command to view these parameters.

java -jar arthas-boot.jar -h
Copy the code

1.1 Allow external access

By default arthas Server listens on 127.0.0.1. If you want remote access, you can use –target-ip.

java -jar arthas-boot.jar --target-ip
Copy the code

1.2 List all versions

java -jar arthas-boot.jar --versions
Copy the code

Using the specified version:

Java -jar arthas-boot.jar --use-version 3.1.0Copy the code

1.3 Listen only on the Telnet port, not the HTTP port

java -jar arthas-boot.jar --telnet-port 9999 --http-port -1
Copy the code

1.4 Print the details of the operation

java -jar arthas-boot.jar -v
Copy the code