0, background

For those of you who do a lot of back-end service development, you have encountered a very high CPU load. Especially on weekends or in the middle of the night, someone in the group suddenly reported that the load on the online machine is particularly high, and students who are not familiar with the positioning process and ideas may get on the server and get confused, and the positioning process will turn around.

In this regard, many students have organized the relevant process or methodology. For example, it takes several steps to put an elephant in the refrigerator. The traditional plan is generally four steps:

  1. Top oder by with P: 1040
  2. Top-hp process PID: 1073 // Find the associated load thread PID
  3. Printf “0x%x\n” Thread PID: 0x431 // Convert thread PID to hexadecimal in preparation for looking up jStack logs later
  4. Jstack process PID | vim + / hex threads PID / /, for example: 1040 | jstack vim + / 0 x431 –

However, every minute counts for online problem location, and the above four steps are still too tedious and time-consuming. Is it possible to encapsulate into a tool that can locate the problem with one click and find the problem line in the second level?

Of course you can! The maturity of toolchain not only reflects the operation and maintenance ability of a developer, but also reflects the efficiency consciousness of the developer. Oldratlee of Taobao has packaged the above process into a tool: show-busy-Java-threads. sh (click to download directly, or refer to the link at the end of the article), which can be very convenient to locate such problems online. I will give two examples to see the actual effect.

Quick installation for use:

source <(curl -fsSL https://raw.githubusercontent.com/oldratlee/useful-scripts/master/test-cases/self-installer.sh)
Copy the code

Java regular expression backtracking causes 100% CPU

import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;


public class RegexLoad {
    public static void main(String[] args) {
        String[] patternMatch = {"([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)",
                "([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)"};
        List<String> patternList = new ArrayList<String>();

        patternList.add("Avg Volume Units product A + Volume Units product A");
        patternList.add("Avg Volume Units /  Volume Units product A");
        patternList.add("Avg retailer On Hand / Volume Units Plan / Store Count");
        patternList.add("Avg Hand Volume Units Plan Store Count");
        patternList.add("1 - Avg merchant Volume Units");
        patternList.add("Total retailer shipment Count");

        for (String s :patternList ){

            for(int i=0;i<patternMatch.length;i++){
                Pattern pattern = Pattern.compile(patternMatch[i]);

                Matcher matcher = pattern.matcher(s);
                System.out.println(s);
                if (matcher.matches()) {

                    System.out.println("Passed");
                }else
                    System.out.println("Failed;");
            }

        }
    }
}
Copy the code

After compiling and running the above code, we can observe that the server has an additional Java process with 100% CPU:

How do you use it?

Show-busy-java-threads. sh # Find the threads that consume the most CPU from all Java processes (default 5) and print their stack.

Show-busy-java-threads. sh -c < Number of threads to display >

Show-busy-java-threads. sh -c < number of thread stacks to display > -p < specified Java Process>

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # note: # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # if the user of the Java process and execution of the script for the current user is different, a jstack is the Java process. Sudo show-busy-java-threads.sh sudo show-busy-java-threads.sh sudo show-busy-java-threads

Example:

work@dev_zz_Master 10.48.186.32 23:45:50 ~/demo > bash show-busy java-threads.sh [1] busy (96.2%) thread(8577/0x2181) stack of java process(8576) under user(work): "main" prio=10 tid=0x00007f0c64006800 nid=0x2181 runnable [0x00007f0c6a64a000] java.lang.Thread.State: RUNNABLE at java.util.regex.Pattern$GroupHead.match(Pattern.java:4168) at java.util.regex.Pattern$Loop.match(Pattern.java:4295) ... at java.util.regex.Matcher.match(Matcher.java:1127) at java.util.regex.Matcher.matches(Matcher.java:502) at Regexload.main (regexload.java :27) [2] Busy(1.5%) Thread (8591/0x218f) Stack of Java processes (8576) under user(work): "C2 CompilerThread1" daemon prio=10 tid=0x00007f0c64095800 nid=0x218f waiting on condition [0x0000000000000000] Java.lang.Thread.State: RUNNABLE [3] Busy(0.8%) Thread (8590/ 0x218E) Stack of Java processes (8576) under user(work): "C2 CompilerThread0" daemon prio=10 tid=0x00007f0c64093000 nid=0x218e waiting on condition [0x0000000000000000] Java.lang.Thread.State: RUNNABLE [4] Busy(0.2%) Thread (8593/0x2191) Stack of Java processes (8576) under user(work): "VM Periodic Task Thread" prio=10 TID =0x00007f0c640a2800 nID =0x2191 Waiting on condition [5] Busy(0.1%) thread(25159/0x6247) stack of java process(25137) under user(work): "VM Periodic Task Thread" prio=10 tid= 0x00007F13340B4000 nID =0x6247 waiting on condition work@dev_zz_Master 10.48.186.32  23:46:04 ~/demo >Copy the code

As you can see, one click directly locate the exception line of code, is not very convenient?

2, thread deadlock, program hang

import java.util.*; public class SimpleDeadLock extends Thread { public static Object l1 = new Object(); public static Object l2 = new Object(); private int index; public static void main(String[] a) { Thread t1 = new Thread1(); Thread t2 = new Thread2(); t1.start(); t2.start(); } private static class Thread1 extends Thread { public void run() { synchronized (l1) { System.out.println("Thread 1: Holding lock 1..." ); try { Thread.sleep(10); } catch (InterruptedException e) {} System.out.println("Thread 1: Waiting for lock 2..." ); synchronized (l2) { System.out.println("Thread 2: Holding lock 1 & 2..." ); } } } } private static class Thread2 extends Thread { public void run() { synchronized (l2) { System.out.println("Thread  2: Holding lock 2..." ); try { Thread.sleep(10); } catch (InterruptedException e) {} System.out.println("Thread 2: Waiting for lock 1..." ); synchronized (l1) { System.out.println("Thread 2: Holding lock 2 & 1..." ); } } } } }Copy the code

Effect after execution:

How to use the tool to locate:

One-click location: you can clearly see that threads are locking each other’s waiting resources, causing deadlocks. You can directly locate the code line and the specific cause.

From the above two examples, I think you should have a deep understanding of this tool and what the tool can solve the problem, when the CPU 100% problem can no longer panic. But more rely on your own practice, after all, the real knowledge comes from practice

3, free and practical script tools package

In addition to show-busy-Java-threads.sh mentioned in the main text, Oldratlee also integrated a lot of common scripting tools involved in development, operation and maintenance. I simply listed the ones I found particularly useful:

(1) the show – duplicate – Java classes

Occasionally, I encounter a class exception that is inexplicable after the launch of the local development and testing, and the cause is a Jar conflict. This tool can find duplicate classes in either the Java Lib (Java libraries, or Jar files) or the Class directory (Class directories).

One of the troubling problems of Java development is Jar conflicts (that is, multiple versions of jars), or duplicate classes. NoSuchMethod and other problems will occur, not necessarily at that time. Look for jars that have duplicate classes to prevent this from happening.

Show-duplicate-java-classes is a duplicate class in all jars in the current directory

Show-duplicate-java-classes path/to/lib_dir1 /path/to/lib_dir2

Find multiple duplicate classes in the specified Class directory. The Class directory uses the -c option to specify show-duplicate-java-classes -c path/to/class_dir1 -c /path/to/class_dir2

Jar show-duplicate-java-classes path/to/lib_dir1 /path/to/lib_dir2 -c path/to/class_dir1 -c path/to/class_dir2

Such as:

$MVN install... $MVN install... $unzip target/*. War -d target/war... $show-duplicate-java-classes -c target/war/ web-INF /classes target/war/ web-INF /lib $show-duplicate-java-classes -c target/war/ web-INF /libCopy the code

(2) the find – in – jars

Look for class or resource files in all JAR files in the current directory.

Usage: Note that Pattern is an extended regular expression of grep.

find-in-jars 'log4j\.properties'
find-in-jars 'log4j\.xml$' -d /path/to/find/directory
find-in-jars log4j\\.xml
find-in-jars 'log4j\.properties|log4j\.xml'Copy the code

Example:

$. / find - in - jars' Service. The class $. / WEB - INF/libs/spring - 2.5.6. SEC03. Jar! Org/springframework stereotype/Service. Class. / RPC - benchmark - 0.0.1 - the SNAPSHOT. The jar! com/taobao/rpc/benchmark/service/HelloService.classCopy the code

(3) Housemd PID [JAVa_HOME]

In the early days, we used BTrace to troubleshoot problems. In addition to marveling at the power of BTrace, we also failed the online system several times. In 2012, Jushi of Taobao wrote HouseMD, integrating several commonly used Btrace scripts to form an independent style application. Its core code is Scala. HouseMD is a diagnostic tool based on bytecode technology, so it can diagnose any language that ends up running on the JVM as bytecode other than Java, such as Clojure(thanks @killme2008 for getting started), Scala, Groovy, JRuby, Jython, Kotlin, etc.

Runtime tracking of Java programs using Housemd includes:

  • Viewing loaded classes
  • Tracking method
  • Viewing environment Variables
  • View object property values
  • For more information: github.com/CSUG/HouseM…

(4) JVM PID

The JVM debug tool allows you to check the status of the Java stack, heap, thread, gc, etc.

Thread = = = = = = = = = = = = = = = 1: see the highest CPU thread condition 2:3: print all threads to print the number of threads 4: according to the thread state statistics the number of threads = = = = = = = = GC relevant = = = = = = = 5: Garbage collection statistics (including the reason) can specify the interval and the number of times, the default is 1 second, 10 times 6: shows the space of each generation in the heap can specify the interval and the number of times, the default is 1 second, 5 times 7: Garbage collection statistics. You can specify the interval and number of executions. The default is 1 second and 10 times. 8: Print the perm area memory * will cause the program to pause the response * 9: check the directbuffer ======== Heap object related ======= 10: Dump heap to file * will cause the program to pause the response * default save to 'PWD' /dump.bin, other path can be specified 11: Trigger full GC. * causes the program to pause the response * 12: prints the JVM heap statistics * 13: prints the top20 objects in the JVM heap. Arguments: 1: sort by number of instances,2: sort by memory usage, default 1 14: prints the top20 objects in the JVM heap after triggering full gc. Arguments: 1: sort by number of instances,2: Sort by memory usage (default: 1) 15: output all objects generated by class loaders in PERm. You can specify the interval and number of executions ======== other ======= 16: Prints finalzer queues 17: displays classloader statistics 18: displays JIT compilation statistics 19: deadlock detection 20: waits X seconds, default is 1 q: exitCopy the code

After entering the JVM tool, you can enter the serial number to execute the corresponding command. You can execute multiple commands at once, using a semicolon (;). Separation, as in: 1; 3; 4. 5; 6 Each command can contain parameters separated by colons (:). Parameters of the same command are separated by commas (,), for example, Enter command queue:1. Days of 00100; 10:/data1/output.bin

(5) greys <PID>[@ip :PORT]

PS: currently Greys only supports Java6+ on Linux/Unix/Mac, but not on Windows

Greys is an exception diagnostic tool for JVM processes that can easily troubleshoot problems without interrupting program execution. Like HouseMD, Greys-Anatomy was named after the eponymous TV series “Grey’s Anatomy” as a tribute to its predecessor. The code was written with reference to the ideas of two predecessors, BTrace and HouseMD.

Use Greys for runtime tracking of Java programs (no parameters, need to first greys -C PID, then Greys). The following operations are supported:

  • View loading class, method information
  • View the current basic information about the JVM
  • Method execution monitoring (call volume, failure rate, response time, etc.)
  • Methods perform data observation, recording, and playback (parameters, return results, exception information, etc.)
  • Method calls track rendering
  • For more information: github.com/oldmanpushc…

(6) SJK < CMD > <arguments> SJK –commands SJK –help < CMD >

Use SJK for Java diagnostics, performance checks and optimization tools

  • Ttop: Monitors CPU usage for individual threads of a specified JVM process
  • JPS: improved version
  • Hh: jmap-histo enhanced version
  • Gc: Reports garbage collection information in real time
  • For more information: github.com/aragozin/jv…

Refer:

[1] oldratlee/useful-scripts

Github.com/oldratlee/u…

[2] awesome-scripts

Github.com/superhj1987…