The author | He Bo

Arthas official Community is holding an essay call for prizes. Click to submit.

1. Introduction

background

Excessive CPU usage often occurs in online codes. According to my previous experience, I will use the TOP instruction and further check the specific information with the help of JSTACK to troubleshoot problems. However, basically, I cannot escape the situation of reissuing packages, even if it is an incremental package, and the application also needs to stop for a short time. Then the operations brothers asked me to give Arthas a try and said it would do hot updates to the code, so I came to give it a try.

For installation and basic use of Arthas, please refer to the following two articles:

Arthas installs and listens to SpringBoot applications
Arthas basic instructions to use

The environment

JDK1.8 SPringBoot 2.2.2 Arthas Linux

Test code:

@RequestMapping(value = "/bigThread")
@ResponseBody
public String bigThread(int id) {
    ArthasService.test();
    while (true) {
        Thread t2 = new Thread();
        t2.start();
        id ++;
        if(100000 == id) {
            returnString.valueOf(id); }}}Copy the code

Train of thought

2. Thread-b checks whether a blocked thread exists

Thread-b: identifies the thread that is currently blocking other threads. If the thread is not found after execution, it indicates that the thread is not always blocking and executing.

3. Thread View the thread with the highest usage

If a thread is not followed by a parameter, the current information about all threads is displayed. I think thread-n 10 is sufficient.

As can be seen from the following figure, our application occupies 77% of the CPU in an instant. (Here, I check the status of thread at the moment of initiating the request, so it is more intuitive. The production environment should be more intuitive only when it is blocked and deadlocked.)

4. Thread ID View detailed information

Building on the previous step, let’s take a closer look at Thread 15 (because ID=15 above).

The thread is waiting for a condition to continue executing. As you can see, the method is called linkedBlockingQueue.take.

public E take(a) throws InterruptedException {
    E x;
    int c = -1;
    final AtomicInteger count = this.count;
    final ReentrantLock takeLock = this.takeLock;
    takeLock.lockInterruptibly();
    try {
        while (count.get() == 0) {
            notEmpty.await();
        }
        x = dequeue();
        c = count.getAndDecrement();
        if (c > 1)
            notEmpty.signal();
    } finally {
        takeLock.unlock();
    }
    if (c == capacity)
        signalNotFull();
    return x;
}
Copy the code

AtomicInteger refers to atomicity under high concurrency, and ReentrantLock refers to ReentrantLock.

Notempty.await () consumes data from the queue. When the queue is empty, the thread blocks, so we know roughly that the problem is thread blocking, but we still don’t know which line of code is causing the problem.

If you know exactly what code has been changed this time, you can directly perform Step 6. If you don’t know, you can use Step 5 to locate the problem.

5. Watch checks which Controller executes the code

watch org.springframework.web.servlet.DispatcherServlet getHandler returnObj
Copy the code

This script detects all methods that are matched to the Handler via the DispatcherServlet, i.e. requests to the Controller, as follows:

After finding the corresponding code, we will further observe the abnormal information. There may be a question: why do I need to operate so tedious when I can check the error information through the log? My business scenario is: the log is still very large, just got it was brushed over, at this time the log location is not very good operation, of course, want to get down the log certainly can, also very intuitive, I generally go to view the log problem location, here is also to provide an idea.

6. Watch Indicates the exception information of the method

Watch class full path method name"{params[0],throwExp}" -e -x 2
Copy the code

As shown above, the error is presented intuitively, and it can be fixed below. Here, we can also check the execution time through the trace command:

The full path method name of the trace class"{params[0],throwExp}" -e -x 2
Copy the code

The following information is returned, as well as error messages and the duration of each method execution.

[arthas@10999]$ trace com.arthas.controller.OrderController bigThread
Press Q or Ctrl+C to abort.
Affect(class count: 1,method count: 1) cost in 53 ms.listenerId: 10 ` -ts=2020- 08 -19 14:45:57; thread_name=http-nio-0.0. 0. 0-8080-exec-10; id=16; is_daemon=true; priority=5; TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `---[1452.684406ms] com.arthas.controller.OrderController:bigThread() [throws Exception]
        +---[0.168814ms] com.arthas.service.ArthasService:test() #20` -throw:java.lang.OutOfMemoryError #-2 [unable to create new native thread]
Copy the code

7. Jad decompiles hot updates

Once we know the problem above, we can locate the problem.

Command: jad class full path method name

[arthas@13190]$ jad com.arthas.controller.OrderController ClassLoader: +-org.springframework.boot.loader.LaunchedURLClassLoader@17f052a3 +-sun.misc.Launcher$AppClassLoader@3d4eac69 +-sun.misc.Launcher$ExtClassLoader@45f45fa1 Location: file:/opt/software/arthas/Arthas.jar! /BOOT-INF/classes! //* * Decompiled with CFR. * * Could not load the following classes: * com.arthas.service.ArthasService * org.springframework.stereotype.Controller * org.springframework.web.bind.annotation.RequestMapping * org.springframework.web.bind.annotation.ResponseBody */
package com.arthas.controller;

import com.arthas.service.ArthasService;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
public class OrderController {
    @RequestMapping(value={"/bigThread"})
    @ResponseBody
    public String bigThread(int id) {
        ArthasService.test();
        do {
            Thread t2 = new Thread();
            t2.start();
        } while (100000! = ++id);return String.valueOf(id);
    }
}

Affect(row-cnt:1) cost in 1513 ms.
Copy the code

The code is decompiled at this point, so we need to output it as a Java file in order to be able to change it.

Instructions: jad com. Arthas. Controller. OrderController > / TMP/OrderController. Java

That is, jad full-path method name > storage path/storage name

Then go to the TMP directory to modify the Java file using vi. After the modification is complete, check the corresponding classloader to prepare for compilation.

sc -d *OrderController | grep classLoaderHash
mc -c 17f052a3 /tmp/OrderController.java -d /tmp
Copy the code

But there is a compilation error here.

So if we compile the class file locally and upload it, it’s the same.

Call before compilation

[arthas@13190]$ trace com.arthas.controller.OrderController bigThread
Press Q or Ctrl+C to abort.
Affect(class count: 1,method count: 1) cost in 77 ms.listenerId: 2 ` -ts=2020- 08 -19 15:51:46; thread_name=http-nio-0.0. 0. 0-8080-exec-1; id=d; is_daemon=true; priority=5; TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `---[6734.666529ms] com.arthas.controller.OrderController:bigThread() [throws Exception]
        +---[0.786517ms] com.arthas.service.ArthasService:test() #20` -throw:java.lang.OutOfMemoryError #-2 [unable to create new native thread]
Copy the code

Pre-update code

@RequestMapping(value = "/bigThread")
@ResponseBody
public String bigThread(int id) {
    ArthasService.test();
    while (true) {
        Thread t2 = new Thread();
        t2.start();
        id ++;
        if(100000 == id) {
            returnString.valueOf(id); }}}Copy the code

Updated code

@RequestMapping(value = "/bigThread")
@ResponseBody
public String bigThread(int id) {
    ArthasService.test();
    Thread t2 = new Thread();
    t2.start();
    return "success";
}
Copy the code

Compile the instruction

[arthas@13190]$ redefine /tmp/OrderController.class
redefine success, size: 1, classes:
com.arthas.controller.OrderController
Copy the code

Called three times after compilation

`---ts=2020- 08 -19 15:52:02; thread_name=http-nio-0.0. 0. 0-8080-exec-3; id=f; is_daemon=true; priority=5; TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `---[5.609405ms] com.arthas.controller.OrderController:bigThread()
        `---[0.204675ms] com.arthas.service.ArthasService:test() #20

`---ts=2020- 08 -19 15:52:04; thread_name=http-nio-0.0. 0. 0-8080-exec-4; id=10; is_daemon=true; priority=5; TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `---[3.900149ms] com.arthas.controller.OrderController:bigThread()
        `---[0.14636ms] com.arthas.service.ArthasService:test() #20

`---ts=2020- 08 -19 15:52:04; thread_name=http-nio-0.0. 0. 0-8080-exec-5; id=11; is_daemon=true; priority=5; TCCL=org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader@1f1c7bf6 `---[1.90945ms] com.arthas.controller.OrderController:bigThread()
        `---[0.147353ms] com.arthas.service.ArthasService:test() #20
Copy the code

It can be found that the time changes from 6734.666529ms to about 3ms, indicating that the hot update code has taken effect.

Make a flame profile for subsequent analysis

As shown below:

Arthas’s essay campaign is in full swing

Arthas is officially holding an essay call if you have:

Problems identified using Arthas
Source code interpretation of Arthas
Advise Arthas
No limit, other Arthas related content

Welcome to participate in the essay activity, there are prizes to win oh ~ click to submit

“Alibaba Cloud originator focuses on micro-service, Serverless, container, Service Mesh and other technical fields, focuses on the trend of cloud native popular technology, large-scale implementation of cloud native practice, and becomes the public account that most understands cloud native developers.”

Arthas implements a CPU sweep and code hot update