I accidentally stepped on Ali’s Arthas pit during an online problem check

Recently, I checked an online question for a new school recruit. The question itself was not too difficult, but I stumbled on an Arthas pit in the process, which was quite interesting.

Some useful tools, including tcpdump, arthas, simpleHTTPServer, etc., are also shared.

1. Problem description

A newly developed function, in a nutshell, is to read the data from the database displayed in the foreground.

Local start service debugging, call API with Postman, return data display normal, the Chinese in the data is normal.

However, after being deployed in an online environment, non-Chinese data is normal, but Chinese characters are displayed as garbled characters through the call interface of Chrome and Postman.

2. Check your thinking

The first response to this problem is that there is something wrong with the content-Type of the request.

The content-type of the request and response is application/json. Charset =UTF-8, no problem.

Then I googled a bunch of garbled issues, which basically refer to Spring’s HttpMessageConverter problem or content-type.

We’ll just have to dig deep.

The main idea of the investigation is to determine which step is generated by the garbled code.

• One is to find the place in the database, we need to capture a package to confirm, but our local service call is correct, so there should be no problem in this step.

• One is where the application service returns, which needs to be confirmed by capturing a packet. In an online deployment environment, packets are captured when application services return data using tcpdump.

• One is that there is data transformation in the code logic, and this requires arthas to look at runtime data for online applications.

3. Use tcpdump to capture packets and view the response of the server

3.1 What is tcpdump?

Tcpdump is a tool for analyzing network packet capture in Linux. Tcpdump is frequently used in daily Network management on Linux. Being familiar with tcpdump helps improve work efficiency.

3.2 Packet Capture

To obtain service request packets, log in to the corresponding server (or THE POD of K8S) and use tcpdump to capture them.

As a warm man, I will write down to you step by step from installation to use 🙂

1) Installation tools

If tcpdump has not been installed on your server, run the following command to install it

yum -y install net-tools
Copy the code

2) Check the network status

If the service has multiple network adapters, run the following command to view the network adapters

Netstat -i
Copy the code

3) Deploy packet capture

tcpdump -i eth0 tcp -w xxx.cap
Copy the code

• En0 indicates the listening network adapter
• TCP Indicates the packet type
• -w Specifies the output file name

There are a lot of other options that you can filter, but you can look them up on the Internet and I won’t go through them here.

4) Invoke the request

After tcpdump is deployed, API requests are made to the server. In this case, all related TCP packets are output to the xxx.cap file.

3.3 Packet Parsing

1) Send the xxx.cap file locally

Generally, you can use the SCP command to send it directly

scp xxxx.cap [email protected]:/path
Copy the code

If SCP is inconvenient to use when transferring server files to local, such as some firewall restrictions.

You can also use Python to start a Web service on the server (port is customizable).

python -m SimpleHTTPServer 18888 &
Copy the code

Then download the file locally using wget.

2) Parse the CAP file

After the CAP file is obtained locally, you can use the Wireshark to parse the cap file. The following information is displayed:

After the API packet is parsed, the returned Chinese characters are garbled. It is confirmed that the response content sent by the server is garbled.

So, you just have to continue to troubleshoot the application itself.

4. Arthas check for running code online

Arthas is Alibaba, open source Java diagnostic tool, when you meet the following similar problems and helpless, can try to use Arthas (the use of the official documents in more detail: arthas.aliyun.com/doc/quick-s…

• From which JAR is this class loaded? Why are all kinds of class-related exceptions reported?
• Problems can not be debug online, and do not want to frequently log and re-publish
• There is a problem with a user’s data processing online, but it cannot be debugged online, and it cannot be reproduced offline!

4.1 Quick Installation and Startup

curl -O https://arthas.aliyun.com/arthas-boot.jar

java -jar arthas-boot.jar
Copy the code

4.2 Running the code Returns a Troubleshooting

The screen, use the function of arthas’s watch (the use of the official documents in more detail: arthas.aliyun.com/doc/watch.h…

Let’s take a look at the response of the controller layer of the online running application to the request. Without adding logs and redeploying, we can immediately see the result of the online code.

watch xxx.xxx.controller method "{params,returnObj}" -x 2
Copy the code

Then an API call is made and arthas displays the following result:

As you can see, the controller method returns something that is garbled.

Therefore, there is a conversion problem in the code logic.

5. Locate the fault

From the business logic, you can guess that there is a problem converting byte[] from the business to string.

New string() does not specify a character set:

Therefore, the system variable file.encoding is read as the character set by default during conversion.

Then we use arthas to look directly at the system variable, and sure enough, it’s not UTF8.

So, there are two solutions.

The first is to specify the character set **** when new String (bytes).

The second is to set the system variable file.encoding= UTF-8.

Step further

We initially chose to fix the code by specifying the character set when converting in code.

After republishing, I looked at arthas and found that it was still garbled? !!!!!

Then I went back and looked at the code for a long time. I couldn’t find the reason.

All of a sudden, I look at the offline page, and it’s all right. Nani? Is arthas the problem?

Then I googled arthas and found that a lot of people had problems with arthas displaying Chinese garble…

The solution is to start arthas and specify the character set as well.

java -jar -Dfile.encoding=UTF-8 arthas-boot.jar

And then the problem was solved… Hehe…

At this point arthas’s results were normal.

What does that tell us? !!!!!!!!!!

Arthas output interface must not specify character set for string conversion…

One foot stepped in a chain hole…

7. To summarize

In fact, the whole issue is rather superficial, but arthas’s garbled Chinese characters at the end are a bit painful…

Tcpdump, arthas, simpleHTTPServer, etc.

See the end, the original is not easy, point a concern, point a like it ~

Reorganize the knowledge fragments and construct the Java knowledge graph: github.com/saigu/JavaK… (Easy access to historical articles)