First, background

I have used tcpdump and Wireshark in the past, and the delay detection tool TCprstat is available online. However, I always felt that I was too poor. Every time I captured packets or looked at the response time, I could not match the correct number, so I could not locate problems properly.

I know that I found this improved TCPRSTAT by chance, which is a great treasure, and the effect is proved to be good, so I write this blog.

2. Tool Description

2.1 tcprstat

Tcprstat is a free, open source TCP analysis tool that can be used to observe network traffic status and calculate latency between request and Response. It allows you to get the reply time statistics and display them as output. The output format is similar to that of the various -stat tools in Unix, such as vmstat, iostat, and mpstat.

The tool can optionally observe traffic load on a given port, which makes it useful for measuring request-Response time against single-instance Daemons, such as mysqld, HTTPD, memcached, etc.

(But tcPRstat source has not how to maintain, we can refer to my github project according to y123456Yz improved version of the arrangement, for tcPRstat installation and use.)

Y123456yz added the timestamp function to record the time when the response time exceeds the threshold, which is very helpful for packet analysis.

2.2 tcpdump

Tcpdump is the most effective tool for capturing network database packages in Unix and Linux.

Tcpdump intercepts the header of packets sent over the network for analysis. It supports filtering by network layer, protocol, host, network or port, and provides logical statements such as AND, OR, or not to help you get rid of useless information. In addition, you can use Wireshark and Java code to further filter and analyze the files that can be imported by tcpdump.

There are many tutorials on the Internet, I will not give an example.

2.3 Wireshark

Wireshark (formerly known as Ethereal) is a network packet analysis software. The function of network packet analysis software is to capture network packets and display the most detailed network packet information as possible. The Wireshark uses the WinPCAP interface to exchange data packets with network adapters.

Wireshark is a powerful network packet analysis tool. It helps you analyze packets captured using tcpdump on Linux and Windows, and locate network problems quickly and effectively.

Three, the use of steps

According to the above introduction, it can be known that:

  • Tcprstat can obtain the timestamp of the packet with the longest response time
  • Tcpdump captures network data packets of a service on the server
  • The Wireshark can visually analyze network packets captured by the tcpdump

Learn how to use the three tools by performing the following operations.

3.1 Packet Capture Using tcpdump

Capture all network data packets of Redis service on port XXX of eth0 and save them to the xxx.pcap file.

tcpdump -i eth0 port xxx -w xxx.pcap
Copy the code

Because tcpdump captures binary files, we can’t open them directly to view them. (Although tcpdump -r can also be used for viewing, but the format and content are relatively simple, which is not conducive to our analysis)

Note: To analyze the data at the same time, it is necessary to open two terminals and run tcpdump and TCPRstat at the same time.

3.2 Use TCPRstat to record delay detection

  • -l Indicates statistics on xx.xx.xx.xx server
  • -p indicates that the statistics port is XXX
  • -t 1 Indicates the statistics collection every 1s, that is, the printing interval of each line
  • -t 1 Records the requests whose latency exceeds 1ms
  • -o tcprstat.log records the time stamps of packets whose latency exceeds 1ms set by -t to the file and the response time of the packet. For details, see tcprstat.log later
[root@xxx delay_test]# tcprstat -l xx.xx.xx.xx -p xxx -n 1000 -t 1 -T 1 -o tcprstat.log

timestamp       count   max     min     avg     med     stddev  tc      95_max  95_avg  95_std  99_max  99_avg  99_std
1619774295      40      16451   7       532     37      2555    1       704     107     150     760     124     180
1619774296      29      140866  2       4942    69      25687   1       325     79      90      329     88      99
1619774297      16      31528   1       2203    228     7578    2       1284    248     339     1284    248     339
1619774298      28      40989   5       1556    62      7590    1       266     72      71      714     96      140
1619774299      31      290226  9       9537    62      51247   1       804     157     211     862     181     243
1619774300      50      445072  1       8984    21      62298   1       502     62      109     672     84      150
1619774301      35      281037  1       8144    24      46801   2       970     85      190     1188    118     264
1619774302      46      192983  8       4308    49      28126   2       351     77      82      1236    115     205
1619774303      64      216234  5       3477    33      26805   2       446     69      96      1145    100     179
1619774304      41      323947  3       8180    30      49947   2       144     42      42      9195    286     1430
1619774305      18      213248  3       12633   13      48753   2       13485   832     3163    13485   832     3163
1619774306      16      514698  0       32295   17      124556  2       1292    135     320     1292    135     320
1619774307      37      28758   3       854     58      4651    1       246     68      61      454     79      87
1619774308      24      46614   2       2061    95      9291    1       507     107     126     509     124     149
1619774309      40      225787  2       5786    31      35230   2       698     89      138     2290    145     373
Copy the code

Tcprstat. log records the following information:

[root @ XXX delay_test] # cat tcprstat. Log timestamp: 1619774294.779693 delay_time: 16451 timestamp: 1619774295.775643 Delay_time: 140866 timestamp: 1619774296.676406 delay_time: 1284 timestamp: 1619774296.771553 delay_time: 31528 Timestamp: 1619774297.769542 delay_time: 40989 timestamp: 1619774298.767789 delay_time: 290226 timestamp: 1619774299.765110 Delay_time: 445072 timestamp: 1619774300.761624 delay_time: 281037 timestamp: 1619774301.65521 delay_time: 1188 Timestamp: 1619774301.513685 delay_time: 1236 timestamp: 1619774301.757251 delay_time: 192983 timestamp: 1619774302.591191 Delay_time: 1145 timestamp: 1619774302.752262 delay_time: 216234 timestamp: 1619774303.749226 delay_time: 323947 Timestamp: 1619774304.73682 delay_time: 9195 timestamp: 1619774304.745029 delay_time: 213248 timestamp: 1619774304.959391 Delay_time: 13485 timestamp: 1619774305.740336 delay_time: 514698 timestamp: 1619774306.257507 delay_time: 1292 Timestamp: 1619774306.735689 delay_time: 28758 timestamp: 1619774307.731100 delay_time: 46614 timestamp: 1619774308.727788 Delay_time: 225787 timestamp: 1619774309.242648 delay_time: 2290 timestamp: 1619774309.722615 delay_time: 48644Copy the code

3.3 Using The Wireshark to Analyze Packages

After you upload the packet xxx.pcap captured by tcpdump to your PC and open it using Wireshark, the following information can be displayed:

However, the amount of data is too much, artificial search is not realistic. The tcprstat.log file tells us when the response latency is high, so we can search the timestamp directly to locate the specific response data.

For example, the tcprstat.log file has the highest response time

Timestamp: 1619774305.740336 delay_time: 514698Copy the code

The time stamp is 1619774305.740336 and the time is 514698us.

To search for the Wireshark, press CTRL + F or click Magnifying Glass.

As you can see, the command executed by this timestamp is a REPLCONF ACK command.

Four,

Through the above steps, we can quickly find the longest command is REPLCONF ACK, and then combined with the actual situation, we can better analyze the cause of the slow response of Redis.

Improved TCprstat github.com/Damanchen/t…