Systrace for Linux- Using Systrace to analyze Linux and Android scheduling Problems – scheduler – linux – debug categories: – scheduler thumbnail: blogexcerpt: The author’s daily kernel performance optimization work mainly involves terminal (Android), Server (Server), embedded (RTOS) and other scenarios. When kernel development and scheduling optimization are done in terminal scenarios, tools such as Atrace and Systrace are often used. Marveling at Google’s technical capabilities, I also wondered if these tools could be used in the server and embedded space. Use Systrace to capture sched, IRQ and frame information, frame information is certainly not available in our server and embedded field, but sched, IRQ and other information is also meaningful to the server field. If we could use Systrace in these scenarios, it would be significant for our performance tuning.

This work is licensed under creative Commons Attribution – Non-commercial Use – Same way Share 4.0 International license, please indicate the source of reproduction, thank you for your cooperation

Due to my limited technical level and knowledge, if there is any mistake or need to correct the content, welcome to correct, I thank you here

Please be sure to indicate the source of reprint, thank you, appreciate

1 Problem Source

The author’s daily kernel performance optimization work mainly involves terminal (Android), Server (Server), embedded (RTOS) and other scenarios. When kernel development and scheduling optimization are done in terminal scenarios, tools such as Atrace and Systrace are often used. Marveling at Google’s technical capabilities, I also wondered if these tools could be used in the server and embedded space.

Use Systrace to capture sched, IRQ and frame information, frame information is certainly not available in our server and embedded field, but sched, IRQ and other information is also meaningful to the server field. If we could use Systrace in these scenarios, it would be significant for our performance tuning.

Some information about Systrace is shown below:

Screen capture of Systrace report

2 Introduction to Systrace & Principles

There is a special documentation for Systrace (Android System Trace) on the official website of Android developers, or after entering the home page of the official website, Follow the Path of Android Developers > Android Studio > USER GUIDE > Command Line Tools > Systrace to access the document.

The principle of Systrace is not wordy here, you can search online:

Systrace calls Atrace to grab Android fTrace buffer data. Atrace will use the parameters specified by the user. Enable trace Events corresponding to the framework layer and kernel. 2. Systrace Combines the trace buffer data with prefix-. HTML, suffix.html, and systrace_trace_viewer. HTML Chrome (trace-viewer) directly parses trace.html files; 3. Use Chrome browser to open trace. HTML to view and analyze trace data in a graphical form. The script trace-viewer is running behind it;

In systrace, in addition to parsing the information of sched, IRQ and other native Trace events, frame information needs to be identified. Therefore, trace data stored in kernel mode and user mode can only be parsed by Systrace if it is output in a certain format.

1. No formatting is required for kernel trace information. Trace-viewer parses and draws data in the kernel’s native Trace format. Only a few types of data are supported, such as sched_switch and sched_wakeup for mapping CPU runs, and IRQ for mapping interrupts and soft interrupts by CPU.

2. User mode (APP/Java Framework/Native) uses Trace class to record Trace information. Thanks to the Trace Marker mechanism of FTrace, user mode is allowed to append data to the fTrace buffer of the kernel. User mode only need to use the/sys/kernel/debug/tracing/trace_marker “interface, can easily will be formatted trace data written to the kernel buffer. Therefore, Android Systrace conventions a series of trace formats. These trace classes are marked into the fTrace buffer according to the conventions, which can be parsed and drawn to the trace-viewer.

Systrace Trace Event format See systrace Trace Event format

3 systrace for Linux Desktop/Server

3.1 Introduction to Systrce for Linux

After analyzing systrace, we know that if we want to use Systrace on the server, it is ok. Because Chrome supports parsing native kernel sched, IRQ trace, so we directly use these to analyze scheduling performance, basically meet the requirements. Do I need to append the trace_view header to parse?

I initially thought this was necessary, but later I tried it and threw the native Trace buffer to Chrome ://tracing. The systrace header contains tags that tell Chrome that the HTML is actually a Systrace file. Use trace-viewer to parse.

So what we need to do now is to open the trace event we need and dump the Trace buffer after the test is completed. We can use The Google Systrace tool to complete this task. Of course, we can also choose to write some scripts manually to help us work. I found this on Github (thanks to the work of YOUNG HO CHA (aka Ganachoco)/ Ganadist, who was our first mover, without whom I would have taken a lot of wrong turns) and made some improvements, I published the final product on Systrace for Linux Desktop/Server and recommended it to my colleagues, using this tool to provide us with some tuning inputs for MYSQL and other scenarios.

Github address:

github https://github.com/gatieme/systrace
Copy the code

The first version of the relatively simple, later will be constantly updated and optimized.

3.2 systrace. Py

The systrace. Py script is relatively simple. It is the script we provide to grab trace buffers.

For example, you can use the following command to capture trace information about the current scheduling and interruption within the first 1s

# -t 1grab1S # -v indicates the output details # -e"sched,irq"Fetch scheduling and interrupt information python systrace.py -t1 -v -e "sched,irq"
Copy the code

Open the captured trace file in Chrome :// Tracing, as shown below.

3.3 Using the fix_time Script

Systrace can parse and graphically display limited information, so in the actual development process, we often manually open some additional Tracepoints, systrace can not be graphically displayed, but can help us analyze the problem. Generally, after we find or find some problems on the Systrace map, we open the Systrace log to find the time when the problem occurred, and then analyze the detailed trace to find out the cause. However, there is some inconvenience here. The time displayed on Chrome Systrace is relative time based on the point of capture (0MS), while the time in the trace log is the absolute time of the system.

Trace-viewer displays the relative time stamps in the log from the point at which the capture began

But the trace log text displays the system’s time (absolute time) stamp

So every time we try to find the time point of the problem, we need to add the time point of the capture, and then find the actual system time point of the problem. This is always inconvenient every time. So we provide a Perl script fix_time.pl to automate this work.

3.4 Capturing logs using scripts

As mentioned earlier, the original trace log can also be parsed directly by trace_view.

But be careful

The original trace file is a systrace format log file without HTML header tags. Therefore, it cannot be saved as an HTML file. Double-click the trace file to open it in Chrome.

The simplest script to grab trace is shown below

#! /bin/bash TRACING_PATH=/sys/kernel/debug/tracingget_systrace()
{
        local sleepTime=$1

        rm -rf trace.html
        echo > $TRACING_PATH/trace

        echo 1 > $TRACING_PATH/events/sched/sched_wakeup/enable
        echo 1 > $TRACING_PATH/events/sched/sched_wakeup_new/enable
        echo 1 > $TRACING_PATH/events/sched/sched_switch/enable
        echo 1 > $TRACING_PATH/events/irq/enable

        echo 1 > $TRACING_PATH/tracing_on

        sleep $sleepTime

        echo 0 > $TRACING_PATH/tracing_on
        echo 0 > $TRACING_PATH/events/enable

        cat $TRACING_PATH/trace > trace.html
        echo > $TRACING_PATH/trace
}

get_systrace $1
Copy the code

The script is also integrated into the Github repository. You can run the following command to capture the log information of 10S.

sh ./systrace.sh 10
Copy the code

4 Reference Materials

The tools will be tools (three) systrace (1) official website translation systrace analysis

4.1C: Using the Systrace and dumpsys tools

Ftrace implementation principle and development practice

• This work/blog (AderStep- Purple Night Appendix – Qingling Lane Grass Copyright ©2013-2017), by Cheng Jian (Gatieme).

• usingCreative Commons Attribution – Non-commercial Use – Same way Share 4.0 International LicenseWelcome to reprint, use and re-publish the article, but be sure to keep the bylineAs thou dost gatieme(Includes links to:blog.csdn.net/gatieme) shall not be used for commercial purposes.

• Any work modified based on this article must be distributed under the same license. If you have any questions, please contact me.

• Please be sure to indicate the source of reprint, thank you, appreciate