Related articles

Performance analysis Flame chart (2) – Principle of flame chart generation

Performance analysis Flame Chart (3) – Performance optimization for actual combat

Earlier with master zhang (zhang master of digging holes) have studied something flame figure, a great help in the later work and practice, therefore, the flame figure series comb, summarize and share in the company, will share the output document to summarize out now, there are some information is online many predecessors before some, if the same, is purely for reference

This series mainly introduces what is the flame map, introduces the principle of SystemMap, and the principle of flame map, how to generate flame map, and how to use flame map to locate the problem.

We can recall how to locate the performance problem in our current service. If most of our interface requests take longer and the service performance decreases, but the dependent related services all report that their service is normal, what will we do?

If experienced colleagues this time, they may be from GC, log, and a request for clues on the link, such as under the dump memory snapshot, thread stack, and query multiple requests interface before judging the common abnormal points, but if there is no experience may could not a classmate, no ideas (such as when I first hired)

As the saying goes, “if you want to do a good job, you must sharpen your tools first.” As a developer of services and code writers, it is difficult for us to do a good job without brushes and diamonds. There are many performance monitoring and tuning tools in Linux, and the flame chart is based on the PERF tool to more intuitively reflect the data collected.

Since we are primarily analyzing the FIRE map of the CPU, let’s start by reviewing some CPU knowledge


C P U Context switch and o n – c p u , o f f – c p u CPU context switch and on-CPU, off-CPU 

We all know that our CPU at run time, because of the multitasking "parallel" will be a lot of CPU time slice of switching, that is to say, the CPU may perform this task for a while, and then perform another task after a while, because each task is different, so at the time of task switching is bound to the thread context switchingCopy the code

1. CPU context: The environment on which the CPU must depend before performing a task

  • CPU registers
  • Program counter

2. The type of context switch

  • Process context switch (tens of nanoseconds to subtle CPU time per process context switch)

    • Saves the kernel stack and CPU registers for the current process
    • Holds the virtual memory and stack of the current process
    • Load the kernel stack and registers for the new process
    • Flush the virtual memory and user stack of the process

  • Thread context switch

  • Interrupt context switch (responding to hardware time, the interrupt takes care of the interrupt process’s normal scheduling and execution, and instead invokes interrupt handlers in response to device events)

    In the process of system call, there is a context switch, and there are two context switches (save the user’s instruction position, save the register value), once the user mode switch to the kernel mode, once the kernel mode switch back to the user mode

3. Analysis process CPU usage:

  • The kernel uses the ready queue to maintain all processes in the runnable state, runnable state does not include waiting for I/O, hibernation, etc.

  • The process scheduler is responsible for selecting runnable processes from the ready queue to execute.

  • However, all processes that are not in runnable state do not occupy CPU resources. These processes are waiting to be awakened by related events such as network IO. After waking up, the process changes its state to runnable state and joins the ready queue before being selected to execute by the scheduler algorithm.

    Thus, processes appear to exist throughout the life of a process, but not all of the time, depending on whether the process is on CPU or off CPU.

    When a process is deprived of execution right due to various reasons (preempting by other processes, initiating sleep system call by itself, waiting for network IO, etc.), it will first call deactivate_task function to delete it from the ready queue. The context_switch function is then called to switch the context of the process. This time, the old process loses CPU execution rights and enters the off CPU time

In fact, the flame chart is to analyze on CPU and off CPU time, if on CPU or off CPU time is too long, to see why the process has been holding the CPU, or the process has not been able to get the CPU

4. Do a little experiment: Watch for CPU context switches

We want to understand the principle of fire map generation, first of all, we need to know how to find the CPU context switch timing, with a small example to demonstrate next

  1. Fake_make. c:
#include <pthread.h> #include <stdio.h> #include <stdlib.h> #include <getopt.h> int get_job_num(int argc, char *const *argv); void *busy(void *args) { while (1); } int main(int argc, char *argv[]) { int num = get_job_num(argc, argv); printf("job num: %d\n", num); pthread_t threads[num]; int i; for (i = 0; i < num; ++i) { pthread_create(&threads[i], NULL, busy, NULL); } for (i = 0; i < num; ++i) { pthread_join(threads[i], NULL); } return 0; } int get_job_num(int argc, char *const *argv) { if (argc <= 1) { printf("illegal args\nusage ./fake_make -j8\n"); exit(10); } int num; const char *optstring = "j:"; // there are three options -- ABC, where c must be followed by a colon, so it must be followed by int ret; ret = getopt(argc, argv, optstring); if (ret ! = 'j') { printf("illegal args\nusage ./fake_make -j8\n"); exit(1); } num = atoi(optarg); return num; }Copy the code

4.1 Systemtap Observing CPU context switchover

Systemtap provides a probe and functions built-in library tapSAP, we can write systemTap scripts reuse them, similar to C language library functions, we can use it already written functions and functions.

To observe CPU context switches, in order to find when tasks are switched on the CPU, SystemTap’s built-in library STAP has probes related to the scheduler, including one for CPU_OFF (the CPU being replaced).

Reference: github.com/jav/systemt…

/** * probe scheduler.cpu_off - Process is about to stop running on a cpu * * @name: Name of the probe point * @task_prev: The process leaving the CPU (same as current) * @task_next: Replacing current * The Process replacing current * @idle: Where is it that the CPU is already idle? Where is it that the CPU is already idle? The process leaving the cpu. * */ probe scheduler.cpu_off = kernel.trace("sched_switch") ! , kernel.function("context_switch") { name = "cpu_off" task_prev = $prev task_next = $next idle = __is_idle() }Copy the code

Referring to SystemTap’s STAP, we customize a probe

global csw_count
global idle_count
global csw_total

probe scheduler.cpu_off {
      csw_count[task_prev, task_next]++
      csw_total+=1
      idle_count+=idle
}

function fmt_task(task_prev, task_next) {
   return sprintf("tid(%d)->tid(%d)", task_tid(task_prev), task_tid(task_next))
}

function print_cswtop () {
  printf ("%45s %10s\n", "Context switch", "COUNT")
  foreach ([task_prev, task_next] in csw_count- limit 5) {
    printf("%45s %10d\n", fmt_task(task_prev, task_next), csw_count[task_prev, task_next])
  }
  printf("%45s %10d\n", "csw_total", csw_total)
  printf("%45s %10d\n", "idle", idle_count)

  delete csw_total
  delete csw_count
  delete idle_count
}

probe timer.s($1) {
  print_cswtop ()
  printf("--------------------------------------------------------------\n")
}
Copy the code

  • GCC fake_make. C -o fake_make

  • Execute fake_make./fake_make -j4

  • Systemtap Displays the number of context switches stap cswstap.stap 1

In addition, there are other tools in Linux that can observe CPU context switching

  • The vmstat observationvmstat pid
  • Pidstat observationpidstat -w -p pid