Audio and Video Development Journey (59)- Capture, collect, locate and analyze Native crashes

What are the types of Native crashes

Let’s first build a Native crash and look at the crash information of Native

Signal xx: indicates the error type. We can first determine which type of crash is based on the error type. Common Android Native crashes are as follows: SIGSEGV has the highest probability.

Next is the register snapshot, which is not directly visible, and the fault ADDR is a key information, which will be used in the subsequent analysis of location.

The next step is to call the stack, which is also very important, and can directly help us see the stack information of Crash, but it needs the symbol table so to convert the corresponding function name and line number, otherwise it is also difficult to understand.

How to catch and collect Native crashes

Common Native crash capture tools: Chromium BreakPad, Tencent Bugly Let’s learn and practice BreakPad to collect Natvie crash. Breakpad is a cross-platform open source project. In this section, we will learn how to compile and use it in practice.

2.1 Let’s take a look at how Breakpad works

Image from: learn this trick to make C++ crash no escape!

2.2 The installation process is as follows

Download [Breakpad] source (chromium.googlesource.com/breakpad/br…).
Download dePOT_tools
Breakpad relies on LSS, download it (github.com/adelshokhy1…) Save the linux_SYscall_support. h file in LSS to breakpad/ SRC /third_party/ LSS /.
Breakpad./configure && make && sudo make install

The /usr/local/bin/minidump_dump and /usr/local/bin/minidump_stackwalk command tools will be used later in the analysis

2.3 Integrate Breakpad into the Android project

Copy the SRC folder from the Google-Breakpad source code to the project’s SRC /main/ CPP directory. Configure cmake or makefile. Here we use cmake cmake_minimum_required(VERSION 3.4.1)

Set (BREAKPAD_ROOT ${CMAKE_CURRENT_SOURCE_DIR} ${BREAKPAD_ROOT} / SRC/common/android/include) # classified files to compile the CPP code file (GLOB BREAKPAD_SOURCES_COMMON ${BREAKPAD_ROOT}/src/client/linux/crash_generation/crash_generation_client.cc ${BREAKPAD_ROOT}/src/client/linux/dump_writer_common/thread_info.cc ${BREAKPAD_ROOT}/src/client/linux/dump_writer_common/ucontext_reader.cc ${BREAKPAD_ROOT}/src/client/linux/handler/exception_handler.cc ${BREAKPAD_ROOT}/src/client/linux/handler/minidump_descriptor.cc ${BREAKPAD_ROOT}/src/client/linux/log/log.cc ${BREAKPAD_ROOT}/src/client/linux/microdump_writer/microdump_writer.cc ${BREAKPAD_ROOT}/src/client/linux/minidump_writer/linux_dumper.cc ${BREAKPAD_ROOT}/src/client/linux/minidump_writer/linux_ptrace_dumper.cc ${BREAKPAD_ROOT}/src/client/linux/minidump_writer/minidump_writer.cc ${BREAKPAD_ROOT}/src/client/minidump_file_writer.cc  ${BREAKPAD_ROOT}/src/common/convert_UTF.c ${BREAKPAD_ROOT}/src/common/md5.cc ${BREAKPAD_ROOT}/src/common/string_conversion.cc ${BREAKPAD_ROOT}/src/common/linux/elfutils.cc ${BREAKPAD_ROOT}/src/common/linux/file_id.cc ${BREAKPAD_ROOT}/src/common/linux/guid_creator.cc ${BREAKPAD_ROOT}/src/common/linux/linux_libc_support.cc ${BREAKPAD_ROOT}/src/common/linux/memory_mapped_file.cc ${BREAKPAD_ROOT}/ SRC /common/ Linux /safe_readlink.cc ${BREAKPAD_ROOT}/src/common/android/breakpad_getcontext.S ) set_source_files_properties(${BREAKPAD_ASM_SOURCE} Add_library (breakpad STATIC ${BREAKPAD_SOURCES_COMMON} ${BREAKPAD_ASM_SOURCE}) # link target_link_libraries(breakpad log)Copy the code

2.4 Add breakpad callback

Uncaught exceptions in the Java layer can be handled by UncaughtExceptionHandler.

 google_breakpad::MinidumpDescriptor descriptor(path);
    static google_breakpad::ExceptionHandler eh(descriptor, NULL, DumpCallback, NULL, true, -1);
Copy the code

Details are as follows:

#include <stdio.h> #include <jni.h> #include "client/linux/handler/exception_handler.h" #include "client/linux/handler/minidump_descriptor.h" void onNativeCrash(const char* info) { JNIEnv *env = 0; jclass crashPinClass = env->FindClass( "com/test/crash/TestCrash"); if (crashPinClass == NULL){ return; } jmethodID crashReportMethod = env->GetStaticMethodID(crashPinClass, "onNativeCrash", "(Ljava/lang/String;) V"); if (crashReportMethod == NULL) { return; } jstring crashInfo = env->NewStringUTF(info); env->CallStaticVoidMethod(crashPinClass, crashReportMethod, crashInfo); } / / collapse callback bool DumpCallback (const google_breakpad: : MinidumpDescriptor & descriptor, void * context, Bool Succeeded) {onNativeCrash(""); return succeeded; } extern "C" JNIEXPORT void JNICALL Java_com_sample_breakpad_BreakpadInit_initBreakpadNative(JNIEnv *env, jclass type, jstring path_) { const char *path = env->GetStringUTFChars(path_, 0); google_breakpad::MinidumpDescriptor descriptor(path); static google_breakpad::ExceptionHandler eh(descriptor, NULL, DumpCallback, NULL, true, -1); env->ReleaseStringUTFChars(path_, path); }Copy the code

Then load so to set the storage path of the DMP file generated after the crash. How do we analyze crashes collected? In the following section we continue to learn about practice.

Iii. How to analyze and locate Native crash

Before looking at several common analysis tools, let’s look at the difference between compiling and producing so with and without a sign table.

We can use the file command to see the difference between them

file cmake/debug/obj/arm64-v8a/libcrash-lib.so ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=c776759b526457b5777fc8833f7fc0fcc46055cc, with debug_info, For time to view view Namely take the symbol table file transforms/stripDebugSymbol/debug / 0 / lib/arm64 v8a/libcrash - lib. So the ELF 64 - bit LSB Shared object, ARM aarch64, Version 1 (SYSV), dynamically linked, BuildID [sha1] = c776759b526457b5777fc8833f7fc0fcc46055cc, stripped - > stripped the debug information, There's no sign tableCopy the code

If it is so developed and compiled by ourselves, the SO with symbol table should be backed up or uploaded when it is released, so as to facilitate analysis and location of native crash. Special attention should be paid to that: the MD5 of SO printed by different machines is different, so the corresponding so with symbol table should be saved after the release (SO of different architectures in the OBj directory).

So let’s take a look at some of the tools that we use

3.1 minidumP_stackWalk Export crash stack information

This is the command tool we compiled in the previous section. Usage:

minidump_stackwalk fd311404-a968-4ce0-17d5fa8a-61a8fdf1.dmp libcrash-lib.so >crash.log
Copy the code

The generated crash.log file is as follows

CPU: arm64 8 CPUs GPU: UNKNOWN Crash reason: SIGSEGV /SEGV_MAPERR Crash address: 0x0 Process uptime: Not available Thread 0 (crashed) 0 libcrash-lib.so + 0x5e0 --> Error address x0 = 0x0000007b14AC4e00 x1 = 0x0000007fedDE9894 x2 =  0x0000000000000000 x3 = 0x0000007b14a56c00 x4 = 0x0000007feddeaa00 x5 = 0x0000007a7e1a7965 @ "crash.log" 2226L, 114492BCopy the code

Addr2line can convert the address to the corresponding function name and line number.

3.2 the addr2line

The basic usage is as follows

Usage: aarch64-linux-android-addr2line [option(s)] [addr(s)] Convert addresses into line number/file name pairs. If no addresses are specified on the command line, they will be read from stdin The options are: -e --exe=<executable> Set the input file name (default is A. out) -f --functions Show function namesCopy the code

The path of the addr2line command is as follows. You can select the addr2line based on the CPU architecture of the device in the crash information.

Arm: $NDK_PATH toolchains/arm - Linux - androideabi - 4.9 / prebuilt/Darwin - x86_64 / bin arm64: $NDK_PATH/toolchains/aarch64 - Linux - android 4.9 / prebuilt/Darwin - x86_64 / binCopy the code

Dump (arm64, 0x5e0); add2line (arm64)

/ Users/yangbin/Library/Android/Android - the NDK - r16b toolchains/aarch64 - Linux - Android - 4.9 / prebuilt/Darwin - x86_64 / bin/aarch64 - linux-android-addr2line -f -e /Users/yangbin/work/avwork/thirdparty/nativecrash/Chapter01/sample/build/intermediates/cmake/debug/obj/arm64-v8a/libcras h-lib.so 0x5e0 _Z5Crashv /Users/yangbin/work/avwork/thirdparty/nativecrash/Chapter01/sample/.externalNativeBuild/cmake/debug/arm64-v8a/.. /.. /.. /.. /src/main/cpp/crash.cpp:10Copy the code

It can be seen that the output of the corresponding error class and the number of lines, combined with the error cause SIGSEGV can quickly analyze the specific cause.

3.3 Script the preceding procedure

Sh./test/TMP/fd311404-a968-4ce0-17d5FA8a-61a8fdf1.dmp crash.log The script comes from: Learn this trick, let C++ crash nowhere to escape!

#! /bin/bash if [ $# != 3 ] ; then echo "USAGE: $0 TARGET_NAME DMP_NAME OUTPUT_NAME" echo " e.g.: $0 test fd311404-a968-4ce0-17d5fa8a-61a8fdf1.dmp crash.log" exit 1; GetStackTrace () {echo "@getStackTrace ": {echo "@getStackTrace" : {echo "@getstackTrace" : Sym_file_name =$target_file_name'. Sym '# $sym_file_name =' head -n1 OIFS=$IFS; IFS=" "; set -- $line1; aa=$1; bb=$2; cc=$3; dd=$4; IFS=$OIFS version_number=$dd Mkdir -p./symbols/$target_file_name/$version_number mv $sym_file_name /symbols/$target_file_name/$version_number # redirects stack trace information to file minidump_stackwalk $dmp_file_name./symbols > $output_file_name } main() { getSymbol if [ $? == 0 ] then getStackTrace fi }Copy the code

3.4 the NDK – stack

The NDK-Stack is also a very useful tool that needs to be analyzed in conjunction with a Tombstone file at crash time.

The ndK-stack is used as follows

usage: ndk-stack.py [-h] -sym SYMBOL_DIR [-i INPUT]

Symbolizes Android crashes.

optional arguments:
  -h, --help            show this help message and exit
  -sym SYMBOL_DIR, --sym SYMBOL_DIR
                        directory containing unstripped .so files
  -i INPUT, -dump INPUT, --dump INPUT
                        input filename

ndk-stack -sym $PROJECT_PATH/obj/local/armeabi-v7a -dump tombstone.txt
Copy the code

Tombstone files can be obtained via adb Bugreport. Let’s take a look at the process of taking down the tombstone file by using the adb bugreport command and combining it with the NDK-Stack analysis

Adb bugreport.unzip bugreport-oneplus5t-qkq1.191014.012-2021-11-28-14-49-22. zip CD FS/data/tombstones We take the most recent one to analyze the NDK-stack-SYM /Users/yangbin/work/avwork/thirdparty/nativecrash/Chapter01/sample/build/intermediates/cmake/debug/obj/arm64-v8a -dump tombstone_09Copy the code

3.5 IDA Pro

If there is no symbol table so, we can use IDA so reverse analysis tool to analyze location analysis. For example, we can use IDA to open libcrash-lib.so without symbol table and query the problem by error address

Open libcrash-lib.so with IDA and skip to address 0x5e0

Let’s use libcrash-lib.so without the symbol table

You can see that you can also locate the corresponding class. But are some assembly language, need to understand. Objdump, another tool, can also be used to find the corresponding assembly information, and further analysis.

This article basically here, more for two months, familiar with this face two months post changes, more important is the target realized suddenly relaxed, actually this is the starting point, through this two months of work to understand familiar, audio and video has a very wide range of knowledge and application really involved, decoding, rendering, transmission, agreement, player, graphics, AI, etc. Come on, the next article begins our entry into the FFMPEG source code parsing series. Try to write at least one essay a week and study together

Four, data

Crash Optimization (PART 1) : All that “crash” stuff
Crash capture mechanism and implementation of Android Native code
Learn this trick, let C++ crash nowhere to escape!
Android uses Google Breakpad for crash log management
Native crash log analysis method developed by Android NDK&JNI
Crash capture mechanism and implementation of exception handling – Native layer
Android NDK Tombstone/Crash Analysis
Android Native crash location
Android NDK tombstone/crash analysis
How to analyze and locate Android Native Crash
Dry goods | android APP crash – xCrash capture scheme

The corresponding open source project - "\ [[https://github.com/iqiyi/xCrash\]] (HTTP: / / https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2Fiqiyi%2FxCrash %255D)Copy the code

Bugly-android Native code crash capture mechanism and implementation
Blade Rider: Common Crashes and errors in Android Native

Five, the harvest

Through this study, we are familiar with how to capture and analyze native crashes. The summary is as follows:

The learning practice is to capture and collect native crashes through breakpad
Minidump_stackwalk is used to convert dump files generated by Breakpad into native crash information files, and then use add2line and the corresponding so with symbol table to resolve the crashed classes and the corresponding line number
It is practical to obtain tombstone file and analyze natvie crash stack with NDK_stack
So analysis of unsigned tables by IDA Pro is practiced

Thank you for reading

In the next article, we will enter the FFMPEG series again and study the source code level. Welcome to pay attention to the public account “audio and video development journey”, learn and grow together.

Welcome to communicate