series
- Google Breakpad
- Google Breakpad
preface
In the previous section, we learned that Breakpad consists of three main parts:
- When a Client crashes, the minidump file is generated by default.
- Symbol Dumper, a tool used to generate breakpad-specific Symbol tables, works in the original library with debug information.
- Processor, this tool reads the Minidump file generated by Client, matches the corresponding Symbol table generated by Symbol Dumper, and finally generates human-readable C/C++ stack trace.
In this section, we will start with the Client part of Breakpad. The Client part is to learn how Breakpad listens for crashes and generates minidump files. So we choose the Android platform for analysis here, Android is based on the Linux system.
Breakpad Client usage
To integrate Breakpad Client with Android, follow this document, which is relatively simple.
After integration, we can get the minidump file generated after the crash with the following code:
#include "client/linux/handler/exception_handler.h"
static bool dumpCallback(const google_breakpad::MinidumpDescriptor& descriptor,
void* context, bool succeeded) {
printf("Dump path: %s\n", descriptor.path());
return succeeded;
}
void crash(a) { volatile int* a = (int(*)NULL); *a = 1; }
int main(int argc, char* argv[]) {
google_breakpad::MinidumpDescriptor descriptor("/tmp");
google_breakpad::ExceptionHandler eh(descriptor, NULL, dumpCallback, NULL.true.- 1);
crash(a);return 0;
}
Copy the codeThe source code parsing
The Breakpad Client process consists of two parts: signal registration and Minidump generation.
Signal registration
ExceptionHandler is the entry file to the entire process, so we start our analysis with this file.
// Runs before crashing: normal context.
ExceptionHandler::ExceptionHandler(const MinidumpDescriptor& descriptor,
FilterCallback filter,
MinidumpCallback callback,
void* callback_context,
bool install_handler,
const int server_fd)
: filter_(filter),
callback_(callback),
callback_context_(callback_context),
minidump_descriptor_(descriptor),
crash_handler_(NULL) {
// ...
if (install_handler) {
// ...
// Register signal processing
InstallHandlersLocked(a); }// ...
}
Copy the codeInstallHandlersLocked methods save the old signal processing callbacks and then re-register the new ones:
// Runs before crashing: normal context.
// static
bool ExceptionHandler::InstallHandlersLocked(a) {
if (handlers_installed)
return false;
// ...
struct sigaction sa;
memset(&sa, 0.sizeof(sa));
sigemptyset(&sa.sa_mask);
// ...
sa.sa_sigaction = SignalHandler;
sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
// Register signals to listen on
for (int i = 0; i < kNumHandledSignals; ++i) {
if (sigaction(kExceptionSignals[i], &sa, NULL) = =- 1) {
// At this point it is impractical to back out changes, and so failure to
// install a signal is intentionally ignored.
}
}
handlers_installed = true;
return true;
}
Copy the codeHere is a list of signals to register:
// The list of signals which we consider to be crashes. The default action for
// all these signals must be Core (see man 7 signal) because we rethrow the
// signal after handling it and expect that it'll be fatal.
const int kExceptionSignals[] = {
SIGSEGV, SIGABRT, SIGFPE, SIGILL, SIGBUS, SIGTRAP
};
Copy the codeThe signal processing
The SignalHandler method is called when a particular signal occurs. SignalHandler is the callback method passed in when the signal is registered.
// This function runs in a compromised context: see the top of the file.
// Runs on the crashing thread.
// static
void ExceptionHandler::SignalHandler(int sig, siginfo_t* info, void* uc) {
// ...
bool handled = false;
for (int i = g_handler_stack_->size() - 1; ! handled && i >=0; --i) {
handled = (*g_handler_stack_)[i]->HandleSignal(sig, info, uc);
}
// Upon returning from this signal handler, sig will become unmasked and then
// it will be retriggered. If one of the ExceptionHandlers handled it
// successfully, restore the default handler. Otherwise, restore the
// previously installed handler. Then, when the signal is retriggered, it will
// be delivered to the appropriate handler.
if (handled) {
InstallDefaultHandler(sig);
} else {
RestoreHandlersLocked(a); }// ...
}
Copy the codeG_handler_stack_ stores the ExceptionHandler instance passed in when we registered the signal. The same process can have multiple ExceptionHandler instances, so we need to iterate through the HandleSignal callback method here.
The GenerateDump method is called in HandleSignal:
// This function runs in a compromised context: see the top of the file.
// Runs on the crashing thread.
bool ExceptionHandler::HandleSignal(int /*sig*/.siginfo_t* info, void* uc) {
// ...
return GenerateDump(&g_crash_context_);
}
// This function may run in a compromised context: see the top of the file.
bool ExceptionHandler::GenerateDump(CrashContext* context) {
// ...
Sys_clone Creates the child process
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL.NULL.NULL);
if (child == - 1) {
sys_close(fdes[0]);
sys_close(fdes[1]);
return false;
}
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0.0.0);
SendContinueSignalToChild(a);int status = 0;
// Wait for the child process to finish executing
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
// ...
boolsuccess = r ! =- 1 && WIFEXITED(status) && WEXITSTATUS(status) == 0;
if (callback_)
success = callback_(minidump_descriptor_, callback_context_, success);
return success;
}
Copy the codeIn the GenerateDump method, sys_clone is first called to create a child process to generate the minidump file. The reason for creating a child process is that there are many limitations on the process where the exception occurred. For example, it may not be able to allocate enough memory. The exception may be caused by insufficient memory.
The child process’s entry method is ThreadEntry:
// This is the entry function for the cloned process. We are in a compromised
// context here: see the top of the file.
// static
int ExceptionHandler::ThreadEntry(void* arg) {
const ThreadArgument* thread_arg = reinterpret_cast<ThreadArgument*>(arg);
// Close the write end of the pipe. This allows us to fail if the parent dies
// while waiting for the continue signal.
sys_close(thread_arg->handler->fdes[1]);
// Block here until the crashing process unblocks us when
// we're allowed to use ptrace
thread_arg->handler->WaitForContinueSignal(a);sys_close(thread_arg->handler->fdes[0]);
return thread_arg->handler->DoDump(thread_arg->pid, thread_arg->context,
thread_arg->context_size) == false;
}
// This function runs in a compromised context: see the top of the file.
// Runs on the cloned process.
void ExceptionHandler::WaitForContinueSignal(a) {
int r;
char receivedMessage;
r = HANDLE_EINTR(sys_read(fdes[0], &receivedMessage, sizeof(char)));
// ...
}
Copy the codeIn ThreadEntry method, the DoDump method is not called directly to generate minidump file, but WaitForContinueSignal method is called to block waiting for write.
In the main process, after the sys_clone call is successful, the current process is set to allow ptrace:
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0.0.0);
SendContinueSignalToChild(a);// This function runs in a compromised context: see the top of the file.
void ExceptionHandler::SendContinueSignalToChild(a) {
static const char okToContinueMessage = 'a';
int r;
r = HANDLE_EINTR(sys_write(fdes[1], &okToContinueMessage, sizeof(char)));
// ...
}
Copy the codeThen, call SendContinueSignalToChild notify the child to continue.
Fdes [0] is used for reading and fDES [1] is used for writing. The reason for setting pTrace is that all threads in the process need to be paused while minidump file is generated. We’ll talk about that later.
The Minidump file is generated
DoDump method will be called MinidumpWriter: : WriteMinidumpImpl method to start generating minidump file.
bool WriteMinidumpImpl(const char* minidump_path,
int minidump_fd,
off_t minidump_size_limit,
pid_t crashing_process,
const void* blob, size_t blob_size,
const MappingList& mappings,
const AppMemoryList& appmem,
bool skip_stacks_if_mapping_unreferenced,
uintptr_t principal_mapping_address,
bool sanitize_stacks) {
LinuxPtraceDumper dumper(crashing_process);
// ...
MinidumpWriter writer(minidump_path, minidump_fd, context, mappings, appmem, skip_stacks_if_mapping_unreferenced, principal_mapping_address, sanitize_stacks, &dumper);
// Set desired limit for file size of minidump (-1 means no limit).
writer.set_minidump_size_limit(minidump_size_limit);
if(! writer.Init())
return false;
return writer.Dump(a); }Copy the codeLinuxDumper is used to obtain the current system information, such as thread information, loaded library information and memory information, etc. LinuxPtraceDumper is the implementation class of LinuxDumper, which represents the implementation based on PTrace. It is mainly the implementation of the following two methods:
// Suspend/resume all threads in the given process.
// Suspend all threads
virtual bool ThreadsSuspend(a) = 0;
// Restore all threads
virtual bool ThreadsResume(a) = 0;
Copy the codeMinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter MinidumpWriter
bool Init(a) {
// First call the initialization of LinuxDumper
if(! dumper_->Init())
return false;
ThreadsSuspend is then called to suspend all threads
if(! dumper_->ThreadsSuspend() | |! dumper_->LateInit())
return false;
// ...
return true;
}
bool LinuxDumper::Init(a) {
return ReadAuxv() && EnumerateThreads() && EnumerateMappings(a); }Copy the codeIn the linuxDumper. Init method, three methods are called:
-
ReadAuxv
Read the /proc/{pid}/auxv file to get some auxiliary information.
-
EnumerateThreads
Read /proc/{pid}/task to obtain thread information in the current process.
-
EnumerateMappings
Read the /proc/{pid}/maps file to obtain information about the memory mapping file loaded in the current process.
After initialization, call Dump to generate the minidump file:
bool Dump(a) {
// A minidump file contains a number of tagged streams. This is the number
// of stream which we write.
unsigned kNumWriters = 13;
TypedMDRVA<MDRawDirectory> dir(&minidump_writer_);
{
// Ensure the header gets flushed, as that happens in the destructor.
// If a crash occurs somewhere below, at least the header will be
// intact.
TypedMDRVA<MDRawHeader> header(&minidump_writer_);
// ...
header.get()->signature = MD_HEADER_SIGNATURE;
header.get()->version = MD_HEADER_VERSION;
header.get()->time_date_stamp = time(NULL);
header.get()->stream_count = kNumWriters;
header.get()->stream_directory_rva = dir.position(a); }unsigned dir_index = 0;
MDRawDirectory dirent;
if (!WriteThreadListStream(&dirent))
return false;
dir.CopyIndex(dir_index++, &dirent);
if (!WriteMappings(&dirent))
return false;
dir.CopyIndex(dir_index++, &dirent);
if (!WriteAppMemory())
return false;
if (!WriteMemoryListStream(&dirent))
return false;
dir.CopyIndex(dir_index++, &dirent);
if (!WriteExceptionStream(&dirent))
return false;
dir.CopyIndex(dir_index++, &dirent);
if (!WriteSystemInfoStream(&dirent))
return false;
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_CPU_INFO;
if (!WriteFile(&dirent.location, "/proc/cpuinfo"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_PROC_STATUS;
if (!WriteProcFile(&dirent.location, GetCrashThread(), "status"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_LSB_RELEASE;
if (!WriteFile(&dirent.location, "/etc/lsb-release"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_CMD_LINE;
if (!WriteProcFile(&dirent.location, GetCrashThread(), "cmdline"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_ENVIRON;
if (!WriteProcFile(&dirent.location, GetCrashThread(), "environ"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_AUXV;
if (!WriteProcFile(&dirent.location, GetCrashThread(), "auxv"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_MAPS;
if (!WriteProcFile(&dirent.location, GetCrashThread(), "maps"))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
dirent.stream_type = MD_LINUX_DSO_DEBUG;
if (!WriteDSODebugStream(&dirent))
NullifyDirectoryEntry(&dirent);
dir.CopyIndex(dir_index++, &dirent);
// If you add more directory entries, don't forget to update kNumWriters,
// above.
dumper_->ThreadsResume(a);return true;
}
Copy the codeThis method looks long, but is actually not that complicated. First, it generates the header information of the Minidump file, such as signature, version, timestamp, Stream count, and so on, represented by MDRawHeader.
The rest of the content is composed of 13 streams, such as the WriteThreadListStream used to write all thread information, the WriteMappings used to write all memory maps, and so on, along with memory information, system information, and so on.
The type of each Stream is distinguished by stream_type. Here is the current defined enumeration of stream_type:
/* For (MDRawDirectory).stream_type */
typedef enum {
MD_UNUSED_STREAM = 0,
MD_RESERVED_STREAM_0 = 1,
MD_RESERVED_STREAM_1 = 2,
MD_THREAD_LIST_STREAM = 3./* MDRawThreadList */
MD_MODULE_LIST_STREAM = 4./* MDRawModuleList */
MD_MEMORY_LIST_STREAM = 5./* MDRawMemoryList */
MD_EXCEPTION_STREAM = 6./* MDRawExceptionStream */
MD_SYSTEM_INFO_STREAM = 7./* MDRawSystemInfo */
MD_THREAD_EX_LIST_STREAM = 8,
MD_MEMORY_64_LIST_STREAM = 9,
MD_COMMENT_STREAM_A = 10,
MD_COMMENT_STREAM_W = 11,
MD_HANDLE_DATA_STREAM = 12,
MD_FUNCTION_TABLE_STREAM = 13,
MD_UNLOADED_MODULE_LIST_STREAM = 14,
MD_MISC_INFO_STREAM = 15./* MDRawMiscInfo */
MD_MEMORY_INFO_LIST_STREAM = 16./* MDRawMemoryInfoList */
MD_THREAD_INFO_LIST_STREAM = 17,
MD_HANDLE_OPERATION_LIST_STREAM = 18,
MD_TOKEN_STREAM = 19,
MD_JAVASCRIPT_DATA_STREAM = 20,
MD_SYSTEM_MEMORY_INFO_STREAM = 21,
MD_PROCESS_VM_COUNTERS_STREAM = 22,
MD_LAST_RESERVED_STREAM = 0x0000ffff./* Breakpad extension types. 0x4767 = "Gg" */
MD_BREAKPAD_INFO_STREAM = 0x47670001./* MDRawBreakpadInfo */
MD_ASSERTION_INFO_STREAM = 0x47670002./* MDRawAssertionInfo */
/* These are additional minidump stream values which are specific to * the linux breakpad implementation. */
MD_LINUX_CPU_INFO = 0x47670003./* /proc/cpuinfo */
MD_LINUX_PROC_STATUS = 0x47670004./* /proc/$x/status */
MD_LINUX_LSB_RELEASE = 0x47670005./* /etc/lsb-release */
MD_LINUX_CMD_LINE = 0x47670006./* /proc/$x/cmdline */
MD_LINUX_ENVIRON = 0x47670007./* /proc/$x/environ */
MD_LINUX_AUXV = 0x47670008./* /proc/$x/auxv */
MD_LINUX_MAPS = 0x47670009./* /proc/$x/maps */
MD_LINUX_DSO_DEBUG = 0x4767000A./ * MDRawDebug 32 (} {* /
/* Crashpad extension types. 0x4350 = "CP" * See Crashpad's minidump/minidump_extensions.h. */
MD_CRASHPAD_INFO_STREAM = 0x43500001./* MDRawCrashpadInfo */
} MDStreamType; /* MINIDUMP_STREAM_TYPE */
Copy the codeEach Stream write is divided into two parts, the first is the Stream’s information:
dirent->stream_type = MD_THREAD_LIST_STREAM;
dirent->location = list.location(a);Copy the codeStream_type we have already mentioned that location is used to mark the location of the Stream content. Because the Stream information and the content are written separately, all the Stream information is written first, and then the Stream content is written successively.
typedef struct {
uint32_t data_size;
MDRVA rva;
} MDLocationDescriptor; /* MINIDUMP_LOCATION_DESCRIPTOR */
Copy the codeData_size in location represents the length of the content, and rva represents the location in the minidump file.
We can use Breakpad’s Minidump_dump tool to convert the original Minidump file into a human-readable format.
MDRawHeader signature = 0x504d444d version = 0xa793 stream_count = 13 stream_directory_rva = 0x20 checksum = 0x0 time_date_stamp = 0x61011550 2021-07-28 08:29:04 flags = 0x0 mDirectory[0] MDRawDirectory stream_type = 0x3 (MD_THREAD_LIST_STREAM) location.data_size = 11668 location.rva = 0xc0 mDirectory[1] MDRawDirectory stream_type = 0x4 (MD_MODULE_LIST_STREAM) location.data_size = 69880 location.rva = 0x10dCA8 the following content is similarCopy the codesummary
To sum up the Breakpad Client workflow, first register the corresponding signal callback, receive the corresponding signal callback when a crash occurs, then copy a child process, then set the current main process to allow Ptrace, and finally suspend all threads in the child process. The Minidump file is generated.
We summarize the process with a comment in the Breakpad code:
// The signal flow looks like this:
//
// SignalHandler (uses a global stack of ExceptionHandler objects to find
// | one to handle the signal. If the first rejects it, try
// | the second etc...)
// V
// HandleSignal ----------------------------| (clones a new process which
// | | shares an address space with
// (wait for cloned | the crashed process. This
// process) | allows us to ptrace the crashed
// | | process)
// V V
// (set signal handler to ThreadEntry (static function to bounce
// SIG_DFL and rethrow, | back into the object)
// killing the crashed |
// process) V
// DoDump (writes minidump)
// |
// V
// sys_exit
//
Copy the code