What is an anomaly
Many books on operating systems will mention “modern operating systems are interrupt-driven software” when explaining the operation mechanism of operating systems.
Interrupt refers to the CPU’s response to an event that occurs in the system. The CPU pauses the program in progress, remains on the scene, and then executes the corresponding handler. After processing the event, the CPU returns to the breakpoint to continue executing the interrupted program.
The original intention of introducing interrupt technology is to improve the UTILIZATION rate of CPU in the multi-program running environment. For example, CPU can execute other instructions in the I/O execution process without idle waiting (or simple polling) for the I/O device to complete the execution. The I/O device completes the execution and then informs CPU through interrupt to improve the UTILIZATION rate of CPU. Interrupt technology gradually developed, became the operating system, the basis of the operations such as process scheduling, modern operating system process scheduling is usually the priority scheduling algorithm based on time slice, the CPU time is divided into very fine-grained time slice, perform a task ran out of time, the clock by giving notice to the CPU clock interrupt to switch tasks, For example, CPU exception handling, discussed below, is also based on the interrupt mechanism.
Interrupts and exceptions have different meanings in different CPU architectures.
- For example, in The Intel architecture, the entry of interrupt processing is defined by the Interrupt Dispatch table (IDT) in the operating system kernel. There are 255 interrupt vectors in IDT, among which the first 20 are defined as the entry of exception processing, that is, the interrupt contains exceptions.
- However, in THE ARM architecture, the entrance of interrupt processing is in the exception vector. Among the 8 exception vectors, 3 are interrupt-related, that is, exceptions contain interrupts.
Regardless of how interrupts and exceptions are defined, when an exception occurs, the CPU transfers control from the pre-exception program to the exception handler, and the CPU gains no lower execution power. For example, when an exception occurs in an application running in user mode, the CPU switches to kernel mode and executes the corresponding exception handler. The life cycle of an instruction in the classical CPU five-level pipeline is [retrieving finger, decoding, executing, accessing and writing back], and CPU anomalies may occur at each stage, such as in ARM architecture:
- Data abort exception during execution: if processorData access instructionA data abort exception occurs when the address does not exist or is not accessible by the current instruction.
- Prefetch abort exception during fetch phase: if processorPrefetching instructionsThe address does not exist, or the address is not allowed to be accessed by the current instruction, the memory will send an abort signal to the processor, but when the prefetch instruction is executed, the instruction prefetch abort exception will be generated.
The handlers for these two exceptions either directly or indirectly call the Mach kernel’s Exception_triage () function and pass EXC_BAD_ACCESS as an input. Exception_triage () will use Mach messaging to post exceptions. Although CPU exception handling is somewhat different between the Intel and ARM architectures, exception handlers either directly or indirectly pass exception_type_T to exception_triage() to mask the difference between different machine platforms.
Exception types (EXCEPtion_type_t) are stored in the Mach layer with an int variable. In the OSFMK/Mach/Exception_types.h file, you can see more than a dozen exceptions defined by the Mach layer, such as the common ones
#define EXC_BAD_ACCESS 1 /* Could not access memory */
/* Code contains kern_return_t describing error. */
/* Subcode contains bad memory address. */
#define EXC_CRASH 10 /* Abnormal process exit */
#define EXC_CORPSE_NOTIFY 13 /* Abnormal process exited to corpse state */
Copy the codeint main(int argc, const char * argv[]) {
int *pi = (int*)0x00001111;
*pi = 17;
return 0;
}
Copy the codeException_triage () : exception_triage() : exception_triage();
2. Debug and track CPU anomalies
Xnu exception handling is explained in Mac OS & iOS in depth, but it is not particularly detailed, and the reference code of the book is not the latest code, to understand the exception handling process of the kernel, you need to read, read the source code, of course, not break debugging.
2.1 debugging xnu
Debugging XNU on MacOS is easier than on iOS, using the following tools: LLDB + VMware Fusion + Kernel Debug Kit to set up a debugging environment, please refer to the MacOS Kernel Debugging Environment. Build your own iOS Kernel debugger is available on iOS 11.1.2. For details, see build your Own iOS Kernel Debugger
If the VIRTUAL machine reaches the “Wait for the Debugger” phase and successfully connects to the VIRTUAL machine on the host through “KDP -remote”, The virtual machine is stuck in “Waiting for link to become available”, causing debugging to fail, as described in this thread
I couldn’t find the exact cause of the problem, but I figured out a solution: Press Option, Command, P, and R at the same time to reset NVRAM, which will put the VM into recovery mode. Use the terminal tool to disable THE SIP on the VM by typing csrutil disable. Then restart, and then go through the process of “Kernel replacement” -> “Set boot-args” ->” clear kext cache “->” Restart vm “->” Host connect to VM “. There is a 70% chance that the VM can be started properly and debugged. If not, try again.
Note: THE MacOs version I used is 10.13.5, and the corresponding XNU is 4570.61.1. The source code of the corresponding version has not been released. Compared with the previous versions, the source code I need to refer to has not changed, so the reference source is xNU-4570.1.46 on Github
2.2 Tracing CPU Anomalies
int main(int argc, const char * argv[]) {
char c = getchar();
int *pi = (int*)0x00001111;
*pi = 17;
return 0;
}
Copy the codeFirst use GCC to compile the above program into a binary executable, then run it. While the program is waiting for keyboard input, you can use the ps command to check that the process PID is 352.
Before running the program I made three breakpoints at the implementation of the OSfmk /kern/exception.c exception_triage_thread() function
breakpoint set --file exception.c --line 447
breakpoint set --file exception.c --line 459
breakpoint set --file exception.c --line 472
Copy the code447, 459 and 472 are delivery exceptions to the abnormal port array of thread layer, Task layer and host layer respectively, corresponding to the following three lines of code
Kr = Exception_deliver (Thread, exception, code, codeCnt, thread->exc_actions, mutex); (459) kr = Exception_deliver (Thread, exception, code, codeCnt, task-> exc_Actions, mutex); (472) kr = Exception_DELIVER (Thread, exception, code, codeCnt, host_priv-> exc_Actions, mutex);Copy the codeOnly one of the three breakpoints is broken, and that’s line 472, where you can verify the following conclusion
First, the function call stack, thread status and process PID are output at the terminal through LLDB
(lldb) bt
* thread #1, stop reason = breakpoint 4.1
* frame # 0: 0xffffff800f97f0c9 kernel.development`exception_triage_thread(exception=1, code=0xffffff8014debf50, codeCnt=2, thread=0xffffff801c7c2a10) at exception.c:472 [opt]
frame # 1: 0xffffff800fad71fb kernel.development`user_trap [inlined] exception_triage(code=0x0000000000000001) at exception.c:504 [opt]
frame # 2: 0xffffff800fad71df kernel.development`user_trap [inlined] i386_exception(exc=1, code=
) at trap.c:1152 [opt]
frame # 3: 0xffffff800fad71d7 kernel.development`user_trap [inlined] user_page_fault_continue(kr=
) at trap.c:232 [opt]
frame #4: 0xffffff800fad71d1 kernel.development`user_trap(saved_state=0xffffff8017246b20) at trap.c:1093 [opt]
frame #5: 0xffffff800f921102 kernel.development`hndl_alltraps + 226
(lldb) e struct proc *$p_proc = (struct proc *)thread->task->bsd_info
(lldb) po $p_proc->p_pid
352
(lldb) po thread->state
4
Copy the code(Note: Thread state is stored in an int variable, int state,#define TH_SUSP 0x02
Copy the codeThe above log combined with the source code and in-depth analysis of Mac OS & iOS operating system can be concluded:
On the Intel architecture, when an exception occurs in the user-mode program, the CPU suspends the corresponding process, sets the CPU working state to kernel mode, and executes the exception handler of the XNU kernel. Instead of having a separate handler for each trap (exception), most operating systems have a single handler for all traps, which then performs different processing via switch() or jumps to different functions based on predefined tables. XNU does the same, hndl_alltraps are the common trap handler, user_trap handles the actual traps, hndl_alltraps are written in assembly language, and user_trap is written in C, The I386_exception function is called in the user_trap implementation. The I386_exception function calls Exception_triage to convert the trap to a Mach exception. In the above example the Mach exception is EXC_BAD_ACCESS.
The exception_triage() function is implemented in only two lines of code
kern_return_t
exception_triage(
exception_type_t exception,
mach_exception_data_t code,
mach_msg_type_number_t codeCnt)
{
thread_t thread = current_thread();
return exception_triage_thread(exception, code, codeCnt, thread);
}
Copy the codeThe first line obtains the current thread, because the current thread is needed when the second line calls Exception_triage_thread to deliver exceptions to the exception port. The array of exception ports of thread and task needs to be obtained through thread:
thread->exc_actions;
task = thread->task;
task->exc_actions;
host_priv = host_priv_self();
host_priv->exc_actions;
Copy the codeThe exception port of thread and task is NULL by default. The exception port of host is set when the first user-mode process (PID 1) is initialized. After the kernel initialization, all user-mode processes are sub-processes of launchd. The child process inherits the abnormal port of the parent process through the parent process fork. Therefore, all user-mode processes can be handled uniformly in the host layer when exceptions occur.
How does the launchd process set the host exception port? How to handle an exception message received?
During kernel initialization, the first user-mode process launchd is started in the bsdinit_task() function. Before starting the launchd process, the host_set_Exception_ports () function is called. Redirect all Mach exception messages to the port ux_Exception_port, which is held by a kernel thread that executes the ux_Handle () function, This function calls mach_MSg_receive () in an infinite loop to receive messages from the Ux_Exception_port port, and mach_MSg_receive () blocks the thread.
Mach_exc_server () is called when a Mach message is received in the ux_handle() function, and the handlers below are called for mach_exc_server. The specific call is determined by the Exception_behavior_t behavior parameter, which is passed in when the abnormal port is set by calling Host_set_Exception_ports ()
Catch_mach_exception_raise () equals EXCEPTION_DEFAULT 1, Xx catch_mach_EXCEPtion_RAISe_state () = define EXCEPTION_STATE 2, Catch_mach_exception_raise_state_identity () corresponds to define EXCEPTION_STATE_IDENTITY 3Copy the codeCatch_mach_exception_raise () These handles call ux_Exception () to convert Mach exceptions to Unix signals, such as EXC_BAD_ACCESS will convert to SIGSEGV or SIGBUS, as shown in the code
static
void ux_exception(
int exception,
mach_exception_code_t code,
mach_exception_subcode_t subcode,
int *ux_signal,
mach_exception_code_t *ux_code)
{
switch(exception) {
case EXC_BAD_ACCESS:
if (code == KERN_INVALID_ADDRESS)
*ux_signal = SIGSEGV;
else
*ux_signal = SIGBUS;
break; . }... }Copy the codeIn catch_mach_exception_raise(), threadSignal () is used to send Unix signals to the catch_mach_exception_raise(). Finally, act_set_astbsd() is called, where the AST (asynchronous software interrupt) signal is set
void
act_set_astbsd(
thread_t thread)
{
act_set_ast( thread, AST_BSD );
}
Copy the codeAST is a human-triggered non-hardware-triggered trap. AST is a key part of kernel operation, and is the underlying mechanism for scheduling events. It is also the basis for BSD signal delivery (Unix signal). When the system returns from a trap (return_FROm_trap), the system does not immediately return to user state, but checks the AST field of the thread to see if there is an AST that needs to be processed. As the code shows, the flag bit of the AST is AST_BSD, and the handler for this flag bit is the bsd_ast() function. In this case, if the breakpoint is set at Exception_triage (), the breakpoint will be broken. At this time, LLDB can be used to output function call stack, process PID and thread status at the terminal
(lldb) bt
* thread #1, stop reason = breakpoint 1.17
* frame # 0: 0xffffff800fe75fc9 kernel.development`proc_prepareexit [inlined] exception_triage(exception=10, code=0x000000000b100001, codeCnt=2) at exception.c:504 [opt]
frame # 1: 0xffffff800fe75fbc kernel.development`proc_prepareexit [inlined] task_exception_notify(exception=10, exccode=185597953, excsubcode=4369) at exception.c:547 [opt]
frame # 2: 0xffffff800fe75f96 kernel.development`proc_prepareexit(p=0xffffff8018d90b60, rv=
, perf_notify=1) at kern_exit.c:889 [opt]
frame # 3: 0xffffff800fe75d86 kernel.development`exit_with_reason(p=0xffffff8018d90b60, rv=11, retval=
, thread_can_terminate=1, perf_notify=1, jetsam_flags=
, exit_reason=
) at kern_exit.c:830 [opt]
frame #4: 0xffffff800fe90675 kernel.development`postsig_locked(signum=11) at kern_sig.c:3140 [opt]
frame #5: 0xffffff800fe90b07 kernel.development`bsd_ast(thread=<unavailable>) at kern_sig.c:3420 [opt]
frame #6: 0xffffff800f973e44 kernel.development`ast_taken_user at ast.c:207 [opt]
frame #7: 0xffffff800f9211bc kernel.development`return_from_trap + 172
(lldb) e struct proc *$proc_1 = (struct proc *)thread->task->bsd_info
(lldb) po $proc_1->p_pid
478
(lldb) po thread->state
4
Copy the codeAs you can see, bsd_ast() will call postsig_locked(). From the implementation of/BSD /kern/kern_sig.c postsig_locked(), if the current process has not set sigaction to catch Unix signals, The default handling is to call exit_with_reason(), which indirectly calls task_Exception_notify (), Task_exception_notify () notifies LaunchD to start ReportCrash to generate CrashLog. This notification is also done through Mach messaging, so the breakpoint breaks at Exception_triage ().
Launchd is that it set up in the process of initialization of the port, and sets the MachExceptionHandler to/System/Library/CoreServices/ReportCrash path of (iOS), ReportCrash will generate Crash logs. Exception_triage_thread (); exception_triage_thread(); exception_triage_thread(); exception_deliver(); If you look at frame #0 in the log above, you can see exception=10 (EXC_CRASH). This is the second time the breakpoint has been broken here. The first break was when the CPU exception turned into a Mach exception, exception=1 (EXC_BAD_ACCESS). The exception_Deliver () function will use the input exception to extract the specific exception port from the exception array, so the first (CPU exception to Mach exception) and the second (ReportCrash) exception will not conflict.
When the breakpoint is dropped, a second breakpoint will occur at exception_triage_thread()
(lldb) bt
* thread #1, stop reason = breakpoint 2.1
* frame # 0: 0xffffff800f97ef47 kernel.development`exception_triage_thread(exception=13, code=0xffffff806fce3e40, codeCnt=2, thread=0xffffff801cded250) at exception.c:445 [opt]
frame # 1: 0xffffff800f9acffe kernel.development`task_deliver_crash_notification(task=0xffffff801d9af000, thread=0xffffff801cded250, etype=
, subcode=
) at task.c:1798 [opt]
frame #2: 0xffffff800f9b6537 kernel.development`thread_terminate_self at thread.c:594 [opt]
frame #3: 0xffffff800f9bab30 kernel.development`thread_apc_ast(thread=0xffffff801cded250) at thread_act.c:934 [opt]
frame #4: 0xffffff800f973e6b kernel.development`ast_taken_user at ast.c:220 [opt]
frame #5: 0xffffff800f9211bc kernel.development`return_from_trap + 172
Copy the codeException =13 (EXC_CORPSE_NOTIFY), indicating that the process is dead.
The interrupt points show that a CPU exception caused by an illegal memory access by a user-mode application will use EXC_BAD_ACCESS, EXC_CRASH, and EXC_CORPSE_NOTIFY in sequence.
2.3 summary
CPU exceptions -> Mach exceptions -> Unix signals in the BSD layer -> user-mode App Handler/system Crash Log generation process can be easily drawn as a rough diagram
3. Exception collection
Although iOS \ macOS provides ReportCrash to collect Crash information and LLDB debugServer to catch program exceptions in Debug mode, it is not convenient for developers to collect Crash after App is launched. For example, on iOS, the user needs to allow the developer to share the analysis data, so that the developer can view the Crash report information from iTunes Connect. However, the Crash information can be viewed only when the device where the Crash occurred is obtained.
In order to facilitate rapid Crash location and solution, we can use ReportCrash or DebugServer to capture exceptions to create a three-party Crash collection framework. There are three main collection ideas:
- Catching Mach exceptions
- Capturing Unix signals
- NSSetUncaughtExceptionHandler
3.1 Catching Mach exceptions
Mach, while very low-level, also provides apis for user-mode applications to use, and you can use the following apis to catch Mach exceptions
- Call mach_port_ALLOCATE to create the exception handling port
- Call mach_port_insert_right to get permission for the port
- Call xxx_set_EXCEPtion_ports to set the exception port
- Call mach_msg to wait for messages on the abnormal port
// There are two things to note here:
3.2 the TODO
Because there are mature three-party frameworks for exception collection, such as KSCrash and PLCrashReport, I will refer to the open source framework and combine my RDA to do some work. I will continue to share here after I have enough practical experience