APM Monitoring System: Crash Monitoring (Part 2)

3. Use of KSCrash packaging

Then encapsulate its own Crash processing logic. Here are some things to do:

• Inherit from the abstract class KSCrashInstallation, set the initialization work (such as the abstract class NSURLProtocol must inherit after use), implement the sink method in the abstract class.

  /** * Crash system installation which handles backend-specific details. * * Only one installation can be installed at a time. * * This is an abstract class. */
  @interface KSCrashInstallation : NSObject
  Copy the code

  #import "APMCrashInstallation.h" #import <KSCrash/KSCrashInstallation+Private.h> #import "APMCrashReporterSink.h" @implementation APMCrashInstallation + (instancetype)sharedInstance { static APMCrashInstallation *sharedInstance = nil;  static dispatch_once_t onceToken; dispatch_once(&onceToken, ^{ sharedInstance = [[APMCrashInstallation alloc] init]; }); return sharedInstance; } - (id)init { return [super initWithRequiredProperties: nil]; } - (id<KSCrashReportFilter>)sink { APMCrashReporterSink *sink = [[APMCrashReporterSink alloc] init]; return [sink defaultCrashReportFilterSetAppleFmt]; } @endCopy the code

• Sink within a method APMCrashReporterSink class, follow the KSCrashReportFilter agreement, declared the defaultCrashReportFilterSetAppleFmt public methods

  // .h
  #import <Foundation/Foundation.h>
  #import <KSCrash/KSCrashReportFilter.h>
  
  @interface APMCrashReporterSink : NSObject<KSCrashReportFilter>
  
  - (id <KSCrashReportFilter>) defaultCrashReportFilterSetAppleFmt;
  
  @end
  
  // .m
  #pragma mark - public Method
  
  - (id <KSCrashReportFilter>) defaultCrashReportFilterSetAppleFmt
  {
      return [KSCrashReportFilterPipeline filterWithFilters:
              [APMCrashReportFilterAppleFmt filterWithReportStyle:KSAppleReportStyleSymbolicatedSideBySide],
              self,
              nil];
  }
  Copy the code

  The defaultCrashReportFilterSetAppleFmt internal () method returns a KSCrashReportFilterPipeline class methods filterWithFilters results.

  APMCrashReportFilterAppleFmt is an inherited from KSCrashReportFilterAppleFmt class, follow the KSCrashReportFilter protocol. The protocol approach allows developers to work with Crash’s data format.

  /** Filter the specified reports.
   *
   * @param reports The reports to process.
   * @param onCompletion Block to call when processing is complete.
   */
  - (void) filterReports:(NSArray*) reports
            onCompletion:(KSCrashReportFilterCompletion) onCompletion;
  Copy the code

  #import <KSCrash/KSCrashReportFilterAppleFmt.h> @interface APMCrashReportFilterAppleFmt : KSCrashReportFilterAppleFmt<KSCrashReportFilter> @end // .m - (void) filterReports:(NSArray*)reports onCompletion:(KSCrashReportFilterCompletion)onCompletion { NSMutableArray* filteredReports = [NSMutableArray arrayWithCapacity:[reports count]]; for(NSDictionary *report in reports){ if([self majorVersion:report] == kExpectedMajorVersion){ id monitorInfo = [self generateMonitorInfoFromCrashReport:report]; if(monitorInfo ! = nil){ [filteredReports addObject:monitorInfo]; } } } kscrash_callCompletion(onCompletion, filteredReports, YES, nil); } /** @brief Get Crash time, Mach name, signal name, and Apple Report */ - (NSDictionary) in Crash JSON *)generateMonitorInfoFromCrashReport:(NSDictionary *)crashReport { NSDictionary *infoReport = [crashReport objectForKey:@"report"]; / /... id appleReport = [self toAppleFormat:crashReport]; NSMutableDictionary *info = [NSMutableDictionary dictionary]; [info setValue:crashTime forKey:@"crashTime"]; [info setValue:appleReport forKey:@"appleReport"]; [info setValue:userException forKey:@"userException"]; [info setValue:userInfo forKey:@"custom"]; return [info copy]; }Copy the code

  /**
   * A pipeline of filters. Reports get passed through each subfilter in order.
   *
   * Input: Depends on what's in the pipeline.
   * Output: Depends on what's in the pipeline.
   */
  @interface KSCrashReportFilterPipeline : NSObject <KSCrashReportFilter>
  Copy the code

• Set up an initiator for the Crash module in APM capability. The initiator internally sets up the initialization of KSCrash and the assembly of the data needed to monitor when a Crash occurs. For example: SESSION_ID, App start time, App name, crash time, App version number, current page information and other basic information.

  /** C Function to call during a crash report to give the callee an opportunity to
   * add to the report. NULL = ignore.
   *
   * WARNING: Only call async-safe functions from this function! DO NOT call
   * Objective-C methods!!!
   */
  @property(atomic,readwrite,assign) KSReportWriteCallback onCrash;
  Copy the code

  + (instancetype)sharedInstance
  {
      static APMCrashMonitor *_sharedManager = nil;
      static dispatch_once_t onceToken;
      dispatch_once(&onceToken, ^{
          _sharedManager = [[APMCrashMonitor alloc] init];
      });
      return _sharedManager;
  }
  
  
  #pragma mark - public Method
  
  - (void)startMonitor
  {
      APMMLog(@"crash monitor started");
  
  #ifdef DEBUG
      BOOL _trackingCrashOnDebug = [APMMonitorConfig sharedInstance].trackingCrashOnDebug;
      if (_trackingCrashOnDebug) {
          [self installKSCrash];
      }
  #else
      [self installKSCrash];
  #endif
  }
  
  #pragma mark - private method
  
  static void onCrash(const KSCrashReportWriter* writer)
  {
      NSString *sessionId = [NSString stringWithFormat:@"\"%@\"", ***]];
      writer->addJSONElement(writer, "SESSION_ID", [sessionId UTF8String], true);
      
      NSString *appLaunchTime = ***;
      writer->addJSONElement(writer, "USER_APP_START_DATE", [[NSString stringWithFormat:@"\"%@\"", appLaunchTime] UTF8String], true);
      // ...
  }
  
  - (void)installKSCrash
  {
      [[APMCrashInstallation sharedInstance] install];
      [[APMCrashInstallation sharedInstance] sendAllReportsWithCompletion:nil];
      [APMCrashInstallation sharedInstance].onCrash = onCrash;
      dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(5.f * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
          _isCanAddCrashCount = NO;
      });
  }
  Copy the code

  In installKSCrash method calls the [[APMCrashInstallation sharedInstance] sendAllReportsWithCompletion: nil], internal implementation is as follows

  - (void) sendAllReportsWithCompletion:(KSCrashReportFilterCompletion) onCompletion { NSError* error = [self validateProperties]; if(error ! = nil) { if(onCompletion ! = nil) { onCompletion(nil, NO, error); } return; } id<KSCrashReportFilter> sink = [self sink]; if(sink == nil) { onCompletion(nil, NO, [NSError errorWithDomain:[[self class] description] code:0 description:@"Sink was nil (subclasses must implement method \"sink\")"]); return; } sink = [KSCrashReportFilterPipeline filterWithFilters:self.prependedFilters, sink, nil]; KSCrash* handler = [KSCrash sharedInstance]; handler.sink = sink; [handler sendAllReportsWithCompletion:onCompletion]; }Copy the code

  Method internally assigns the KSCrashInstallation sink to the KSCrash object. Internal or call the KSCrash sendAllReportsWithCompletion method, implemented as follows

  - (void) sendAllReportsWithCompletion:(KSCrashReportFilterCompletion) onCompletion { NSArray* reports = [self allReports]; KSLOG_INFO(@"Sending %d crash reports", [reports count]); [self sendReports:reports onCompletion:^(NSArray* filteredReports, BOOL completed, NSError* error) { KSLOG_DEBUG(@"Process finished with completion: %d", completed); if(error != nil) { KSLOG_ERROR(@"Failed to send reports: %@", error);  } if((self.deleteBehaviorAfterSendAll == KSCDeleteOnSucess && completed) || self.deleteBehaviorAfterSendAll == KSCDeleteAlways) { kscrash_deleteAllReports();  } kscrash_callCompletion(onCompletion, filteredReports, completed, error); }]; }Copy the code

  The method internally calls the object method sendReports: onCompletion:, as shown below

  - (void) sendReports:(NSArray*) reports onCompletion:(KSCrashReportFilterCompletion) onCompletion
  {
      if([reports count] == 0)
      {
          kscrash_callCompletion(onCompletion, reports, YES, nil);
          return;
      }
      
      if(self.sink == nil)
      {
          kscrash_callCompletion(onCompletion, reports, NO,
                                   [NSError errorWithDomain:[[self class] description]
                                                       code:0
                                                description:@"No sink set. Crash reports not sent."]);
          return;
      }
      
      [self.sink filterReports:reports
                  onCompletion:^(NSArray* filteredReports, BOOL completed, NSError* error)
       {
           kscrash_callCompletion(onCompletion, filteredReports, completed, error);
       }];
  }
  Copy the code

  [self.sink filterReports: onCompletion: ] is the sink getter method set in APMCrashInstallation. Internal object returned to APMCrashReporterSink defaultCrashReportFilterSetAppleFmt methods return values. The internal implementation is as follows

  - (id <KSCrashReportFilter>) defaultCrashReportFilterSetAppleFmt
  {
      return [KSCrashReportFilterPipeline filterWithFilters:
              [APMCrashReportFilterAppleFmt filterWithReportStyle:KSAppleReportStyleSymbolicatedSideBySide],
              self,
              nil];
  }
  Copy the code

  As you can see, there are multiple filters inside this function. One of them is self, the APMCrashReporterSink object, so [self.sink filterReports: OnCompletion :], which calls the data processing method in the APMCrashReporterSink. After that, pass kscrash_callCompletion(onCompletion, reports, YES, nil); Tell KSCrash that Crash logs saved locally have been processed and can be deleted.

  - (void)filterReports:(NSArray *)reports onCompletion:(KSCrashReportFilterCompletion)onCompletion { for (NSDictionary *report in reports) {// Process Crash data, send data to a unified data reporting component for processing... } kscrash_callCompletion(onCompletion, reports, YES, nil); }Copy the code

  At this point, the summary of what KSCrash does is to provide various crash monitoring capabilities. After crash, process information, basic information, exception information, thread information and other information are efficiently converted into JSON files by C. After the next startup, the App reads the crash logs in the local crash folder, allowing developers to define their own keys and values and report the logs to the APM system, and then delete the logs in the local Crash folder.

4. Symbolic

After crash is applied, the system generates a crash log and stores it in the Settings. Information about the running status, call stack, and thread of the application is recorded in the log. But these logs are addresses and are not readable, so symbolic restoration is required.

4.1. DSYM file

The debugging symbol (.dsym) file is a transfer file that stores the address mapping information of hexadecimal functions. The debugging information (symbols) is contained in this file. Xcode projects generate new.dsym files every time they are compiled and run. By default,.dsym is not generated in debug mode. You can modify DWARF to DWARF with DSYM File in Build Settings -> Build Options -> Debug Information Format. This recompile runs to generate.dsym files.

So every time your App is packaged, you need to save the.dsym file for each version.

Information contained in DWARF DSYM file, open the file package Contents Test. App. DSYM/Contents/Resources/DWARF/Test is saved to DWARF file.

The.dsym file is a directory that extracts debugging information from the Mach-o file. During distribution, debugging information is stored in a separate file for security purposes. The.dsym file is actually a file directory with the following structure:

4.2 DWARF file

DWARF is a debugging file format used by many compilers and debuggers to support source level debugging. It addresses the requirements of a number of procedural languages, such as C, C++, and Fortran, and is designed to be extensible to other languages. DWARF is architecture independent and applicable to any processor or operating system. It is widely used on Unix, Linux and other operating systems, as well as in stand-alone environments.

DWARF is a debug file format that is widely used by many compilers and debuggers to support source-level debugging. It meets the requirements of many process languages (C, C++, Fortran), and it is designed to support extension to other languages. DWARF is architecture-independent and is suitable for any other processor and operating system. It is widely used on Unix, Linux, and other operating systems, as well as standalone environments.

DWARF, for Debugging With Arbitrary Record Formats, is a Debugging file that uses an attribute Record format.

DWARF is a compact representation of the relationship between executable programs and source code.

Most modern programming languages are block structures: each entity (a class, a function) is contained within another entity. A C program may contain multiple data definitions, variables, and functions per file, so DWARF follows this model and is also a block structure. The basic description in DWARF is DIE (Debugging Information Entry). A DIE has a tag that indicates what the DIE describes and a list of attributes (analogous to HTML, XML structures) that fill in the details to further describe the item. A DIE (except for the topmost DIE) is contained by a parent DIE. There may be sibling dies or child dies, and attributes may contain various values: constants (such as the name of a function), variables (such as the starting address of a function), or references to another DIE (such as the return value type of a function).

The data in DWARF are as follows:

Data column	The information that
.debug_loc	A list of locations used in the DW_AT_location property
.debug_macinfo	The macro information
.debug_pubnames	Lookup tables for global objects and functions
.debug_pubtypes	Lookup table of global type
.debug_ranges	The address range used in the DW_AT_ranges property
.debug_str	A list of strings used in.debug_info
.debug_types	Type description

Common tags and attributes are as follows:

Data column	The information that
DW_TAG_class_type	Represents the class name and type information
DW_TAG_structure_type	Represents structure name and type information
DW_TAG_union_type	Represents joint name and type information
DW_TAG_enumeration_type	Represents enumeration name and type information
DW_TAG_typedef	Represents the name and type information for a typedef
DW_TAG_array_type	Represents array name and type information
DW_TAG_subrange_type	Represents the size information of an array
DW_TAG_inheritance	Represents inherited class name and type information
DW_TAG_member	Represents a member of a class
DW_TAG_subprogram	Represents the name information of a function
DW_TAG_formal_parameter	Represents parameter information about a function
DW_TAG_name	Name string
DW_TAG_type	Presentation type information
DW_TAG_artifical	Set by the compiler at creation time
DW_TAG_sibling	Represents sibling location information
DW_TAG_data_memver_location	Indicate location information
DW_TAG_virtuality	Set when virtual

A quick example of DWARF: Parse the DWARF files in the.dsym folder of the test project with the following command

dwarfdump -F --debug-info Test.app.DSYM/Contents/Resources/DWARF/Test > debug-info.txt
Copy the code

Open the following

Test.app.DSYM/Contents/Resources/DWARF/Test:	file format Mach-O arm64

.debug_info contents:
0x00000000: Compile Unit: length = 0x0000004f version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x00000053)

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer [DW_FORM_strp]	("Apple clang version 11.0.3 (clang-1103.0.32.62)")
              DW_AT_language [DW_FORM_data2]	(DW_LANG_ObjC)
              DW_AT_name [DW_FORM_strp]	("_Builtin_stddef_max_align_t")
              DW_AT_stmt_list [DW_FORM_sec_offset]	(0x00000000)
              DW_AT_comp_dir [DW_FORM_strp]	("/Users/lbp/Desktop/Test")
              DW_AT_APPLE_major_runtime_vers [DW_FORM_data1]	(0x02)
              DW_AT_GNU_dwo_id [DW_FORM_data8]	(0x392b5344d415340c)

0x00000027:   DW_TAG_module
                DW_AT_name [DW_FORM_strp]	("_Builtin_stddef_max_align_t")
                DW_AT_LLVM_config_macros [DW_FORM_strp]	("\"-DDEBUG=1\" \"-DOBJC_OLD_DISPATCH_PROTOTYPES=1\"")
                DW_AT_LLVM_include_path [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include")
                DW_AT_LLVM_isysroot [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk")

0x00000038:     DW_TAG_typedef
                  DW_AT_type [DW_FORM_ref4]	(0x0000004b "long double")
                  DW_AT_name [DW_FORM_strp]	("max_align_t")
                  DW_AT_decl_file [DW_FORM_data1]	("/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include/__stddef_max_align_t.h")
                  DW_AT_decl_line [DW_FORM_data1]	(16)

0x00000043:     DW_TAG_imported_declaration
                  DW_AT_decl_file [DW_FORM_data1]	("/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include/__stddef_max_align_t.h")
                  DW_AT_decl_line [DW_FORM_data1]	(27)
                  DW_AT_import [DW_FORM_ref_addr]	(0x0000000000000027)

0x0000004a:     NULL

0x0000004b:   DW_TAG_base_type
                DW_AT_name [DW_FORM_strp]	("long double")
                DW_AT_encoding [DW_FORM_data1]	(DW_ATE_float)
                DW_AT_byte_size [DW_FORM_data1]	(0x08)

0x00000052:   NULL
0x00000053: Compile Unit: length = 0x000183dc version = 0x0004 abbr_offset = 0x0000 addr_size = 0x08 (next unit at 0x00018433)

0x0000005e: DW_TAG_compile_unit
              DW_AT_producer [DW_FORM_strp]	("Apple clang version 11.0.3 (clang-1103.0.32.62)")
              DW_AT_language [DW_FORM_data2]	(DW_LANG_ObjC)
              DW_AT_name [DW_FORM_strp]	("Darwin")
              DW_AT_stmt_list [DW_FORM_sec_offset]	(0x000000a7)
              DW_AT_comp_dir [DW_FORM_strp]	("/Users/lbp/Desktop/Test")
              DW_AT_APPLE_major_runtime_vers [DW_FORM_data1]	(0x02)
              DW_AT_GNU_dwo_id [DW_FORM_data8]	(0xa4a1d339379e18a5)

0x0000007a:   DW_TAG_module
                DW_AT_name [DW_FORM_strp]	("Darwin")
                DW_AT_LLVM_config_macros [DW_FORM_strp]	("\"-DDEBUG=1\" \"-DOBJC_OLD_DISPATCH_PROTOTYPES=1\"")
                DW_AT_LLVM_include_path [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include")
                DW_AT_LLVM_isysroot [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk")

0x0000008b:     DW_TAG_module
                  DW_AT_name [DW_FORM_strp]	("C")
                  DW_AT_LLVM_config_macros [DW_FORM_strp]	("\"-DDEBUG=1\" \"-DOBJC_OLD_DISPATCH_PROTOTYPES=1\"")
                  DW_AT_LLVM_include_path [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include")
                  DW_AT_LLVM_isysroot [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk")

0x0000009c:       DW_TAG_module
                    DW_AT_name [DW_FORM_strp]	("fenv")
                    DW_AT_LLVM_config_macros [DW_FORM_strp]	("\"-DDEBUG=1\" \"-DOBJC_OLD_DISPATCH_PROTOTYPES=1\"")
                    DW_AT_LLVM_include_path [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include")
                    DW_AT_LLVM_isysroot [DW_FORM_strp]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk")

0x000000ad:         DW_TAG_enumeration_type
                      DW_AT_type [DW_FORM_ref4]	(0x00017276 "unsigned int")
                      DW_AT_byte_size [DW_FORM_data1]	(0x04)
                      DW_AT_decl_file [DW_FORM_data1]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/fenv.h")
                      DW_AT_decl_line [DW_FORM_data1]	(154)

0x000000b5:           DW_TAG_enumerator
                        DW_AT_name [DW_FORM_strp]	("__fpcr_trap_invalid")
                        DW_AT_const_value [DW_FORM_udata]	(256)

0x000000bc:           DW_TAG_enumerator
                        DW_AT_name [DW_FORM_strp]	("__fpcr_trap_divbyzero")
                        DW_AT_const_value [DW_FORM_udata]	(512)

0x000000c3:           DW_TAG_enumerator
                        DW_AT_name [DW_FORM_strp]	("__fpcr_trap_overflow")
                        DW_AT_const_value [DW_FORM_udata]	(1024)

0x000000ca:           DW_TAG_enumerator
                        DW_AT_name [DW_FORM_strp]	("__fpcr_trap_underflow")
// ......
0x000466ee:   DW_TAG_subprogram
                DW_AT_name [DW_FORM_strp]	("CFBridgingRetain")
                DW_AT_decl_file [DW_FORM_data1]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObject.h")
                DW_AT_decl_line [DW_FORM_data1]	(105)
                DW_AT_prototyped [DW_FORM_flag_present]	(true)
                DW_AT_type [DW_FORM_ref_addr]	(0x0000000000019155 "CFTypeRef")
                DW_AT_inline [DW_FORM_data1]	(DW_INL_inlined)

0x000466fa:     DW_TAG_formal_parameter
                  DW_AT_name [DW_FORM_strp]	("X")
                  DW_AT_decl_file [DW_FORM_data1]	("/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObject.h")
                  DW_AT_decl_line [DW_FORM_data1]	(105)
                  DW_AT_type [DW_FORM_ref4]	(0x00046706 "id")

0x00046705:     NULL

0x00046706:   DW_TAG_typedef
                DW_AT_type [DW_FORM_ref4]	(0x00046711 "objc_object*")
                DW_AT_name [DW_FORM_strp]	("id")
                DW_AT_decl_file [DW_FORM_data1]	("/Users/lbp/Desktop/Test/Test/NetworkAPM/NSURLResponse+apm_FetchStatusLineFromCFNetwork.m")
                DW_AT_decl_line [DW_FORM_data1]	(44)

0x00046711:   DW_TAG_pointer_type
                DW_AT_type [DW_FORM_ref4]	(0x00046716 "objc_object")

0x00046716:   DW_TAG_structure_type
                DW_AT_name [DW_FORM_strp]	("objc_object")
                DW_AT_byte_size [DW_FORM_data1]	(0x00)

0x0004671c:     DW_TAG_member
                  DW_AT_name [DW_FORM_strp]	("isa")
                  DW_AT_type [DW_FORM_ref4]	(0x00046727 "objc_class*")
                  DW_AT_data_member_location [DW_FORM_data1]	(0x00)
// ......
Copy the code

I won’t paste the whole thing here (it’s too long). DIE contains the start address, end address, function name, file name and number of lines. For a given address, find the DIE between the start address and end address of the function that contains the address, then you can reduce the function name and file name information.

Debug_line restores information such as the number of file lines

dwarfdump -F --debug-line Test.app.DSYM/Contents/Resources/DWARF/Test > debug-inline.txt
Copy the code

Post partial information

Test.app.DSYM/Contents/Resources/DWARF/Test:	file format Mach-O arm64

.debug_line contents:
debug_line[0x00000000]
Line table prologue:
    total_length: 0x000000a3
         version: 4
 prologue_length: 0x0000009a
 min_inst_length: 1
max_ops_per_inst: 1
 default_is_stmt: 1
       line_base: -5
      line_range: 14
     opcode_base: 13
standard_opcode_lengths[DW_LNS_copy] = 0
standard_opcode_lengths[DW_LNS_advance_pc] = 1
standard_opcode_lengths[DW_LNS_advance_line] = 1
standard_opcode_lengths[DW_LNS_set_file] = 1
standard_opcode_lengths[DW_LNS_set_column] = 1
standard_opcode_lengths[DW_LNS_negate_stmt] = 0
standard_opcode_lengths[DW_LNS_set_basic_block] = 0
standard_opcode_lengths[DW_LNS_const_add_pc] = 0
standard_opcode_lengths[DW_LNS_fixed_advance_pc] = 1
standard_opcode_lengths[DW_LNS_set_prologue_end] = 0
standard_opcode_lengths[DW_LNS_set_epilogue_begin] = 0
standard_opcode_lengths[DW_LNS_set_isa] = 1
include_directories[  1] = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include"
file_names[  1]:
           name: "__stddef_max_align_t.h"
      dir_index: 1
       mod_time: 0x00000000
         length: 0x00000000

Address            Line   Column File   ISA Discriminator Flags
------------------ ------ ------ ------ --- ------------- -------------
0x0000000000000000      1      0      1   0             0  is_stmt end_sequence
debug_line[0x000000a7]
Line table prologue:
    total_length: 0x0000230a
         version: 4
 prologue_length: 0x00002301
 min_inst_length: 1
max_ops_per_inst: 1
 default_is_stmt: 1
       line_base: -5
      line_range: 14
     opcode_base: 13
standard_opcode_lengths[DW_LNS_copy] = 0
standard_opcode_lengths[DW_LNS_advance_pc] = 1
standard_opcode_lengths[DW_LNS_advance_line] = 1
standard_opcode_lengths[DW_LNS_set_file] = 1
standard_opcode_lengths[DW_LNS_set_column] = 1
standard_opcode_lengths[DW_LNS_negate_stmt] = 0
standard_opcode_lengths[DW_LNS_set_basic_block] = 0
standard_opcode_lengths[DW_LNS_const_add_pc] = 0
standard_opcode_lengths[DW_LNS_fixed_advance_pc] = 1
standard_opcode_lengths[DW_LNS_set_prologue_end] = 0
standard_opcode_lengths[DW_LNS_set_epilogue_begin] = 0
standard_opcode_lengths[DW_LNS_set_isa] = 1
include_directories[  1] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include"
include_directories[  2] = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include"
include_directories[  3] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/sys"
include_directories[  4] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/mach"
include_directories[  5] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/libkern"
include_directories[  6] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/architecture"
include_directories[  7] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/sys/_types"
include_directories[  8] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/_types"
include_directories[  9] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/arm"
include_directories[ 10] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/sys/_pthread"
include_directories[ 11] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/mach/arm"
include_directories[ 12] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/libkern/arm"
include_directories[ 13] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/uuid"
include_directories[ 14] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/netinet"
include_directories[ 15] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/netinet6"
include_directories[ 16] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/net"
include_directories[ 17] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/pthread"
include_directories[ 18] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/mach_debug"
include_directories[ 19] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/os"
include_directories[ 20] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/malloc"
include_directories[ 21] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/bsm"
include_directories[ 22] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/machine"
include_directories[ 23] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/mach/machine"
include_directories[ 24] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/secure"
include_directories[ 25] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/xlocale"
include_directories[ 26] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/arpa"
file_names[  1]:
           name: "fenv.h"
      dir_index: 1
       mod_time: 0x00000000
         length: 0x00000000
file_names[  2]:
           name: "stdatomic.h"
      dir_index: 2
       mod_time: 0x00000000
         length: 0x00000000
file_names[  3]:
           name: "wait.h"
      dir_index: 3
       mod_time: 0x00000000
         length: 0x00000000
// ......
Address            Line   Column File   ISA Discriminator Flags
------------------ ------ ------ ------ --- ------------- -------------
0x000000010000b588     14      0      2   0             0  is_stmt
0x000000010000b5b4     16      5      2   0             0  is_stmt prologue_end
0x000000010000b5d0     17     11      2   0             0  is_stmt
0x000000010000b5d4      0      0      2   0             0 
0x000000010000b5d8     17      5      2   0             0 
0x000000010000b5dc     17     11      2   0             0 
0x000000010000b5e8     18      1      2   0             0  is_stmt
0x000000010000b608     20      0      2   0             0  is_stmt
0x000000010000b61c     22      5      2   0             0  is_stmt prologue_end
0x000000010000b628     23      5      2   0             0  is_stmt
0x000000010000b644     24      1      2   0             0  is_stmt
0x000000010000b650     15      0      1   0             0  is_stmt
0x000000010000b65c     15     41      1   0             0  is_stmt prologue_end
0x000000010000b66c     11      0      2   0             0  is_stmt
0x000000010000b680     11     17      2   0             0  is_stmt prologue_end
0x000000010000b6a4     11     17      2   0             0  is_stmt end_sequence
debug_line[0x0000def9]
Line table prologue:
    total_length: 0x0000015a
         version: 4
 prologue_length: 0x000000eb
 min_inst_length: 1
max_ops_per_inst: 1
 default_is_stmt: 1
       line_base: -5
      line_range: 14
     opcode_base: 13
standard_opcode_lengths[DW_LNS_copy] = 0
standard_opcode_lengths[DW_LNS_advance_pc] = 1
standard_opcode_lengths[DW_LNS_advance_line] = 1
standard_opcode_lengths[DW_LNS_set_file] = 1
standard_opcode_lengths[DW_LNS_set_column] = 1
standard_opcode_lengths[DW_LNS_negate_stmt] = 0
standard_opcode_lengths[DW_LNS_set_basic_block] = 0
standard_opcode_lengths[DW_LNS_const_add_pc] = 0
standard_opcode_lengths[DW_LNS_fixed_advance_pc] = 1
standard_opcode_lengths[DW_LNS_set_prologue_end] = 0
standard_opcode_lengths[DW_LNS_set_epilogue_begin] = 0
standard_opcode_lengths[DW_LNS_set_isa] = 1
include_directories[  1] = "Test"
include_directories[  2] = "Test/NetworkAPM"
include_directories[  3] = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.5.sdk/usr/include/objc"
file_names[  1]:
           name: "AppDelegate.h"
      dir_index: 1
       mod_time: 0x00000000
         length: 0x00000000
file_names[  2]:
           name: "JMWebResourceURLProtocol.h"
      dir_index: 2
       mod_time: 0x00000000
         length: 0x00000000
file_names[  3]:
           name: "AppDelegate.m"
      dir_index: 1
       mod_time: 0x00000000
         length: 0x00000000
file_names[  4]:
           name: "objc.h"
      dir_index: 3
       mod_time: 0x00000000
         length: 0x00000000
// ......
Copy the code

You can see that the debug_line contains the number of lines for each code address. The AppDelegate section is attached to it.

4.3 symbols

In the link, function and variable are collectively called Symbol, and the function Name or variable Name is the Symbol Name. We can regard symbols as the glue in the link, and the whole link process can be completed correctly based on symbols.

The above text is from Self-cultivation of programmer. So a symbol is a general term for functions, variables and classes.

By type, symbols can be divided into three categories:

• Global symbol: a symbol visible outside an object file that can be referenced by, or needs to be defined by, another object file
• Local symbols: symbols that are visible only in object files. Functions and variables that are visible only in object files
• Debug symbol: debug symbol information including line number information, the line number information records the function and variable corresponding file and file line number.

Symbol Table: a mapping Table of memory addresses with function names, file names, and line numbers. Each defined Symbol has a corresponding Value, called Symbol Value. For variables and functions, Symbol Value is the address. The Symbol table is composed as follows

< start address > < end address > < function > [< file name: line number >]Copy the code

4.4 How do I get an address?

When the image is loaded, it will be relocated relative to the base address, and the base address is different each time. The address of the function stack frame is the absolute address after relocation, and we need the relative address before relocation.

Binary Images

For example, in the crash log of the test project, open a portion of Binary Images

// ...
Binary Images:
0x102fe0000 - 0x102ff3fff Test arm64  <37eaa57df2523d95969e47a9a1d69ce5> /var/containers/Bundle/Application/643F0DFE-A710-4136-A278-A89D780B7208/Test.app/Test
0x1030e0000 - 0x1030ebfff libobjc-trampolines.dylib arm64  <181f3aa866d93165ac54344385ac6e1d> /usr/lib/libobjc-trampolines.dylib
0x103204000 - 0x103267fff dyld arm64  <6f1c86b640a3352a8529bca213946dd5> /usr/lib/dyld
0x189a78000 - 0x189a8efff libsystem_trace.dylib arm64  <b7477df8f6ab3b2b9275ad23c6cc0b75> /usr/lib/system/libsystem_trace.dylib
// ...
Copy the code

It can be seen that the Binary Images of the Crash log contain the loading start address, loading end address, Image name, ARM architecture, UUID and Image path of each Image.

Information in crash logs

Last Exception Backtrace:
// ...
5   Test                          	0x102fe592c -[ViewController testMonitorCrash] + 22828 (ViewController.mm:58)
Copy the code

Binary Images:
0x102fe0000 - 0x102ff3fff Test arm64  <37eaa57df2523d95969e47a9a1d69ce5> /var/containers/Bundle/Application/643F0DFE-A710-4136-A278-A89D780B7208/Test.app/Test
Copy the code

Therefore, the relative address of frame 5 is 0x102FE592c-0x102FE0000. The symbol information can be restored using the command.

Using ATOS to resolve, 0x102Fe0000 is the initial address of image loading, and 0x102FE592c is the address that frame needs to restore.

atos -o Test.app.DSYM/Contents/Resources/DWARF/Test-arch arm64 -l 0x102fe0000 0x102fe592c
Copy the code

4.5 UUID

• UUID of the crash file

  grep --after-context=2 "Binary Images:" *.crash
  Copy the code

  Test  5-28-20, 7-47 PM.crash:Binary Images:
  Test  5-28-20, 7-47 PM.crash-0x102fe0000 - 0x102ff3fff Test arm64  <37eaa57df2523d95969e47a9a1d69ce5> /var/containers/Bundle/Application/643F0DFE-A710-4136-A278-A89D780B7208/Test.app/Test
  Test  5-28-20, 7-47 PM.crash-0x1030e0000 - 0x1030ebfff libobjc-trampolines.dylib arm64  <181f3aa866d93165ac54344385ac6e1d> /usr/lib/libobjc-trampolines.dylib
  --
  Test.crash:Binary Images:
  Test.crash-0x102fe0000 - 0x102ff3fff Test arm64  <37eaa57df2523d95969e47a9a1d69ce5> /var/containers/Bundle/Application/643F0DFE-A710-4136-A278-A89D780B7208/Test.app/Test
  Test.crash-0x1030e0000 - 0x1030ebfff libobjc-trampolines.dylib arm64  <181f3aa866d93165ac54344385ac6e1d> /usr/lib/libobjc-trampolines.dylib
  Copy the code

  The Test App UUID for 37 eaa57df2523d95969e47a9a1d69ce5.

• DSYM file UUID

  dwarfdump --uuid Test.app.DSYM
  Copy the code

  The results for

  UUID: 37EAA57D-F252-3D95-969E-47A9A1D69CE5 (arm64) Test.app.DSYM/Contents/Resources/DWARF/Test
  Copy the code

• The app UUID

  dwarfdump --uuid Test.app/Test
  Copy the code

  The results for

  UUID: 37EAA57D-F252-3D95-969E-47A9A1D69CE5 (arm64) Test.app/Test
  Copy the code

4.6 Symbolization (Parsing Crash Logs)

The above section analyzes how to capture various types of crash. In the hands of users, we can obtain the crash scene information by technical means and report it in combination with certain mechanisms. However, such a stack is a hexadecimal address, so it cannot locate the problem, so symbolic processing is required.

DSYM files are used to restore file names, lines and function names by combining symbolic addresses with DSYM files. This process is called symbolization. However, the.DSYM file must correspond to the bundle ID and version of the crash log file.

To obtain Crash logs, you can run Xcode -> Window -> Devices and Simulators to select the corresponding device and locate the Crash log file according to the time and App name.

App and the DSYM file can be obtained by the product of packaging, path for ~ / Library/Developer/Xcode/Archives.

There are generally two methods of parsing:

• Using symbolicatecrash

  Symbolicatecrash is Xcode’s own crash log analysis tool. First determine the path and run the following command on the terminal

  find /Applications/Xcode.app -name symbolicatecrash -type f
  Copy the code

  It returns several paths to the line where iphonesimulator.platform is located

  /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/PrivateFrameworks/DVTFou ndation.framework/symbolicatecrashCopy the code

  Copy symbolicatecrash to the specified folder (the folder where app, DSYM, and Crash files are stored)

  Execute the command

  ./symbolicatecrash Test.crash Test.DSYM > Test.crash
  Copy the code

  /symbolicatecrash line 69. / DEVELOPER_DIR is not defined at /symbolicatecrash line 69

  export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
  Copy the code

• The use of atos

  Unlike Symbolicatecrash, Atos is more flexible, as long as.crash and.dsym or.crash and.app files correspond.

  -l follows the symbolic address

  xcrun atos -o Test.app.DSYM/Contents/Resources/DWARF/Test -arch armv7 -l 0x1023c592c
  Copy the code

  It can also parse.app files (not.dsym files), where XXX is the segment address and xx is the offset address

  atos -arch architecture -o binary -l xxx xx
  Copy the code

Since there may be a lot of apps, and each App may have different versions in the hands of users, the crash file and.dsym file need to correspond one by one to be symbolized after APM interception so as to be properly symbolized. The corresponding principle is consistent UUID.

4.7 Symbolic analysis of the system library

Every time we connect the real machine to Xcode and run the program, it will tell us to wait. In fact, in order to parse the stack, Saves the current version of Library automatic system symbols imported to/Users/your user name/Library/Developer/Xcode/iOS DeviceSupport directory to install a lot of symbolic file system libraries. You can visit the following directory to have a look

/ Users/your user name/Library/Developer/Xcode/iOS DeviceSupport /Copy the code

5. Server processing

5.1 ELK Log System

Industry design log monitoring system generally uses ELK based technology. ELK is the abbreviation of Elasticsearch, Logstash, and Kibana. Elasticsearch is a distributed, Restful, near-real-time search platform framework. Logstash is a central data flow engine used to from different target (documents/data storage/MQ) to collect data of different format, support after filtered output to different destination (file/MQ/Redis/ElasticsSearch/Kafka). Kibana allows Elasticserarch data to be displayed on friendly pages, providing visual analysis. Therefore, ELK can build an efficient, enterprise-level log analysis system.

In the early era of single application, almost all functions of applications ran on the same machine. When a problem occurred, o&M personnel opened the terminal and entered commands to view system logs to locate and resolve the problem. As the function of the system more and more complex, the user volume is more and more big, the monomer used hardly to meet demand, so the technical architecture iteration, through horizontal expand to support a large population, the monomers were split into multiple application, each application using the cluster deployment, load balancing control scheduling, a problem if a child module, Go to this server terminal for log analysis? Obviously backward, so the log management platform came into being. The log files of each server are collected and analyzed through Logstash, and then filtered according to the defined regular template and transmitted to Kafka or Redis. Then another Logstash reads the logs from Kafka or Redis and stores them in ES to create indexes. Finally, Kibana was used for visual analysis. In addition, the collected data can be analyzed for further maintenance and decision-making.

The figure above shows an ELK log architecture diagram. A quick note:

• There is a Kafka layer before Logstash and ES. Because Logstash is set up on the data resource server, the collected data is filtered in real time. Filtering takes time and memory, so Kafka exists, which plays a role of data buffer storage. Kafka has excellent read and write performance.
• The Logstash step is to read the data from Kafka, filter and process the data, and transmit the results to ES
• This design is not only good performance, low coupling, but also extensible. For example, you can read from n different Logstash stash files and transfer them to n KafKas, which are then filtered by n Logstash stash files. There can be m log sources, such as App logs, Tomcat logs, and Nginx logs

Below is a screenshot of an “Elastic APM hands-on” theme shared by the Elasticsearch community.

5.2 the service side

When the Crash log is uniformly stored in Kibana, it is not symbolized, so it needs symbolic processing to facilitate problem location, Crash report generation and subsequent processing.

So the whole process is: client APM SDK to collect crash log -> Kafka storage -> Mac to execute scheduled task symbolization -> data return Kafka -> product side (display end) to classify data, report, alarm and other operations.

Because the company has multiple product lines, corresponding apps, and different App versions used by users, correct.dsym files must be available for crash log analysis, and automation becomes very important for so many different versions of APPS.

There are two ways to automate this. Smaller companies or companies can add runScript script code to Xcode to automatically upload DSYM in Release mode.

Because we have a system on the front end, Can simultaneously manage iOS SDK, iOS App, Android SDK, Android App, Node, React, React Native engineering project initialization, dependency management, build (continuous integration, Unit) Test, Lint, unified hop detection), testing, packaging, deployment, dynamic capabilities (hot update, unified hop route delivery) all in one. You can insert the ability based on each stage, so you can upload.dsym files to qiuniuyun storage on the packaging machine after calling the package in the packaging system (the rules can be AppName + Version as key, value as.dsym files).

Many of today’s architecture designs are microservices, and why they are chosen is beyond the scope of this article. So the symbolization of crash logs is designed as a microservice. The architecture diagram is as follows

Description:

• As part of the overall monitoring system, the Symbolization Service is a microservice focused on crash Report symbolization.

• Receive the pre-processed crash report and DSYM index requests from the task scheduling framework, pull the corresponding DSYM from the seven cows, symbolically parse the crash report, calculate the hash, and send the hash response to the “Data processing and Task scheduling framework”.

• Receive the request containing the original crash report and DSYM index from APM management system, pull the corresponding DSYM from Seven Cows, make symbolic analysis of the crash report, and send the symbolized crash report to THE APM management system.

• Scaffolding cli a ability is called packaging systems, packaging building ability, can according to the characteristics of the project, select the appropriate packing machine (packing platform is to maintain a number of tasks, according to the characteristics of different task has been distributed to different packaging machine, can see depend on the task details page to download, compile, operation process, etc., Packaged products include binary packages, download QR codes, etc.)

The symbolic service is the product of a large front-end team in a large front-end context, so it is implemented by NodeJS (single thread, so in order to improve machine utilization, it is necessary to enable multi-process capability). The symbolic machine of iOS is dual-core Mac Mini, which requires an experiment to evaluate how many worker processes need to be opened for symbolic services. As a result, dual-process processing crash log is nearly twice as efficient as single-process processing, while four-process processing is not significantly more efficient than dual-process processing, which is consistent with the characteristics of dual-core MAC Mini. Therefore, open two worker processes for symbolic processing.

Below is the complete design

For a brief description, the symbolization process is a master-slave mode, with one master machine and multiple slaves. The master machine reads the cache of.dsym and crash results. The Data Processing and Task Scheduling Framework schedules the symbolization service (the two internal Symbolocate workers) to simultaneously get the.dsym file from Qiuniuyun.

The system architecture diagram is as follows

The content of the article is too long, divided into several chapters, please click to view, if you want to view the whole coherent, please visit here