IOS crash log online symbolization practice

What is symbolization? How to do local symbolization? Why do online symbolization?

The full text is 14328 words, and the expected reading time is 28 minutes.

First, what is symbolization?

In daily development, it is inevitable that applications will crash. Typically, crash logs we export directly from the user are unsymbolic or partially symbolic, and are a collection of hexadecimal memory addresses that are not readable. Unsymbolized or partially symbolized crash logs do little to resolve the flash back problem, as shown below:

Last Exception Backtrace:
0  CoreFoundation  0x1ca4cd27c 0x1ca3b5000 + 1147516
1  libobjc.A.dylib  0x1c96a79f8 0x1c96a2000 + 23032
2  CoreFoundation  0x1ca3ded94 0x1ca3b5000 + 171412
3  TestBacktrace  0x102a47464 0x102a40000 + 29796
4  UIKitCore  0x1f6c86e30 0x1f63d3000 + 9125424
Copy the code

Only the symbolized crash log shows function calls for individual threads, not just meaningless virtual memory addresses. Symbolic after the collapse of the log as shown below, at this point, we can know from the stack information directly applied TestBacktrace collapse when the function is [AppDelegate Application: didFinishLaunchingWithOptions:]. The file in which the function crashed was appdelegate. m on line 23:

Last Exception Backtrace:
0   CoreFoundation                  0x1ca4cd27c __exceptionPreprocess + 228
1   libobjc.A.dylib                 0x1c96a79f8 objc_exception_throw + 55
2   CoreFoundation                  0x1ca3ded94 -[__NSSingleObjectArrayI objectAtIndex:] + 127
3   TestBacktrace                   0x102a47464 -[AppDelegate Application:didFinishLaunchingWithOptions:] + 29796 (AppDelegate.m:23)
4   UIKitCore                       0x1f6c86e30 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 411
Copy the code

Two, symbolic principle

2.1 What are dSYM files?

On the iOS platform, a dSYM file is a target file with debugging information. The file name is usually xxx.app.dSYM, where XXX usually indicates the binary package name of the application program, as shown in the following figure:

Usually we can see dSYM files and directory structure in Xcode packaged file xcarchive:

DSYM stores information such as file name, method name, and line number, which corresponds to the address of the hexadecimal function of the executable file one by one. You can accurately know the specific crash information by analyzing the crash file.

DWARF(Debuging With Arbitrary Record Format) is the standard Format for storing and processing debugging information in file formats such as ELF and Mach-O. Data in DWARF is highly compressed and readable information can be extracted using commands such as dwarfdump and otool. For example, you can use commands to extract key debugging information, such as debug_info and debug_line

dwarfdump --debug-line /Users/xxxx/Desktop/resource/TestBacktrace.app.dSYM > debug_line.txt
Copy the code

Export debug_line information to the debug_line. TXT file. You can also export debug_info using similar commands.

ELF and Mach-O are the names of files used by the Linux and Mac OS platforms to store binaries, executables, object code, and shared libraries, respectively.

2.2 How do I Generate a dSYM File

During the compilation of engineering, the Debug mode will be selected by default doesn’t generate dSYM file, the configuration can be in the Build Setting | Build change in the Option, dSYM Release mode is generated by default. In addition, if bitcode optimization is enabled, Apple will do a secondary compilation optimization, so the final dSYM will need to be downloaded manually from Apple Connect. The dSYM generated by each compilation is different. Usually, a unique IDENTIFIER, called a UUID, is used to distinguish different dSYM files.

2.3 How can I Find a Matching dSYM Based on the UUID applied in crash Logs?

Restoring the crash stack requires that the dSYM UUID be the same as the application UUID at the time of the crash. Typically, each dSYM file has a UUID that corresponds to the UUID in the App file, representing an application. Each crash message records the application UUID, which can be checked against the dSYM UUID.

The UUID of TestBacktrace is as follows:

6 be881754f573769926b838490e39857.

Binary Images:
0x102a40000 - 0x102a6bfff TestBacktrace arm64  <6be881754f573769926b838490e39857> /var/containers/Bundle/Application/B44844E6-AFF4-491E-8168-F4ED93D644C2/TestBacktrace.App/TestBacktrace
0x102d14000 - 0x102d6bfff dyld arm64  <9c893b6aa3b13d9596326ef6952e7195> /usr/lib/dyld
Copy the code

2. Run the following command to query the UUID of the dSYM file. If the UUID in the dSYM file is the same as the UUID in the first step, delete the -.

Xcrun dwarfdump --uuid < dSYM file >Copy the code

3. If there are too many LOCAL DSYMs, it is too troublesome to check one by one. You can also use the MDFind name to find the dSYM on the local computer according to the UUID. For example, run the following command on the terminal.

mdfind "com_apple_xcode_dsym_uuids == 6BE88175-4F57-3769-926B-838490E39857"
Copy the code

2.4 Symbolic process

The process of converting the APP binary address in the crash log to a function is as follows:

The key lines of the crash log App are obtained

It can be seen from the figure above that the construction bank of APP is:

3 TestBacktrace 0x102a47464 0x102a40000 + 29796
Copy the code

Where TestBacktrace is our binary package name and the rest of the lines are the system stack.

Get the offset, runtime stack address, and runtime APP starting address

The key line information shows that the offset of TestBacktrace relative to the start address is 29796, the runtime stack address is 0x102a47464, and the runtime APP start address is 0x102a40000.

Obtain the dSYM start address

Start address of the TEXT segment in the symbol table saved in the dSYM file. You can run the following command to obtain the start address:

otool -l /Users/xxxxx/Desktop/TestBacktrace.app.dSYM/Contents/Resources/DWARF/TestBacktrace | grep __TEXT -C 5
Copy the code

It can be seen from the figure above that the starting address of the code segment in dSYM is 0x10000000.

Calculate the crash address corresponding to the address in the dSYM symbol table

IOS uses Address Space Layout Randomization (ASLR) when loading mach-O files for security. As a result, the starting Address of binary Mach-O files is different every time they are loaded into memory. The calculation rules for the starting address are the same; From the above we can get the formula 0x102a47464 (runtime address) = 0x102A40000 (start address) + 29796(offset). Therefore, based on the start address and offset of dSYM, you can calculate that 0x102A47464 corresponds to 0x100007464 = 0x0000000100000000 + 29296 in dSYM.

Get the specific function/line number/file

After obtaining the address 0x100007464 corresponding to the dSYM file, you can find the DIE(Debug Information Entry) unit containing the address in the DEBUg-info of the dSYM file. Commands can be used in Mac OS.

dwarfdump TestBacktrace.app.dSYM --lookup 0x100007464
Copy the code

Get the corresponding information, as shown in the figure:

DW_TAG_Subprogram indicates that the DIE unit represents the function method.
DW_AT_low_pc indicates that the method starts at 0x1000073b4.
DW_AT_high_pc indicates that this method ends at 0x1000074C4. This means that 0x100007464, the converted offset of 0x102a47464 in the crash log, is between DW_AT_low_p and DW_AT_high_pc.
DW_AT_name said our function called [AppDelegateApplication: didFinishLaunchingWithOptions:].
DW_AT_decl_file indicates that the file path where the function resides is appdelegate.m.
DW_AT_decl_line indicates that the function starts with 19 rows.

Assemble and format

Finally, after format optimization, the corresponding method of 0x102a47464 symbol in crash log is as follows:

3   TestBacktrace                   0x102a47464 -[AppDelegate Application:didFinishLaunchingWithOptions:] + 29796 (AppDelegate.m:23
Copy the code

3. Local symbolization

3.1 Symbolic method

Xcode symbolization

Put crash logs, dSYM files, and executables in the same directory, then drag and drop crash logs into Devicelog and right-click symbolicate Log or Re-symbolicate Log to symbolize it.

Use the symbolicatecrash command line symbolization

Locate the Symbolicatecrash script

The usual path for Symbolicatecrash is /Applications/Xcode.App/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash

Front-run command

Before running Symbolicatecrash, you generally need to run:

export DEVELOPER_DIR="/Applications/XCode.App/Contents/Developer"
Copy the code

Run the symbolicatecrash command

First copy the crash logs, dSYM, and Symbolicatecrash into the same folder, then CD it to the current folder and run the following command to symbolize it

./symbolicatecrash TestBacktrace-2021-07-30-135514.ips TestBacktrace.app.dSYM > symbol.log
Copy the code

3.2 Symbolizing system Logs

It is worth noting that sometimes the crash log does not contain App calls, but may contain all system library calls, as follows:

Thread 32 Crashed: 0 libobjc.A.dylib 0x19aaf6c10 0x19aad3000 + 146448 1 CFNetwork 0x187545a28 0x18737d000 + 1870376 2 Foundation 0x18808db4c 0x187f6c000 + 1186636 3 Foundation 0x187f8a908 0x187f6c000 + 125192 4 Foundation 0x18808fde8 0x187f6c000 + 1195496 5 Foundation 0x187f8a5c4 0x187f6c000 + 124356 6 Foundation 0x1880907e0 0x187f6c000 + 1198048 7 Foundation 0x1880902ac 0x187f6c000 + 1196716 8 libdispatch.dylib 0x1869863e4 0x186976000 + 66532 9 libdispatch.dylib 0x1869d7298 0x186976000 + 397976 10 libdispatch.dylib 0x18697c028 0x186976000 + 24616 11 libdispatch.dylib 0x18697b828 0x186976000 +  22568 12 libdispatch.dylib 0x186988bb8 0x186976000 + 76728 13 libdispatch.dylib 0x186989378 0x186976000 + 78712 14 libsystem_pthread.dylib 0x1cf2c5580 0x1cf2ba000 + 46464Copy the code

The symbolized log is as follows:

Thread 32 Crashed:0 libobjc.A.dylib _objc_release (in libobjc.A.dylib) 161 CFNetwork __CFNetworkHTTPConnectionCacheSetLimit (in CFNetwork) 1547282 Foundation ___NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__  (in Foundation) 163 Foundation -[NSBlockOperation main] (in Foundation) 1004 Foundation ___NSOPERATION_IS_INVOKING_MAIN__ (in Foundation) 205 Foundation -[NSOperation start] (in Foundation) 7846 Foundation ___NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ (in Foundation) 207 Foundation ___NSOQSchedule_f (in Foundation) 1808 libdispatch.dylib __dispatch_block_async_invoke2 (in libdispatch.dylib) 1049 libdispatch.dylib __dispatch_client_callout  (in libdispatch.dylib) 1610 libdispatch.dylib __dispatch_continuation_pop$VARIANT$mp (in libdispatch.dylib) 41211 libdispatch.dylib __dispatch_async_redirect_invoke (in libdispatch.dylib) 78412 libdispatch.dylib __dispatch_root_queue_drain (in libdispatch.dylib) 37613 libdispatch.dylib __dispatch_worker_thread2 (in libdispatch.dylib) 12014 libsystem_pthread.dylib __pthread_wqthread (in libsystem_pthread.dylib) 212Copy the code

It can be seen that the problem is caused by the wild pointer in the network request of CFNetwork, so we can make further investigation for the network related request.

It can be concluded that symbolic system libraries are necessary, especially for crash logs where App stack information is completely absent.

How to symbolize system library symbols

To symbolize the method name of your App, you need to compile the generated dSYM file. To convert system library symbols into complete method names, you also need symbolic files for each iOS system library.

Matches the corresponding symbol file version

Crash logs of users come from various system versions and need the system symbol files of corresponding versions to be symbolized. The system library symbol files are not generic, but correspond to the system version and CPU model of the device on which the crash occurred.

The crash log contains the following information:

Code Type: ARM-64 OS Version: iOS 10.2 (14C82)......... Binary Images: 0x102a40000 - 0x102a6bfff TestBacktrace arm64 <6be881754f573769926b838490e39857> /var/containers/Bundle/Application/B44844E6-AFF4-491E-8168-F4ED93D644C2/TestBacktrace.App/TestBacktrace 0x102d14000 - 0x102d6bfff dyld arm64 <9c893b6aa3b13d9596326ef6952e7195> /usr/lib/dyldCopy the code

Code Type Indicates whether the CPU architecture of the device is ARMV7, ARMV7S, ARM64, or ARM64E.

OS Version indicates the system Version number of the device. The string in parentheses indicates the build number of the device.

9 c893b6aa3b13d9596326ef6952e7195 Binary Images of < > inside the character representation of the corresponding system library dyld UUID, only build + UUID matching system library symbol files to symbolic symbols.

Puts the symbol file in the specified location

To access to the corresponding version of the symbol file into the Mac OS ~ / Library/Developer/Xcode/iOS DeviceSupport directory, you can use the Xcode’s own symbolic tool symbolicatecrash symbolization. This tool automatically searches for the symbol files of the native system library based on its UUID in the crash log.

3.3 Two methods of obtaining system symbol files

Get it from the real machine

Most of the system library symbol files are only available on real phones, and Apple does not provide a direct download address. But when you use Xcode to first connect to a device for real debugging, you’ll see that Xcode displays Processing Symbol Files, This is in the copy machine really symbol files to Mac OS/Users/XXX/Library/Developer/Xcode/iOS DeviceSupport directory.

The corresponding symbol files are folders like 14.7.1 (18G82) in the directory and are usually 1-5GB in size.

Extract symbol files from firmware

There are several ways to extract system library symbol files from firmware (iPSW). Firmware decryption can be divided into two steps: downloading and extracting system symbols and extracting system library symbols.

1. Download and extract system symbols

IOS9 and before iOS9

A. Download the iPSW firmware of the corresponding version and decompress it directly. After decompression, several DMG image files are found.

B. Locate the decryption key (the key of the Root Filesystem field on the page) in the Firmware_Keys command.

C. Use the DMG tool for decryption. CD go to the iPSW folder and run./ DMG extract XXX-XXXX-xxx. DMG dec.dmg -k. Extract extract (DMG); extract (DMG); extract (DMG); If the key is incorrect, decryption will fail. Decryption success will generate a dec.dmg file, double click to open to load the system image.

IOS10 and after iOS10

Download the iPSW firmware of the corresponding version and decompress it directly. After decompression, there are several IMAGE files in DMG format. The largest DMG file is the system image.

2. System library symbol extraction

Starting with iPhone OS 3.1, all system libraries are packaged into a single file:

Dyld_shared_cache_xxx, XXX said specific architecture, this file is located in the: / System/Library/Caches/com. Apple. Dyld directory.

The dyLD_SHARED_cache_xxx file can be decompressed using the dsc_extractor. CPP code in dyld, but with some modifications.

A. Download the source code of dyLD library from Apple’s open source website. Note that you need to download the source code of DYLD-7.

B. After downloading, change the code before and after dsc_extractor. CPP and main functions to the following code:

#if 1
// test program
#include <stdio.h>
#include <stddef.h>
#include <dlfcn.h>
typedef int (*extractor_proc)(const char* shared_cache_file_path,const char* extraction_root_path,void (^progress)(unsigned current,unsigned total));
int main(int argc, const char* argv[]){
if ( argc != 4 ) {
fprintf(stderr,"usage: dsc_extractor <dsc_extractor.bundle path> <path-to-cache-file> <path-to-device-dir>\n");
return 1;
    }
void* handle = dlopen(argv[1],RTLD_LAZY);
if ( handle == NULL ) {
fprintf(stderr,"dsc_extractor.bundle could not be loaded\n");
return 1;
    }
    extractor_proc proc = (extractor_proc)dlsym(handle,"dyld_shared_cache_extract_dylibs_progress");
if ( proc == NULL ) {
fprintf(stderr,"dsc_extractor.bundle did not have dyld_shared_cache_extract_dylibs_progress symbol\n");
return 1;
    }
int result = (*proc)(argv[2],argv[3],^(unsigned c, unsigned total) { printf("%d/%d\n", c, total); } );
fprintf(stderr, "dyld_shared_cache_extract_dylibs_progress() => %d\n",r esult);
return 0;
}
#endif
Copy the code

C. On the terminal, CD to the DYLD source directory launch-cache, compile and generate the DSC_Extractor tool on the terminal command line.

clang++ -o dsc_extractor dsc_extractor.cpp dsc_iterator.cpp
Copy the code

D. from Xcode package/Applications/Xcode. The App/Contents/Developer/Platforms/iPhoneOS. The platform can be derived from the/usr/lib dsc_extractor. The bundle files. Dsc_extractor. bundle is strongly associated with the iOS system to be extracted. For example, iOS14 system symbols need to be exported from Xcode12 dsc_extractor.bundle, while iOS15 requires Xcode 13 Beta. If they do not match, the system symbol may not be extracted.

E. Run the following command to extract system symbols: As follows, the system library finally extracted is under the directory 17C81. The files we need to parse system symbols are dylib and framework.

dsc_extractor  dsc_extractor.bundle   /System/Library/Caches/com.Apple.dyld/dyld_shared_cache_arm64   17C81
Copy the code

Four, online symbolization

4.1 Why to realize online symbolization

At packaging time symbol files are generated by continuous integration baler and there is a cost to obtain locally.
Convenient for developers to quickly symbolize the crash log. A lot of times crashes happen when non-developers (production, QA, etc.) are using the application; After synchronization to the developer, the developer needs to be able to symbolise the crash stack quickly without the packaging environment because of local environment differences
The crash logs uploaded by online users are large. Procedure Most crashes occur during user use after release. If a large number of online logs are synchronized to the developer without being symbolized, it will increase the burden of the developer and reduce the efficiency of problem solving.
There are many user systems and it is difficult to collect. Users are running iOS9 through iOS14, and it’s so weird that it’s a fantasy for developers to be able to parse all the symbols locally.

4.2 Online App/dynamic library symbolization

Normally, we only need to symbolize a very small number of crash logs, in which case we can symbolize them locally. However, once our app was up and running and crash log collections could be in the millions or more per day, it wasn’t appropriate to use scripts/tools to symbolize a single log on Mac OS (symbolicatecrash takes more than a second on Mac OS). At this point, a more general, fast way of symbolizing is needed.

In order to speed up the symbolization of iOS crash logs on Linux servers, we deeply investigated the principle of iOS local symbolization, and finally adopted the following solution after many times of research and consultation with the platform on technical solutions:

Generating a Mapping File

Extract dSYM files through scripts to generate a mapping file in the following format:

Format: Mach-O/ 64-bitarch: arm64Symbols: 5Tool Version:: 1.0.1File Version: 1.0.0UUID: e569d81abb2c372e89a2410edc3d368fBuilt Time: 2021-07-29 13:31:08Symbol table:6c64 6c78 -[ViewController viewDidLoad] (in TestBacktrace) (ViewController.m:17)6c78 6c84 -[ViewController viewDidLoad] (in TestBacktrace) (ViewController.m:0)Copy the code

The extraction operation involves the symbolization of debug_line segment data in DWARF, and the relevant extraction algorithm can be referred to the official data of DWARF. The debug_line segment contains the detailed code offset address and file name, which can be parsed according to DWARF algorithm and then matched with Symbol Table function symbols one by one to generate the mapping relationship between code address offset and function, file and line number. It should be noted that Most of Apple’s Mach-O formats now use DWARF2 and DWARF4 versions, and the compatibility and algorithm differences between the two formats need to be paid attention to when extracting. Finally, you can see that each row of the Symbol table corresponds to the offset of a Symbol.

You can see 7464 falls between 7454 and 7478, Match the symbol is just – [AppDelegate Application: didFinishLaunchingWithOptions:] (in TestBacktrace) (AppDelegate. M: 23), This is consistent with the symbolicatecrash script used on Mac OS.

Symbolized according to the Mapping file

With the help of the symbol mapping file extracted by the script tool, the server can get away from the platform limit and match the mapping file according to the UUID in the crash log, so as to speed up the symbolic crash log on Linux and provide efficient real-time symbolic service.

4.3 Online Symbolization of iOS System Library Symbols

On The Mac OS platform, we can use the system library symbols directly and use scripts to de-symbolize symbols, but when it comes to symbolizing all user-uploaded crash logs, this mechanism is bound by speed and platform limitations. And with hundreds of versions released in iOS from 2.0 to iOS 14, it’s almost impossible to manually extract system library symbols. In order to solve this problem, after referring to the dSYM cross-platform symbolization scheme, we made a set of system symbol automation symbolization scheme, and finally realized the efficient real-time symbolization system stack on Linux platform.

1. Regularly export the iPSW file download address of each system and the latest release system from theiphoneWiki.

2. Decompress the iPSW and load the system image DMG file to find the dyLD_SHARED_cache_xxx file.

3. Use the tool dsc_extractor to export system library symbol files, which are basically mach-O files with suffixes dylib and framework.

4. Use a tool to extract all Dylib and Framework and generate a mapping file in the following format. This step and dSYM extraction operation will have a certain difference, generally speaking, the system library only symbol table segment, do not need to do the extraction of debug_LINE segment, relatively simple.

20b8    20fa    +[ZoomServicesUI enableZoomServices]    (in AccessibilitySettingsLoader)20fa    2120    +[ZoomServicesUI disableZoomServices]   (in AccessibilitySettingsLoader)2120    21b6    -[ZoomServicesUI init]  (in AccessibilitySettingsLoader)21b6    2222    -[ZoomServicesUI dealloc]   (in AccessibilitySettingsLoader)
Copy the code

5. After the crash log is uploaded to the symbolized server, the server uniquely identifies the mapping file based on the UUID of the system library in the crash log and the UUID in the mapping file and symbolizes the system stack.

Five, the effect of

1. After the online symbolic system goes online, user logs are automatically uploaded to the performance by the crash component, and the symbolic parsing system directly symbolizes and clusters the crash logs. The final symbolic crash log details are as follows. The address of the application is finally shown as function + file + line number, and the system stack concretely displays the crashed function, in real time and efficiently.

Last Exception Backtrace:
0   CoreFoundation                  0x1ca4cd27c __exceptionPreprocess + 228
1   libobjc.A.dylib                 0x1c96a79f8 objc_exception_throw + 55
2   CoreFoundation                  0x1ca3ded94 -[__NSSingleObjectArrayI objectAtIndex:] + 127
3   TestBacktrace                   0x102a47464 -[AppDelegate Application:didFinishLaunchingWithOptions:] + 29796 (AppDelegate.m:23)
4   UIKitCore                       0x1f6c86e30 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 411
5   UIKitCore                       0x1f6c88594 -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 3351
6   UIKitCore                       0x1f6c8dd20 -[UIApplication 
Copy the code

Thread 0 name: Dispatch queue: com.Apple.main-thread Thread 0 Crashed: 0 libsystem_kernel.dylib 0x00000001ca06a0dc __pthread_kill + 8 1 libsystem_pthread.dylib 0x00000001ca0e3094 pthread_kill$VARIANT$mp + 380 2 libsystem_c.dylib 0x00000001c9fc3ea8 abort + 140 3 libc++abi.dylib 0x00000001c9690788 __cxa_bad_cast + 0

2. Part of the crash logs are from the research and development and testing. The platform provides an online symbolized entrance.

Six, earnings

1. The online problem location speed has been greatly improved, from the occurrence of new online stuck/crash problem to the specific research and development response time is greatly reduced, from the occurrence of crash to the location of the problem, basically within 10 minutes.

2. At present, the performance platform has more than one million daily online symbolic crash/stall logs, and more than 30+ product lines are connected to the factory. The symbolic function can be uploaded and analyzed immediately, without the intervention of r&d personnel in the whole process. Truly achieve automation, online, real-time symbolic crash, stalled log, and real-time according to the symbolic problem code to locate specific developers, efficient response and solve online problems.

References:

Necessary: [1] iOS Crash analysis method of symbolic system library zuikyo. Making. IO / 2016/12/18 /… Talk about extracting system library symbols from iOS firmware crash.163.com/#news/! News… [3] Xcode and symbols on several Settings www.jianshu.com/p/11710e7ab… IOS_SDK en.wikipedia.org/wiki/IOS\_S… IOS_version_history en.wikipedia.org/wiki/IOS\_v… [6] dyld source opensource.apple.com/tarballs/dy download address… The DWARF was Debugging Standard www.dwarfstd.org/\ \ [8] before iOS9 Firmware_Keys www.theiphonewiki.com/wiki/Firmwa… [9] DMG tool download address github.com/Zuikyo/iOS-… Symbols to download address index wiki www.theiphonewiki.com/wiki/Firmwa…

Job Information:

Senior/Senior engineer of mobile terminal

We are baidu APP mobile technology platform team, responsible for Baidu APP performance optimization, network optimization, architecture upgrade, engineering efficiency improvement and new technology exploration, etc. We are deeply engaged in online and offline research and development tools, intelligent performance optimization, network library development, network monitoring, compilation and construction systems, dynamic solutions and other aspects. In terms of technical depth, we are a beacon of mobile terminals. We support rapid business iteration while ensuring the r&d efficiency of our very large team, serving more than 600 million MAU users.

We are committed to creating the best performance APPs in the industry, with excellent performance stability and fluency on high and low end phones.

We are committed to building a mobile technology platform suitable for Baidu, enabling Baidu’s mobile products, smart speakers and vehicle-mounted equipment.

Technology stack: Java/OC/C/C++/Kotlin/Swift/Ruby/PHP etc

Support services: Baidu APP search, Feed, small programs, live broadcast, video, etc. Baidu Tieba, watch videos and other mobile products; IOT devices such as small speakers; Intelligent transportation, Internet of vehicles and other on-board equipment.

Education requirement: Bachelor’s degree

If you have expertise in a certain direction, technical depth; If you love research, challenge difficult problems; If you love to break through innovation and solve various problems with new technologies such as terminal intelligence, this is the stage for you to display your talent!

Mailing Address:

[email protected](Post notes) _