This is the full text of “How to Systematically Manage iOS Stability Issues” delivered by Feng Yadong at ArchSummit global Architects 2021
First of all, I would like to introduce myself: I am Feng Yadong. I joined ByteDance in April 2016 and have been responsible for the engineering architecture, basic library and experience optimization of Toutiao App. Since December 2017, I have been focusing on APM. From 0 to 1, I have participated in the construction of BYteDance APM platform and served the whole series of bytedance products. Currently, I am mainly responsible for the performance stability monitoring and optimization of iOS terminal.
I. Classification of stability problems
Before we get into categories, let’s take a look at the background: We all know that flash backlogging is the worst bug a user can encounter with a mobile app, because after the flash backlogging, the user can no longer use the product, and then the subsequent retention and business value of the product itself are not discussed. Here are some numbers I’d like to share with you: For 20% of mobile users, flashbacks are the most annoying problem, second only to inappropriate ads. Of the users who lose because of experience problems, 1/3 will switch to competing products, which shows that the flashback problem is very bad and serious.
Ii. Methodology for governance of stability issues
On the right side of the figure are two important governance principles we summarized: The first is to control the increase and control the stock. Generally speaking, the new stability problems may be some easy to break out, the impact is more serious. The stock problem is relatively difficult, and the repair cycle is long. The second is easier to understand: first hurry, then slow; first easy, then difficult. We should prioritize fixing the problems that break out and the ones that are relatively easy to fix.
If we want to manage the stability problem well, we need all r &D students to pay attention to each of the above links to achieve the final goal. But what is the point of all this? From byteDance’s experience in problem management, we believe that the most important step is the second one — attribution of problems online. According to the internal statistical data, it is found that the reason why there is no conclusion for a long time and no way to fix the problems online is mainly because the RESEARCH and development failed to locate the root cause of these problems. So the next chapter is also the focus of this sharing: difficult problem attribution.
3. Attribution of difficult problems
We rank these issues in order of developer familiarity: Crash, Watchdog, OOM, and CPU&Disk I/O. I’ll share background and solutions for each of these problems, and demonstrate how various attributional tools can solve these problems in combination with real-world examples.
3.1 The first kind of difficult problem — Crash
- First of all, unlike OC and C++ exceptions, the developer may get a crash call stack is a pure system call stack, which is obviously very difficult to fix;
- In addition, some Crash problems may be accidental rather than inevitable, and it is very difficult for developers to reproduce the problems offline, because they cannot reproduce, it is difficult to troubleshoot and locate these problems through IDE debugging.
- In addition, crashed call stacks may not be the first scene for problems such as illegal address access. Here is A simple example: The memory allocation of service A overflows and the memory of service B is trampled. At this time, we believe that service A should be the main cause of the problem, but it is possible that service B will use this memory at A later time and crash. Obviously, this kind of problem is actually caused by A business, but finally collapsed in the call stack of B business, which will bring great interference to the developer’s troubleshooting and solving this problem.
Seeing this, you may have a question in mind again: since such problems are so difficult to solve, is there no way at all? Not really, but I’m going to share two attributions tools inside bytes that are very useful for solving these problems.
3.1.1 Zombie detection
The first one is Zombie detection, which if you’ve used Xcode’s Zombie monitoring, you’re probably familiar with. If we turn Zombie Objects on before debugging, and we run a crash caused by an OC object wild pointer, the Xcode console prints a log that tells the developer which object crashed when calling what message.
Zombie is an OC object that has been released. What’s the attribution advantage of Zombie monitoring? First, it can directly locate the class where the problem occurred, rather than some random crash call stack. In addition, it can improve the recurrence rate of occasional problems, since most of the occasional problems are likely to be related to a multi-threaded runtime environment. If we can turn an occasional problem into a mandatory problem, then developers can easily troubleshoot the problem with the IDE and debugger. However, this solution has its own scope, because its underlying principle is based on the OC Runtime mechanism, so it only applies to memory problems caused by the OC object field pointer.
The MainTabbarController is generally the root view controller of the home page and should not be released in theory for its entire life cycle. Why does it become a wild pointer object? As you can see, this simple error message is sometimes not enough for developers to locate the root cause of the problem. So here we go one step further and extend the functionality to report the call stack information when a Zombie object is released.
3.1.2 Coredump
As mentioned earlier, the Zombie monitoring scheme has some limitations. It only applies to the problem of wild Pointers to OC objects. Again, you might wonder: C and C++ code can also have wild pointer problems. In Mach and Signal exceptions, there are many other types of exceptions besides memory problems such as EXC_BAD_INSTRUCTION and SIGABRT. So how do we solve the other difficult problems? Here we present another solution — Coredump.
What is the attribution advantage of the Coredump scheme? First, because it is a file format defined by LLDB, it naturally supports LLDB instruction debugging, which means that developers can implement post-debugging of online difficult problems without having to reproduce the problem. In addition, because it has all the memory information at the time of the crash, it provides a huge amount of problem analysis material for developers.
This scheme can be used to analyze any Mach exception or Signal exception.
As you can see, the crash call stack is also full of system library methods and eventually crashes a method in the libDispatch library with an exception type that hits a system library assertion.
Here we finally figure out that the x0 sender of the crash thread’s stack frame 0 (the first line calls the stack) is actually the queue structure information defined in libDispatch. At its starting address offset 0x48 bytes, this is the label attribute of the queue (which can be simply interpreted as the queue name). The name of the queue is critical to us, because to fix the problem, we need to know which queue is causing the problem. We through the memory read instructions directly read the memory of information, it was subsequently found to be a C string, name is com. Apple. CFFileDescriptor, this information is critical. We searched the source code globally for this keyword and found that the queue was created in the network library at the bottom of the byte, which explains why all byte products have this crash.
3.2 The second type of difficult problem — Watchdog
Let’s move on to the second category of difficult problems — Watchdog.
First of all, the problem of deadlock usually occurs in the cold startup stage when users open the App. Users may wait for 10 seconds and do nothing, and the App will crash, which will do great harm to user experience. In addition, our online monitoring found that if there is no treatment for the problem of deadlock, its magnitude may be 2-3 times that of ordinary Crash. In addition, currently the industry is generally monitoring OOM crash is the elimination method, if not excluded from the jammed crash, the corresponding will increase the probability of OOM crash misjudgment.
What are the attributional difficulties of stuck problems? Firstly, it is based on the traditional scheme — the stalled monitoring: the main thread is considered to be dead after the non-response time exceeds 3 ~ 5 seconds. This traditional scheme is very prone to false positives. As for the reason of false positives, we will talk about it in the next page. In addition, the cause of the deadlock can be very complex, and it is not necessarily a single problem: the main thread deadlock, lock wait, main thread IO and other causes can cause the deadlock. The third point is that deadlocks are a common cause of stuck problems. The threshold of deadlock analysis in traditional schemes is relatively high, because it strongly depends on the experience of developers. Developers must rely on manual experience to analyze which or which threads the main thread is waiting for each other to cause the deadlock, and why the deadlock occurs.
In view of the above mentioned pain points, we provide two solutions: first, the main thread call stack can be captured for many times when the monitoring is stuck, and the thread state of the main thread at different times can be recorded. The information about the thread state will be mentioned in the next page. In addition, we can automatically identify the deadlock caused by deadlocks, identify such problems, and help developers automatically restore the lock wait relationship between threads.
On the right is an explanation of the thread’s running state and thread flags. When we see the thread status, we have two main analysis ideas: first, if we see the main thread CPU usage is 0, the current waiting state, has been swapped out, then we have reason to suspect that the current deadlock may be caused by deadlock; On the other hand, the CPU usage of the main thread is always high and in the running state, so it should be suspected that the main thread has some cpu-intensive tasks such as endless loops.
The figure above lists some of the lock waiting methods we’ve covered so far, including mutex, read/write, spin, GCD, and so on. Each lock wait method defines an argument that passes in information about the current lock wait. We can read the lock wait information from the register and force it into the corresponding structure. Each structure will define a thread ID attribute, indicating which thread is waiting for the lock release. After completing this sequence of operations for each thread in the wait state, we can obtain the lock wait relationship of all threads and construct the lock wait relationship diagram.
You can see that the main thread here we’ve labeled as a deadlock, it has zero CPU usage, it’s in a wait state, and it’s been swapped out, which is consistent with the way we used to analyze thread state.
3.3 The third type of difficult problem — OOM
OOM is an app that uses too much Memory and crashes when it is forcibly killed.
So what are the attribution difficulties of OOM problem? The first is that the composition of memory is very complex, and there is no very clear information about the exception call stack. Also, we have offline tools to troubleshoot memory problems, such as Xcode MemoryGraph and Instruments Allocations, but these offline tools do not work for online scenarios. For this same reason, it is very difficult for developers to simulate and reproduce online OOM problems offline.
Here’s a refresher on the basics of MemoryGraph online: First of all, we will regularly detect the physical memory usage of App. When it exceeds the danger threshold, memory dump will be triggered. At this time, THE SDK will record the symbolized information of each memory node and the reference relationship between them. This strong and weak reference relationship will be reported at the same time, and finally the overall information reported to the background, can assist the developer to analyze the large memory usage and memory leaks and other abnormal problems.
Here’s a practical example of how MemoryGraph solves OOM problems.
An example of MemoryGraph file analysis is shown above, where the red boxes indicate the different areas: in the upper left corner is a list of classes, which summarizes the number of objects of the same type and how much memory they occupy; On the right is the address list of all instances of the class, on the lower right is an area where the developer can manually trace the references to the object (which other objects the object is currently referenced by, and which other objects it references), and in the middle is the reference diagram.
If we look at the class list, we can see that there are 47 objects of ImageIO type, but these 47 objects take up over 500 MB of memory, which is obviously not a reasonable memory usage. We click on the list of ImageIO classes, take the first object as an example, and trace its references. We discovered that this object had only one reference, the VM Stack: Rust Client Callback, which is actually the Rust Network library thread at the bottom of the fly book. After checking in here, you must wonder: Do all 47 objects have the same reference relationship? Here we can use the add Tag function in the lower right corner of the path tracing to automatically filter whether all 47 objects have the same reference relationship. As you can see in the upper right corner of the image, after filtering, we confirm that 100% of the 47 objects have the same reference relationship.
Let’s look at the VM Stack: Rust Client Callback object. It is found that two of the objects it references are very sensitive, one is ImageRequest and the other is ImageDecoder. From these two names we can easily infer that it should be the object of the ImageRequest and ImageDecoder.
Finally, together with the students of Feishu Photo Library, we located the cause of this problem: it is not a reasonable design to request 47 images simultaneously and decode them. The root cause of the problem is that the fetobook library’s downloader relies on NSOperationQueue for task management and scheduling, but is not configured with a maximum number of concurrent tasks, which can cause excessive memory usage in extreme scenarios. The corresponding solution is to set the maximum concurrency for the image loader and adjust the priority based on whether the image to be loaded is in the viewable area.
3.4 Type 4 Troubleshooting – CPU and disk I/O exceptions
The reason why these two types of problems are combined here is that they are highly similar: first, they are abnormal appropriation of resources; In addition, they are different from flash backdowns. The cause of the crash is not a moment, but a period of time of abnormal resource occupation.
What are the attributional difficulties of this kind of problem? The first is that it’s a very long duration, so there’s probably no single cause; It is also difficult for developers to reproduce such problems offline due to the complex use environment and operation path of users. In addition, if the App wants to monitor and attribute such problems in the user mode, it may need to sample the call stack information at a high frequency for a period of time. However, this monitoring method obviously has very high performance loss.
The screenshot on the right of the above picture is taken from a session of WWDC2020 by me. For such problems, Apple officially offers some suggestions for attribution schemes: Firstly, Xcode Organizer, which is the problem monitoring background provided by Apple officially. Then there is the suggestion that developers can also access MetricKit, which has diagnostic information about CPU anomalies.
Apple’s official documentation, on the right, also gives advice on attribution for these types of issues. There are also two suggestions: one is to rely on Xcode Organizer and the other is to rely on MetricKit. In the selection process, we finally decided to adopt MetricKit scheme, mainly considering that we still want to control the data source in our own hands. Because Xcode Organizer is, after all, a black box background of Apple, we cannot get through with the background of the group, and it is not convenient to build follow-up processes such as alarm, automatic problem allocation, and issue status management.
Finally, we located the cause of this problem together with the students of Feishu: there was an animation in the small program business of Feishu that did not pause when it was hidden, resulting in a high CPU usage. The solution is as simple as pausing the animation while it’s hidden.
Iv. Summary and review
