preface
Toutiao iOS app has been concerned about the problem of package size since 2016, and has started package size optimization. In 2017, we published our experience at that time as a technical article “Dry Goods | Toutiao iOS installation package size optimization — Ideas and Practice” [1].
Now three years have passed. Toutiao continues to explore package size optimization with more ideas, including build configuration, image compression, __TEXT segment migration, binary segment compression, and more. These optimizations have brought significant package size benefits to Toutiao with less business intrusion. At the same time, the industry is producing more solutions for package size optimization. Therefore, we update this article and look forward to discussing package size optimization with you.
I. Composition of the installation package
After we obtained an IPA file with App Slicing through construction, we decompressed it with zip and entered the.app file, and we could see the contents of the installation package intuitively.
An installation package usually contains resources and an executable file on iOS called Mach -o. Resources can be divided into Assets. Car and other resources. Among them, the Assets. Car file and the Mach-O file are the parts that we put much effort into optimizing.
1.1. Assets. Car file
The assets. car file is a build product of the Asset Catalog in the project. Actool in the Xcode tool chain is responsible for building Assets.car. During the process of building Assets.car, Actool will select an encoding algorithm according to a certain policy and re-encode the PNG image.
1.2. Mach-o files
A Mach-O file is an executable on iOS that is compiled and statically linked from a code source file. Mach-o files with App Slicing tend to have a single architecture. Using tools such as MachOView, we can get an intuitive idea of what is involved in Mach-O.
At the same time, the Link Map file can further help us analyze the composition of The Mach-O file.
LD_GENERATE_MAP_FILE in Build Settings generates a TXT file called Link Map that shows the distribution and size of each section, section, and function in Mach-O. This information is often used in package size optimization.
Second, resource size optimization
“Compact resources” is often the most commonly associated package size optimization solution, but there are tricks involved in practice. Toutiao has made many attempts in resource optimization.
2.1. Use appropriate resource compression configuration
The lowest version of iOS currently supported by Toutiao is iOS 9. However, the deployment_target (the lowest supported version) specified in the Podspec file of most Pod libraries remains in iOS 8 due to untimely changes, This results in the fact that resource_bundles specified in these Pod libraries are built with iOS 8 as the minimum supported version of Assets.car.
We found through experiments that:
1. Upgrade the Pod library and main project from iOS 8.0 to iOS 9.0
2. Enable the Pod library and ASSETCATALOG_COMPILER_OPTIMIZATION space in the main project Xcode Build Settings
These two Settings can change the encoding compression algorithm selected by Actool when building Assets.car to reduce the package size. We can use the xcrun assetutil –info Assets.car command to examine the encoding compression algorithm used for each image in Assets.car. In the context of Toutiao, the results are as follows:
Due to a change in the encoding compression algorithm for PNG images in Assets.car, these two configurations gained a package size benefit of 2.31MB when Toutiao landed.
2.2. Use RGB with Palette to compress the image
In the early stage of toutiao’s package size optimization, we tried to do lossless compression of PNG images in The Asset Catalog. However, after practice, we found that although the size of images put into the Asset Catalog was significantly reduced, the size of the built products was almost unchanged.
After exploration, we found that in Xcode, actool, the tool for building Asset Catalog, would first decode PNG images in Asset Catalog to obtain Bitmap data. And then use actool’s coding compression algorithm for coding compression processing. Lossless compression reduces the size by changing the image’s encoding compression algorithm, but does not change the Bitmap data. For Actool, the input it receives does not change, so lossless compression cannot optimize the size of assets.car.
Is there any other compression method that can optimize the size of Assets.car? We suspect that appropriate lossy compression of images is one idea.
So we tried RGB with Palette coding [2]. The resulting byte stream encoded in RGB with Palette first maintains a color array. Each member of the color array maintains a color with four RGBA components. For each pixel in the image, the subscript of the stored color array represents the color of the point. The type and number of colors maintained by the color array are determined by the image, and the maximum number of colors maintained by the color array can be artificially limited. The default is a maximum of 256 colors. This code is exactly what its name means: Palette.
Although most of the pictures in the App use a variety of colors, most of these colors are very close and it is difficult to distinguish visually, such as a large number of flat style ICONS. This type of image is very suitable for lossy compression by using the method of palette coding and reducing the size of color array, which can reduce the number of colors to achieve lossy compression and ensure that the retained colors are close to the original image, so that the quality of the image after lossy compression is not damaged. We landed on Toutiao and got a 3.15MB package size gain.
In the implementation, we used the ImageOptim tool to change the encoding mode of the image to RGB with Palette:
imageoptim -Q --no-imageoptim --imagealpha --number-of-colors 16 --quality 40-80 ./1.png
Copy the code–number-of-colors controls the number of colors maintained in the array; Quality Controls the quality of the image to change to the original percentage. Our experience shows that when –number-of-colors is adjusted upwards from 16, –quality remains 40-80, which can significantly reduce package size while maintaining an invisible quality change. After the pixel eye examination of UI students, it is confirmed that the pictures before and after optimization look the same.
2.3. Merge Assets. Car
Component integration is performed using CocoaPods. The Asset Catalog files carried by each component are imported as resource_bundles in Podspec. The installation package will be presented as an Assets. Car file under the Bundle.
Take version 7.9.4 as an example, there are 106 bundles containing assets. car:
The assets. car file is essentially a BOM file. At the same time, when Xcode uses Actool to construct an asset. car file, it also has some optimization operations, such as automatically merging several small pictures into a Packed Image. Therefore, combining several Assets.car can reduce repeated BOM blocks and maximize the optimization effects of Actool.
During the Build process, Toutiao adds scripts in Build Phases to merge images from Asset Catalog in multiple libraries into one Asset Catalog, which is then constructed into Assets. Car product by Actool. This optimization resulted in a 2.1MB package size gain. At the same time, theoretically, this optimization can also reduce the parsing operation of Assets. Car at runtime, and the response time of picture reading has a positive benefit.
2.4. Text file compression
In addition to the largest image resources, Toutiao installation package also has many text file resources, such as JSON files, HTML files, etc. Compression of these text files also results in package size optimization.
The text file compression scheme implemented by Toutiao consists of three parts:
1. Compression Phase: add scripts in Build Phase and zip compress text files in the dialogue list during construction;
2. Decompression stage: In the startup stage of App, decompression operation is carried out in asynchronous thread and decompression products are stored in sandbox;
3. Reading stage: When App runs, hook reads these files by changing the reading path from Bundle to the corresponding path in sandbox;
This scheme can achieve compression optimization under the premise of less service intrusion. We first applied this scheme to JSON files for Lottie animations, yielding a 400KB package size gain. This scheme can be extended to more types of files in the future.
3. Mach-o file optimization
While optimizing resources, we also noticed that Mach-O files consistently accounted for about 80% of toutiao’s installation package. Optimization of Mach-O files is essential. Let’s take a look at the Mach-O file optimizations we landed on in chronological order.
3.1. Use -oz compilation parameters
Oz is a compilation optimization option added to Xcode 11. Oz was featured in WWDC 2019’s What’s New in Clang and LLVM [3]. The core principle of Oz is to reuse the function of the repeated continuous machine instructions, and the principle of “inline function” is just the opposite. Thus, enabling Oz reduces the size of the binary, but at the same time theoretically incurs an additional cost in execution efficiency. Performance (CPU) sensitive code usage needs to be evaluated.
Apple gives a reference figure of 4.5% by volume.
After evaluating execution efficiency, stack resolution, stability, and compilation speed, we turned on Oz compilation for most of the source code and reduced package size by more than 4MB.
3.2. Optimize LTO when using links
Link-time Optimization is a compilation/linking parameter that comes with Xcode. According to WWDC 2016 “What’s New in LLVM” [4], LTO has a positive impact on package size and operating efficiency. After Incremental LTO was enabled for both compilation and linking, the package size decreased by 6.5MB.
3.3. Modify the Exported Symbols configuration
The EXPORTED_SYMBOLS_FILE configuration in Xcode Build Settings controls Export Info in the __LINKEDIT section of Mach-o. Dynamic linker DYLD reads the Export Info information of the bound dynamic library or executable file to get the actual calling address corresponding to a symbol when performing symbol binding. If the symbol being bound is missing from the Export Info of the target dynamic library, DyLD will throw an exception, indicating that the App has crashed.
In principle, though, the information in Export Info is indispensable. However, for a Mach-O file, not all symbols need to be exposed to other dynamic libraries or executables. Ideally, private symbols should be coded with __attribute__((visibility(hidden))). However, in cases where it is difficult to add modifiers one by one to historical code, the Exported Symbols configuration gives the project an opportunity to maintain a whitelist of public Symbols. If a valid EXPORTED_SYMBOLS_FILE configuration is filled in, dynamic libraries or executables will remove symbols that are not whitelisted during static linking, reducing package size and increasing reverse difficulty.
Using Exported Symbols reduces the package size by 2.1MB.
3.4. Dynamic properties
Attributes are one of the most common concepts in OC. However, a property is not as small as we think. Analyzing the Mach-O file, we found that a property can be divided into three parts:
(1) Member variable: The essence of a member variable is a structure with a size of 32B. The size of the contents pointed by the three Pointers (Offset, Name and Type) in the structure are 8B, 10B and 10B respectively. The size of the contents pointed by the Pointers of Name and Type is related to the Type and Name length of the member variable. The total size is about 60B.
@interface presentViewController(a)
@property (nonatomic.strong) UIImageView *imageView;
@property (nonatomic.strong) UIButton *button;
@property (nonatomic.strong) NSString *name;
@end
Copy the code
(2) Automatically generated set/ GET method part: the essence of set/ GET method is a structure with a size of 24B, which contains three Pointers Name, Type and Implementation. The size of the content pointed to is about 10B, 10B and 20B. A method is about 64B in size, and the set and GET methods are 128B.
(3) Property part: The essence of the property is still a structure, the size is 16B, the structure of the two Pointers to the size of the content is about 10B, 10B, and related to the name and type of the property. The total size is about 36B.
That is, the package size of an attribute is about 224B.
If we modify a property with @dynamic and do not generate member variables, get/set methods, then a property can be reduced from 224B to 36B, containing only the size of the property part.
At the same time, there are a number of JSONModel subclasses automatically generated by scripts in the code, and these subclasses tend to have a large number of attributes. There is room for package size optimization.
By modifying the script that generated the JSONModel subclass, we achieved:
1. All attributes are modified with @dynamic, and the base variable generates IVAR in addition
2. All JSONModel subclasses inherit from the new parent class, which implements resolveInstanceMethod. In this method, get/set methods are added to attributes using class_addMethod. Properties of object types are accessed using associated objects, and properties of base types are accessed using additional generated IVAR.
This optimization yielded a packet size gain of 800KB, and the estimated performance impact on read and write loss was acceptable.
3.5. Migration of __TEXT segments
If the Download Size of the compressed installation package exceeds 200 MB, the Download of the App on the cellular network will be limited, which will have a great impact on new additions. In the second half of 2020, we explored and practiced __TEXT segment migration technology: In the link stage, the -rename_section option is used to migrate __TEXT and __TEXT to __BD_TEXT and __TEXT, reducing apple’s encryption scope for executable files and improving the compression efficiency of executable files, thus reducing the Download Size.
Using this scenario we ended up reducing the Download Size by 60 MB and the Install Size by 2 MB. For detailed principles, please refer to toutiao Optimization Practice: binary optimization of iOS package size, Reducing download size of 60 MB for a line of code [5].
3.6. Binary compression
The Mach-o file takes up a large portion of Install Size, but not every section/section in the file is needed the first time the program starts. You can compress the sections/sections in the Mach-O file during build and then simply unpack them into memory before they are used to reduce the package size while keeping the program running. Due to some limitations of Apple, we currently only compress the __TEXT, __gCC_except_tab and __TEXT,__objc_methtype sections, It is then decompressed in a callback to _dyLD_register_func_FOR_add_image. This solution optimizes the cumulative Install Size of 3.5 MB.
Four,
In the implementation of the above optimization items, we also cooperate with the business to further optimize the size of the installation package by mining useless code, useless resources and other means. Under the rapid business iteration, toutiao’s package size can still remain stable.
Join us
In the process of practicing package size optimization, we found that to do optimization, we need to settle down to do technical problems, but also need to cooperate with all parties. We as today’s headline iOS platform architecture team, performance optimization, basic components, business architecture, research and development system, security compliance, online and offline quality infrastructure positioning attribution of direction such as continuous deep platform, responsible for the security and improve product quality and development efficiency of today’s headlines, focusing on today’s headlines extends outward at the same time.
If you’re passionate about technology, you love to push the envelope, and you want to change the experience of hundreds of millions of people with your own code, join us. We look forward to your growing with us. At present, we have recruitment requirements in Beijing and Shenzhen, resume email: [email protected]; Email subject: Name – Years of work – Today’s Headlines – Platform architecture – iOS/Android.
The resources
[1] dry | headlines today iOS terminal installation package size optimization mp.weixin.qq.com/s/PufNDzf9V – train of thought and practice…
[2] the Palette Images www.manifold.net/doc/mfd9/pa…
[3] WWDC 2019 What ‘s New in Clang and LLVM developer.apple.com/videos/play…
[4] WWDC 2016 What ‘s New in LLVM developer.apple.com/videos/play…
[5] of best practice in today’s headlines: iOS binary package size optimization, one line of code to reduce 60 MB download size mp.weixin.qq.com/s/TnqAqpmuX…
Welcome to Bytedance Technical Team
Resume mailing address: [email protected]