What is cocoapods-MAP-Prebuilt?

Cocoapods-map-prebuilt is a cocoapods plug-in developed by Meituan Platform iteration group. Based on Header Map technology, it further improves the speed of code compilation and improves the search mechanism of Header files.

While building apps as binary components is the dominant solution for HPX, But in some scenarios (Profile, Address/Thread/UB/Coverage Sanitizer, static App level examination, ObjC method call compatibility check, etc.), we build work still needs to complete source code to compile manner; And in the actual development process, most of the development is done in the way of source code, so we set the experimental object as a process based on full source code compilation.

Without further ado, let’s take a look at its actual use effect!

In general, cocoapods-MAP-Prebuilt can increase the total link speed by more than 45% and the Xcode packaging speed by more than 50%, using the all-source compilation process of Meituan and Dianping as the experimental object.

To better understand the value and functionality of this plug-in, let’s take a look at the current engineering problems.

Why are existing programs not good enough?

In the Pods/Header/ directory, CocoaPods will add a component name directory and a soft link to the Header file, similar to the following:

/ Users/sketchk/Desktop/MyApp/Pods └ ─ ─ Headers ├ ─ ─ Private │ └ ─ ─ AFNetworking │ ├ ─ ─ AFHTTPRequestOperation. H - > . / XXX/AFHTTPRequestOperation. H │ ├ ─ ─ AFHTTPRequestOperationManager. H - >. / XXX/AFHTTPRequestOperationManager. H │ ├ ─ ─... │ ├─ ├─ ├─ ├─ ├─ ├─ ├─ ├─ │ ├─ ├── ├── ├── ├── ── ── ── ── ── ── ─ AFHTTPRequestOperation. H - >. / XXX/AFHTTPRequestOperation. H ├ ─ ─ AFHTTPRequestOperationManager. H - > . / XXX/AFHTTPRequestOperationManager. H ├ ─ ─... └ ─ ─ UIRefreshControl + AFNetworking. H - >. / XXX/UIRefreshControl + AFNetworking. HCopy the code

It is through this directory structure and soft chain that CocoaPods is able to add the following parameters to the Header Search Path to make the precompilation process go smoothly.

$(inherited)
${PODS_ROOT}/Headers/Private
${PODS_ROOT}/Headers/Private/AFNetworking
${PODS_ROOT}/Headers/Public
${PODS_ROOT}/Headers/Public/AFNetworking
Copy the code

While this approach to building Search Path solves the problem of precompilation, in some projects, such as giant projects with up to 400+ components, it can cause the following problems:

  1. A large number of Header Search Path paths, resulting in compilation parameters-IOptions inflate so fast that they fail to compile after a certain length
  2. At present, there are nearly 5W header files in the project of Meituan, which means that both the search process of header files and the creation process of soft chain will cause a large number of FILE IO operations, which will lead to some time-consuming operations.
  3. The compilation time will increase sharply with the number of components. Taking Meituan and Dianping as reference, which have 400+ components, the whole source code packaging time is more than 1 hour.
  4. Finding headers based on path order has potential risks, such as the case of identical headers, which are never compiled.
  5. Due to the${PODS_ROOT}/Headers/PrivateThe existence of a path makes it possible to reference private header files of other components.

At best, it can take up to an hour to solve the above problem. At worst, it can cause a headache for the engineers to let risky code go live.

What is a Header Map?

Fortunately, cocoapods-map-Prebuilt has made these problems a thing of the past, but to understand why it solves these problems, we need to understand what a Header Map is.

Header Map is actually a set of Header file information mapping table!

For a more intuitive understanding of Header Map, you can enable the Use Header Map option in Build Setting to experience it for real.

Then get the Build command of the corresponding file in the corresponding component in the Build Log, and add the -v parameter at the end to check the secret of its operation:

$ clang <list of arguments> -c some-file.m -o some-file.o -v
Copy the code

In the output of the console, we find something interesting:

From the above figure, we can see that the compiler shows the order in which to find the header files and the corresponding paths, and in these paths we see something strange: files with the suffix.hmap, followed by parentheses that say headermap.

That’s right! That’s the entity of the Header Map.

At this point Clang has already stuffed a map of header file names and header file paths into the hmap file mentioned earlier, but it is in binary format. To verify this claim, we can use the Hmap tool written by Milend to look up its contents.

After executing the relevant command (hmap print), we can find that the structure of the information stored in these Hmaps is roughly as follows, similar to a key-value form, where the Key Value is the name of the header file and the Value is the actual physical path of the header file:

Note that the key contents of the mapping table can vary depending on the usage scenario. For example, header references are in “…” Or <… Or Header configuration in Build Phase. For example, if you set the header file to Public, in some Hmaps its Key may be PodA/ClassA, and in project its Key may be ClassA. Where this information is configured is shown below:

Now I think you know what a Header Map is.

Of course, this technique is not new; Facebook’s Buck tool offers something similar, but the file type is headermap.java.

At this point, I don’t think you will be interested in Buck. Instead, you will start to think about what the Public, Private, and Project of Headers in the previous figure mean. It seems that many students have never paid much attention to it and why it affects the content in hMAP.

What is Public, Private, Project?

In the official Apple Xcode Help – What are build Phases? In the document, we can see the following explanation:

Associates public, private, or project header files with the target. Public and private headers define API intended for use by other clients, and are copied into a product for installation. For example, public and private headers in a framework target are copied into Headers and PrivateHeaders subfolders within a product. Project headers define API used and built by a target, but not copied into a product. This phase can be used once per target.

In general, Build Phase-Headers mentions that Public and Private are Headers that can be used by the outside world, while Headers in a Project are not used by the outside world and will not be included in the final product.

If you continue scrolling through information like Stackoverflow-Xcode: Copy Headers: Public vs. Private vs. Project? And StackOverflow-Understanding Xcode’s Copy Headers Phase. You’ll find that in the Project Editor section of early Xcode Help, Setting the Role of a Header File specifies the differences between the three types.

Public: 163.com The interface is devoted and meant to be used by your product’s clients. A public header is included in The product as readable source code without restriction. Private: The interface isn’t intended for your clients or it’s in early stages of development. A private header is included in The product, but it’s marked “private” Thus the symbols are visible to all clients, but clients should understand that they’re not supposed to use them. Project: The interface is for use only by implementation files in the current project. A project header is not included in the target, except in object code. The symbols are not visible to clients at all, only to you.

At this point, we should be able to thoroughly understand the differences between Public, Private and Project. In short, Public is still Public In the usual sense, Private means In Progress, and Project means Private In the usual sense.

Podspec Syntax in CocoaPods contains public_header_files and private_header_files.

Let’s take a closer look at the official documentation here, specifically the private_header_files field.

Private_header_files are not intended to be used by the user, but will be included in the final product when it is built. Private_header_files are not intended to be used by the user. Only headers that are neither Public nor Private are considered truly Private and do not appear in the final product.

It seems that CocoaPods’ official definition of Public and Private is consistent with Xcode’s description. The definition of Private in both places is not what we normally think of as Private, but rather is intended to be open to developers. A header file that is not quite Ready, however, is more like an In Progress.

Is this one a bit of a surprise to you? So, are we using them correctly in the real world?

Why doesn’t using native Hmap improve compilation speed?

We’ve explained what hMap is and how to enable it (enabling the Use Header Map option in Build Setting), as well as some of the factors (Public, Private, Project) that affect generating hMaps.

Can I just enable the Use Header Map provided by Xcode to speed up compilation?

Sadly, the answer is no!

As for why, let’s start with the following example. Suppose we have an all-source project built on CocoaPods with the following overall structure:

  • First, Host and Pod are our two projects, and the product of Target under Pods is Static Library.
  • Second, there will be a Target with the same name under Host, and n+1 targets under Pods, where n depends on the number of components you depend on, and 1 is a Target named PODs-xxx. Finally, Pods-xxx is a Target artifact that is dependent on the Target in Host.

The entire structure looks like this:

When you build Static Library artifacts, CocoaPods uses the following logic to create header artifacts:

  • No matter what you set in your PodSpecpublic_header_files 和 private_header_files, the corresponding header file will be set to Project.
  • inPods/Headers/PublicWill save all declared aspublic_header_filesHeader file.
  • inPods/Headers/PrivateWill save all header files, whetherpublic_header_filesorprivate_header_filesDescribed to, or undescribed, this directory contains the complete set of all header files for the current component.
  • If Public and Private are not marked in the PodSpec,Pods/Headers/Public 和 Pods/Headers/PrivateWill contain all header files.

Because of this mechanism, some interesting problems can occur.

  • First, since all Header files are retained as final products, in conjunction with the Header Search PathPods/Headers/PrivateThe existence of a path, we can refer to the private header file in another component, for example, I just use#import <SomePod/Private_Header.h>Will match the private file’s matching path.
  • Second, in Static Library cases, once we enable Use Header Map, the hmap will only contain the Use Header Map if all Header files in the component are of type Project#import "ClassA.h"Key value reference, that is, only#import "ClassA.h"Otherwise, Header Search Path will be used to find the relevant Path, such as the PodB in the following figure. During the build process, Xcode will generate 5 HMAP files for PodB. In other words, these 5 files will only be used in compiling PodB, where PodB will rely on some PodA header files, but since the header files in PodA are of Project type, their keys in Hmap are allClassA.hThat is to say, we can only use#import "ClassA.h"In the manner of introduction.

As we know, references to other components are usually introduced with #import <A/A.h>. The reason for using this method is that it makes it clear where the header file came from and avoids problems, and it lets you toggle between enabling the Clang Module and not. Of course, Apple has advised developers to introduce headers in this way more than once in WWDC.

Continuing with the topic above, using the Use Header Map option will not speed up compilation in the case of Static Library with #import as A standard way to import headers.

But is there really no way to use Header maps?

Cocoapods hmap – prebuilt was born

Of course, there is always a way to do this. You can make your own Hmap file based on CocoaPods rules. It is based on this idea that Meituan.com developed cocoapods-MAP-Prebuilt.

Its core functions are not many, probably the following:

  • When the CocodPods process the Header Search Path and create Header soft link, we build the Header index table and generate N +1 HMAP files (n is each component’s own Private Header information, 1 is the Public Header information common to all components.
  • Overwrite the Header Search Path in the xcconfig file to the corresponding Hmap file, one pointing to the component’s own private Hmap and one pointing to the public Hmap shared by all components.
  • Public hmap header files with the same name are treated specially and only allowed to be savedComponent name/header nameThe key-value command is used to troubleshoot abnormal behaviors caused by header files with the same name.
  • Disable the Ues Header Map function of the component to reduce unnecessary file creation and reading.

It may sound a bit convoluted and verbose, but you don’t have to worry about that. You just need to follow these two steps to get it going:

  1. Declare the plug-in in Gemfile.
  2. Use the plugin in your Podfile.
// this is part of Gemfile
source 'http://sakgems.sankuai.com/' do gem 'cocoapods-hmap-prebuilt' gem 'XXX'... end // this is part of Podfile target 'XXX' do plugin 'cocoapods-hmap-prebuilt' pod 'XXX'... endCopy the code

In addition, to extend its usefulness, we also provide the ability to patch header files (to solve the directional selection of the same name header files) and inject environment variables (non-invasive use in other systems), so that it can be used in different scenarios.

conclusion

This concludes our introduction to Cocoapods-MAP-Prebuilt.

Looking back at the beginning of the whole story, Header Map is a very small knowledge point that I found in the process of studying Swift and Objective-C mixed programming. Moreover, Xcode itself has implemented a set of functions based on Header Map, but its performance is not ideal in actual use.

Fortunately, as we explored, we found out why Xcode’s Header Map didn’t work and why it wasn’t compatible with CocoaPods. Although it’s not that complicated, the core of Xcode is to encode IO operations like file searches and reads into memory operations. However, when combined with the actual business scenario, we find that its benefits are very considerable.

Perhaps this is a reminder that we should always be curious about technology!

In fact, using Clang Module technology can also solve some of the problems mentioned at the beginning of this article, but it is not the scope of this article. If you are interested in Clang Module or Swift/Objective-C mixing, Read the article “Understanding Swift and Objective-C and mash-up from a Precompiled Perspective” in the Resources for more details.

Reference documentation

  • Apple – WWDC 2018 Behind the Scenes of the Xcode Build Process
  • Apple headermap. CPP source code
  • Meituan Institute of Technology – Understanding Swift and Objective-C and hybrid programming from a precompiled perspective

The author

  • Siqi, pen name SketchK, is an iOS engineer of Meituan. Currently, she is responsible for CI/CD of mobile terminal and matters related to Swift technology on the platform.
  • Xu Tao, iOS engineer of Meituan, is currently responsible for iOS terminal development and efficiency improvement.
  • Shuang Ye joined Meituan in 2015 and has worked on Hybrid container, iOS basic components, iOS development tool chain and client continuous integration portal system.

| want to read more technical articles, please pay close attention to Meituan technical team (meituantech) WeChat public official.

| in the public, the menu bar reply goodies for [2020], [2019] special purchases, goodies for [2018], [2017] special purchases, 【 method 】, such as keywords, to view Meituan technology team calendar year essay collection.