Github open source address, step by step guide how to use

An overview,

After years of development, the code of Meiyou iOS project has reached the scale of 40W lines +, the number of Pod libraries used has reached 110+, and the App Store installation package has reached 210M+. Development machine iMac: Retina 5K, 27-inch, 2017 Converged disk; Time: Build30min +) packaging and compilation problems gradually become an unavoidable pain for our team, seriously affecting our R&D efficiency and collaboration with other teams.

As a 13-year-old CI machine, we need to undertake the packaging tasks of seven or eight projects and multiple branches at the same time. In the case of multiple projects being packaged at the same time, it is especially inadequate.

In the case of limited hardware resources, and on the precondition of no intrusion and no impact on the existing business, how to solve these problems in front of the team has become an urgent and urgent need for us. In recent half a year, WE have been looking for a solution to accelerate the speed of packaging.

Exploration and attempt to speed up compilation

1, CCache

CCache is a compiler cache, a tool that caches compiled intermediates

The principle is to compile the source files of the project with the CCache compiler, and then cache the generated information, so that the next compile time, use this cache to speed up the compilation. Currently supported languages are: C, C++, Objective-C, objective-C ++

The following picture basically illustrates how CCache works.

The actual compilation process in the project

Ccache has really improved the speed of our package delivery in some aspects. Meiyou iOS Ci packaging from the previous fastest 20min+ package delivery to the fastest 10min, can indeed bring us a relatively good improvement, greatly speeding up the package delivery speed of our project. After several months of operation of our project, we have found some problems about the situation of our project. Now we summarize the following points:

Advantages:

  1. What we wantNo intrusion,Existing services are not affectedRequirements, non-invasive, and developer unaware.
  2. Indeed, it can greatly improve the compilation speed, and the fastest time in the Meiyu project is more than 3 times the compilation speed.
  3. No major changes to the project are required, just the deployment of the environment and some scripting support.
  4. There is no need to change the development tool chain.
  5. In the same directory, the cache hit ratio of CCache is relatively stable.

There are some problems in our project:

  1. In the case of no cache, the first package compilation time nearly doubled from 20+min, to nearly 40+min for the first time, and even 70 +min in the case of tight resources.
  2. Modifying some files that reference more (such as public libraries and low-level library changes) will easily cause a large range of cache invalidation, and the speed will become slower than when ccache is not used.
  3. The same component in multiple projects does not support cache sharing. We need to package multiple branches. After changing the directory name, the cache will become invalid.
  4. The maximum cache limit of Ccache on our machine is about 18GB, and Debug/Release is different from cache. The meiguo iOS project occupies 5GB+ cache, and multiple projects and branches can easily exceed the upper limit. A Ci machine supporting multiple projects at the same time will trigger Ccache to clear the cache.
  5. The machine hard disk read and write requirements are high, such as not all solid state drives, speed impact.
  6. CCache does not support Clang Modules. Xcode will not automatically import AVFoundation, CoreLocation, etc.
  7. CCache does not support PCH files
  8. CCache does not currently support Swift

2. Exploration of binary scheme of static library

Although we have applied Ccache in Ci and it has nearly doubled the package delivery speed, there are also obvious problems.

At one of our weekly technology conferences last year, our bosses proposed self-research tasks using binary compilation to further improve r&d efficiency. After getting the big guy’s inspiration, has been in practice and exploration of binary road.

Our project uses CocoaPods to manage third-party and private library dependencies, which should be standard for most projects. For now, it’s a pure Objective-C project with a small amount of C++ and no Swift.

3. Investigated binary component solution

Here’s a list of some of the mainstream options studied and why they weren’t adopted. They had their limitations, but they also inspired me that the thought process was as valuable as the final solution.

3.1, Carthage

Carthage can package some of the libraries that don’t change very often into a framework that can be used as a master project, thus reducing compilation time during development. Carthage makes it easier to debug the source code. Because we have now used CocoaPods on a large scale, to use Carthage to do package management needs to do a lot of conversion work, the change is too big, not to meet our non-invasive, no impact on the existing business, so do not consider this scheme.

3.2,cocoapods-packager

Cocoapods-packager can package any pod into a Static Library, eliminating the need for repeated compilation. This helps to speed up compilation time somewhat, but it has its own problems:

  • The optimization is not thorough. Only the compilation speed of third-party and private Pod can be optimized, and there is nothing to be done for other business code that changes frequently

  • Subsequent updates to private and third-party libraries are cumbersome, and when changes are made to the source code, they need to be repackaged and uploaded to the internal Git repository

  • Too many binaries will slow down Git operations (no LFS for Git deployed yet)

  • Difficult to debug source code, no shared compilation cache

  • Packaging Static Library is slow, requires POD Lint, and requires layers of nested dependencies between components that are difficult to implement at the current stage.

3.3,cocoapods-binary

Cocoapods-binary creates Binary packages and caches them in real time, not for a single private library like CocoapoDS-Packager. The pre_install hook is used by CocoaPods to intercept the pod install context during the prepare phase of the pod install. Fork out a separate Installer to complete the clone of the precompiled source code to the Pod/_Prebuild directory, but there are a few drawbacks:

  • Single private source, unable to achieve server caching, in the case of no corresponding binary package version, POD install will be extra to do binary package generation, to some extent will affect the speed of POD install.
  • When the developer switches back to source debugging, the binary cache is cleared altogether, requiring recompilation.
  • The same component from multiple projects and different branches still cannot be shared
  • Framework only support, which requires a significant change in header reference mode for our current project.
3.4,cocoapods-binDouble private source

The binarization strategy of the plug-in is to use dual private sources, that is, two source addresses, a static server to save the pre-packaged framework, one is the service address we now save the source code, in the install time to choose to use download that, is a very good project, deeply inspired.

Advantages:

  • Source and binary files can be switched back and forth, relatively fast
  • Business teams that are not connected to the binary solution are not affected
  • If no binary version is available, the source version is automatically adopted
  • Close to the native CocoaPods experience

For the shortcomings in our project:

  • A reference to :podspec => “, :git is not supported, and is fatal for projects that need to support multiple branches and lines of business.
  • If you use the Archive binary file, you can only download the source code from the Spec repository, but cannot download the dependency libraries according to the specified branch. As a result, the compilation fails and is confused
  • Dependent components need to be pushed to the Spec repository, many private libraries are not pushed to the repository, and for private libraries that change frequently, the verify push to the repository is slow and inconsistent with our development habits.
  • Not supported. A static file output, a lot of similar projects#import "IMYPulic.h"It takes library by library to compile and replace with#import <IMYPublic/IMYPublic. h>Consider the 110 + component libraries
  • Only one set of environments is supported, which cannot be met for those with Debug/Release/Dev development environment requirements
  • Source debugging of binary components is not supported
  • Business components that change frequently cannot be smoothly supported, and the operation will be complicated.
  • For our project, there is a big obstacle at present and it cannot be used.

4. Think and summarize

After more than a month of analysis and thinking of the existing wheel in the industry, and after some practice, we finally decided to build a flexible, configurable, simple, non-invasive, dual private source binary component auxiliary plug-in.

Then roll up your sleeves and work hard ~ SAO nian

Introduction to dual private source binary components

Inspired by cocoapoD-bin, in reference to its part of the framework, we implemented their own binary auxiliary plug-in CocoapoDS-IMy-bin, and added a few commands and binary source debugging capabilities.

What can be done? As long as it compiles, make it

In cocoapoDS-IMy-bin with the help of non-invasive automatic production of all eligible components for binary, and for frequent business components can also easily apply on binary components, without unnecessary operations, all to cocoapoDS-IMy-bin automated operation.

At the same time for r & D personnel, can also provide independent binary components for r & D personnel to use, to solve the daily compilation efficiency, run the machine efficiency is low, wall and other problems.

Our slogan is: If it compiles, make it. Once compiled everywhere, no intrusion.

Even if the individual component libraries do not compile, the overall project will compile.

The whole environment, did not let our developers change the original development habits, did not change the relevant code in the business, basically achieved the use of personnel without awareness of the state.

2. Ci packing effect

2.1 Single project – compile every 2 minutes at most

The figure above is a rough graph of compilation times for a single project based on our experience of typing thousands of packages. This assumes that a machine has only one job at a time. The Y-axis compilation time, the X-axis compilation time, the red line is native (no Ccache and binary components used), the yellow line is Ccache used, and the blue line is binary components used.

As you can see, the original unaided compile time curve (red) flattens out at around 20min. For the first time, Ccache and binaries are somewhat more time-consuming than native ones without any caching. Ccache mainly spends time caching the compiled artifacts of the project as it is compiled. The main time consuming for binary is the time to assemble the. A compiled product and push it to the private source repository after compilation (this is related to the use, if Jenkins compiled product was not used to make binary, it would not exist). .

In the case that ccache is fully hit and binaries are completely present, ccache is more than twice as fast as the native one, and the binary one is more than twice as fast as the compilation time of ccache, and it is stable in about 2 minutes. Binaries tend to be more stable over time, while ccache can have a much lower hit ratio when modifying a heavily referenced file, such as the underlying public file, sometimes taking more time than when not using ccache, such as location #4. In the case that there are multiple ci jobs running concurrently, ccache needs to read and write IO frequently, so the time performance may be worse. We often have to wait 70 minutes before the package is delivered.

The binary compilation time is much more stable (blue curve). With the strong support of our architecture, there are more than 110 independent components, and each packaging is basically a compilation +archive for one component. If it is a component that changes frequently, we can also consider using ccache for a larger component to make a double-layer compiled cache. The two-layer compilation cache principle is that when the Pods component library has no binary components compiled with source code, the source code is compiled with ccache cache support to speed up the compilation of source components.

At the same time, the component library can cooperate with the application of GITlab-CI Runner, triggering the binary production of independent components every time the code has been submitted, so that the compilation speed of each time can reach the fastest, and the blue binary curve will be closer to the straight line. See below for a detailed tutorial on gitlab-CI.

If independent components cannot be compiled or version dependent, you can run a scheduled Job or another polling Job to provide the latest binary components in time.

2.2. Multiple projects

The ccache for multiple items on one machine is laborious and unstable, and will be cleared if the cache exceeds the maximum value of the CCache.

When binary is used, even multiple projects tend to have a more stable build time. You can probably imagine why.

3, development and use effect – more than 10 times the improvement

After the plug-in is introduced to the Podfile, after pod install/update, it will automatically be converted to binary components if the conditions are met.

On our development machine (iMac: Retina 5K, 27-inch, 2017 converged hard drive;) On, the full code before the Build needs 30min+, now use all binary, compile the fastest only needs 2min+, improve the efficiency of more than 10 times.

When compiling and developing with a standalone component library, try using the binary solution to run the entire project. The binary solution may run faster than the standalone component library.

3.1. Source code compilation

Ps: Of the 110+ Pods library, 20+ stable Pods library has been made as binary library, not all source code compilation, how to convert all source code compilation, the actual number will be much more than this.

3.2. Binary compilation – maximum 2 minutes for full compilation

Ps: There are 2 Pods and 5 Action Extensions compiled with source code, the rest are all binary Pods.

In binary Build 127 seconds (ARM64 and ARMV7), about 45 seconds are spent on copy Pods Resource, in addition to source compilation time.

It actually takes less than 90 seconds to compile the emulator X86_64 schema.

In the full compilation, 13,496 Tasks/727 Tasks, 1710 seconds (28.5 minutes)/127 seconds (2 minutes), the speed improvement is much more than 10 times.

3.3 presentation

Once the environment is set up, developers can add the following two sentences to their Podfiles and enjoy an automatic switch to binary components and an extremely fast compilation experience.

plugin 'cocoapods-imy-bin' 
use_binaries!
Copy the code

More specific situation video demonstration

4. Function points

The cocoapoDS-IMy-bin plug-in currently supports the following functions

  1. No intrusion, no impact on the existing business.
  2. Service teams that are not connected to the binary solution are not affected. The configuration file is provided.
  3. As long as the project can compile and pass, even if the independent components fail to compile.
  4. If no binary version is available, the source version is automatically adopted.
  5. Support for making binary components as long as the project can compile and pass, no need to worry about POD Lint, etc.
  6. supportpod bin localThis command automatically makes, uploads, and stores the binary components that already exist locally in the project, which can be used with the compiled products packaged with CI.
  7. Support for specifying dependent branches, support :podspec => “, : Git references
  8. Support simultaneous. A, Framework static library output
  9. When supporting archive, libraries that depend on Podsepc are automatically obtained based on the Podfile, and you do not need to forcibly pull them from the Spec repository.
  10. Support for multiple sets of isolated environments, such as Debug/Release/Dev configuration, convenient for Debug/Release/Dev various environments to provide dedicated binary components.
  11. Support output. A binary component makes binary. Podsepc without templates.
  12. Support for stable binary components. Podsepc skips POD Lint validation when uploading binary components.
  13. supportpod bin autoAutomatically create, upload, and store a single binary component with one key
  14. supportpod bin auto --all-makeOne-click automatic command production, upload, and storage under this itemAll of the componentsBinary component of
  15. Support whether to use binary, whether to make binary and binary/source debugging whitelist Settings
  16. supportpod install/update Multithreaded mode, speed up the POD process,Pod speed increased by 80%+.
  17. supportpod bin install/updateCommand to achieve non-invasive modification of Podfile content, avoid direct modification of the project’s Podfile file and lead to submission conflict, missubmission.
  18. supportpod bin codeCommand, binary library does not switch source library, program without re-run debugging ability


How does iOS compile more than 10 times faster steadily

Github open source address

There is a shortage of all kinds of talents, welcome to join the big family of beautiful grapefruit.

Send your resume to [email protected]