I recently published to Alibaba mobile technology article (original link), reprint to my personal home page, welcome interested friends can comment section exchange.

Business development has encountered more and more environmental problems, which seriously affect development efficiency. Some seemingly packaging problems are behind the corruption of engineering architecture.

The background,

In recent years, the negative impact of the high complexity of iOS engineering has been gradually exposed. Many students have been “devastated” by the slow and complex packaging of iOS, and the efficiency of business development has been greatly affected. I remember one of my classmates complained to me that he had packaged several modules and integrated them into the main project. In this process, packaging failed in every step, and it took more than half a day in total.

Alibaba.com is a type B cross-border e-commerce business. It started to develop aN iOS client in 2012. In order to support business development, componentized transformation was carried out in 2016, from a single engineering architecture to a modular architecture. With the development of business and wireless technology, the client has evolved from a small modular project to a giant project. The team built more than 100 self-maintenance modules, including business modules, architecture facilities, Hybrid containers, Flutter containers, dynamic technology, basic middleware and other capabilities. On the surface, the engineering architecture is evolving in an orderly fashion, but on the inside it is already a mess. Module relationships are chaotic, and cyclic and reverse dependency behavior is increasing. A large number of modules do not comply with the LLVM Module standard, spec files are incomplete, and header file references are not standard. Cocoapods can’t be upgraded due to substandard engineering and uses only older versions 1.2 and 1.5, leaving it more than three years behind technologically.

In order to solve the problem thoroughly and improve the business development experience, alibaba ICBU terminal architecture group comprehensively manages the iOS engineering architecture. I will also write an article to record my thinking, welcome interested students to guide the exchange.

Steve Mcconnell, Code Complete: “Software’s first technical mission: Managing complexity.”

Second, architecture corruption leads to low development efficiency

Problem 1: Module packaging complexity is high

1. Mixed engineering environment

In 2016, the componentization of Alibaba client is not complete. Many modules are separated in form only, and in fact, there are problems of reverse dependence and cyclic dependence. By 2017, the team wanted to Framework, but found that modules could not be packaged and compiled separately. So, in order for the module to compile, we developed compatible scripts that added all framwwork and header files to the project searchPath and had the module read all dependencies from the synchronized main project Profile directly. Since compatibility logic made it possible to compile spec files without dependency descriptions, no one maintained spec files and cross-module header references got messy.

2. Environment incompatibility & module construction failure

Module compilation scripts add a lot of Workaround logic and are compatible with header indexes because of cyclic dependencies and non-standard header files. As a result, the module Cocoapods environment cannot be upgraded and is stuck at version 1.2. As middleware and community Swift technologies grow, the main project Podfile uses cocoapod 1.5’s new syntax. The environment becomes incompatible. At the same time, the module failed to recognize cocoapod 1.5’s new syntax when parsing the main project Podfile.

3. 90 person-days of development resources are wasted every year

After the module packaging failure, the development needs to analyze the log and find out the cause of the packaging failure. If the analysis cannot find out, the architecture group needs to support. Failure to package a module can keep requirements stuck and unintegrated, blocking testing or other development efforts.

Based on development feedback, it is estimated that an average module packaging failure consumes 2 hours of development resources. According to statistics, during Q1, the total number of module packaging failures was as high as 200 times, among which 70% of the packaging failures were caused by high complexity. Two hours per packaging failure is equivalent to 90 person-days of r&d resources wasted per year.

Robert Martin, Clean Architecture: “No matter how dedicated you are and how many shifts you put in, you’re still stuck with a broken system because most of your energy is spent dealing not with development needs, but with chaos.”

Problem two: the main project packaging slow

If the module is not standardized and needs to reference swift middleware, it cannot be independent static library and can only be integrated into the main project in the form of source code. This leads to the compilation of a large number of source code when the main project is packaged, and the average packaging time is 12 minutes slower than handtao, Youku and other projects. It is necessary to pack the main project when the requirements are proposed, integrated, bug fixed and troubleshooting. Slow packing will block the development and testing work. One biweekly iteration packed 70 times and wasted 14 hours.

Problem three: unstable engineering environment

The Cocoapods environment cannot be upgraded and only older versions 1.2 and 1.5 can be used. But older versions of the environment were not maintained, and the environment was extremely fragile, such as when someone published an illegal spec and the Pod Update would hang. Because modules are not standardized, source development will have all kinds of inexplicable compilation problems. Business development and debugging can be inefficient and time-consuming.

Problem four: Swift development is hard to do

In recent years swift popularity, iOS community and group swift middleware more and more. However, the Swift module strictly follows the “LLVM Modules” specification, which does not allow circular dependencies, declarations for external dependencies, Angle brackets for header references, or “could not build module XXX”, “No such module” errors. Under the high standards, our engineering development into Swift is difficult. Although we may not use Swift ourselves, the trend of swiftization of group and third-party middleware is irreversible.

In the past two years, Alibaba.com has introduced a lot of Swift middleware into its project, and also developed many Swift components independently, which has completely exploded the problem of r&d efficiency. Related modules are frequently packed incorrectly. The problem of irregularities is so complex that one compiler error is often resolved, and another error occurs, and the next generation is endless. Finally, the system began to have uncontrollable risks.

In addition, most of our Modules do not conform to the LLVM Modules specification. If the business requirements use Swift or reference to Swift middleware, it can take a lot of time to resolve the adaptation issues. According to the agile iteration data, plan A required 10 man-days, which actually consumed 20 man-days, and Plan B required 6 man-days, which actually consumed 10 man-days.

The complexity deteriorates to a certain extent, to a certain extent there are many unknowns

Problem 5: Historical code cleaning is difficult

In recent years many old businesses have gone offline or been revamped. However, due to the serious coupling between modules, much of the old code has been afraid to delete, which also leads to the continuous expansion of package size.

Difficulties and strategies

Influence range is wide, governance is difficult to promote

In 2020, I launched the Architecture Governance project in the iOS technology stack. I started the governance of iOS development from all lines of business, but I got stuck. On the one hand, resources are not invested in business development. On the other hand, the call relationship between many business modules is chaotic, and governance risks are high.

Data analysis, top-down push

The chaos of the iOS project has taken a toll on the business, with time wasted trying to figure out how to compile and package. IOS developers in all businesses were frustrated, and many began to report that the packaging difficulties were seriously affecting development efficiency.

To this end, I began to comprehensively comb the data of the R&D process. On the one hand, I counted the failed data of module construction and the time spent in packaging of the main project, and then compared it with the data of other clients. On the other hand, I conducted interviews on business development to understand the data of resource waste from the perspective of users and supplement the links that could not be counted in the R&D platform. Finally, the negative impact of engineering chaos on R&D efficiency is quantified into concrete data.

With the results of the data analysis, you have a starting point to drive architecture governance from the top down.

Iv. Solutions

1. Review the overall situation and clarify module dependencies

The first difficulty is that the relationship between modules is not clear. The dependency list in the module description file is empty, and the relationship between modules is like a ball of wool.

Without a clear relationship between modules, governance projects cannot be disassembled and costs cannot be estimated. Therefore, we should first look at the overall situation and analyze the overall module dependence.

I developed a tool to analyze it. The first step is to find all the files in the module and use the re match to find the external header files that it imports to get a collection of externally referenced headers. Then search the Pods directory of the main project, match the external module to which the header file belongs, and aggregate to get the complete module dependency tree.

The next step is visualization, after which the complexity of module relationships can be seen more intuitively, facilitating governance planning. I used Dot language to describe module relationships, which can automatically generate dependency diagrams for the whole project or for a specific module.

2 dependency inversion and layered governance

The second difficulty is the complexity of the dependent conditions of governance.

The criteria for successful Module governance is that all modules in the entire dependency tree have no cyclic dependencies and are compliant with the LLVM Module specification. For example, governance business Module A, Module A has A Module C in the dependency tree, Module C has cyclic dependencies or does not conform to Module specifications, Module A will report an exception when it is packaged. Cocoapod and XCode report only one exception at a time and can’t analyze the entire dependency tree.

Our project maintains more than 130 modules, and more than 200 third-party libraries and middleware modules. In addition to its own dependencies, business modules also have many indirect dependencies, and the dependency tree is very complex. In this case, the direct governance of business modules is extremely complex and the governance process will be chaotic.

In the example above, modules C, I, and G are central modules with complex relationships. For example, “module I” directly depends on 30 external modules, and indirectly depends on more than 100 modules. Its direct coupling relationship has 5 cycles, and its indirect coupling relationship has 15+ cycles. If “Module I” is directly managed, 15 cyclic relationships need to be decoupled, and more than 100 modules need to be transformed into modules. According to this idea of governance, the logic of modification is extremely complex, and it is likely that governance will not go on halfway.

To solve this dilemma, I layered and categorized modules. There are three basic logic for partitioning:

  1. The lower-level module dependencies are simpler;

  2. Modules without loop dependencies are easier to govern;

  3. Governance completed modules can be ignored.

In this vein, I first sort out the hierarchy of modules, and then work from the bottom up. When the underlying modules are managed, the burden of modules that depend on more will be greatly reduced. When the low-level loop dependencies are decoupled, the upper-level modules do not have to deal with the indirect loop dependencies.

Finally, four quadrant analysis method was used to divide modules into four groups: 1 basic module without cyclic dependency, 2 basic module with cyclic dependency, 3 service module without cyclic dependency, and 4 service module with cyclic dependency. Each group was governed in order.

3 Automatic Repair

The third difficulty is that the code changes rapidly. Module governance faces many sub-problems, such as “incomplete dependency description of module Spec file”, “Umbralla header file is not missing”, “public header file reference is not standard”, “cyclic dependency decoupling”.

Just fixing the “incomplete dependency description of the module spec file” is difficult.

The method of completing dependencies is to find all source files of the import description “(import <xxxFramework/xxx.h)”, statistics since all the framework. It then reverse-searches for the owning module based on the framework name. In addition, many import formats are not standard, some are direct reference file name (import “xxx.h”), some are path reference (import < XXX/XXX/XXX.h >), encounter such non-standard reference, also need global search to find which module belongs to. For example, the dependence description for module A is empty, but it actually depends on more than 20 modules. Module A has more than 60 source files, with an average of 10 lines of import references per source file and A total of 600 lines of reference code. If you were to analyze these 600 lines of code manually, it would take a day. This is just one of the fixes, not including “umbralla headers are not missing”, “public header references are not standard”, and “loop dependency decoupling”.

Therefore, pure manual governance simply does not work, must be automated way to improve efficiency. So I developed an architecture management engine that can be used to analyze module dependencies, fix incomplete spec dependency descriptions, automatically generate umbralla headers, modify non-standard header references, and more. Automated repair tools can cover 95% of code changes, and development is only responsible for routing changes, service apis, code migration, module splitting and merging, and other major logical changes.

The architecture management engine can not only do architecture governance, but also serve as a team management tool, such as analyzing Git repository activity, batch setting CodeReview rules, and logging the development process.

The following code uses The Ruby language and cocoapods-Core framework to analyze module import code and fix podspec dependencies for modules.

require 'cocoapods' require 'cocoapods-core' require 'xcodeproj' def DependencesAnalyser.main(contextHelper, projectToolPath, moduleName, AllModuleNames) # 1 Fix import format iOSProjectDir = contexthelper. projectDir podDir = contexthelper. podDir iOSProjectName = ContextHelper. ProjectName # read source_files path sourceDir = contextHelper. SourceDir if sourceDir. Nil? The puts '[error] dependency failed to be fixed, AllheadPaths = getSourceHeaderPath(sourceDir) # 2 Run through all source files, Read every line of the file, The re matches all import lines # 2.2 and resolves the dependent headers importHeaders = if they are referenced by import "" or import <xx.h> rules ParseHeaderNameFromQuotationImport (allheadPaths) # 2.1 if it is the import < xx/xx. H > refer to truncate the dependences framework name = directly ParseFrameworkNameFromAngleBracketsImport (allheadPaths) # 3 if the import reference, "" rules, judge whether the header of the reference existing Pod directory, If there is a record in the Pod Framework of # 3.1 read the main engineering Pod file directory of all the dependent libraries. H file path dependencesFromQuatationImport = findFrameNameFromQuatationImportHeader(podDir, importHeaders) dependences = dependences + dependencesFromQuatationImport filtedDependences = Dependences (projectToolPath, moduleName, allModuleNames) Modify_spec_file (filtedDependences, contextHelper) # 5 Return filtedDependences endCopy the code

4 Architecture and business cooperation governance

The fourth difficulty is that decoupling involves a lot of business logic. A lot of code is branch logic of business, it is difficult to test after reconstruction, if not fully verified, it is easy to go out of line failure.

Decoupling involves a lot of business logic, and the best way to reduce risk is to hand it over to business development to modify. Therefore, the horizontal iOS engineering governance project is led by the Architecture group, which provides governance solutions and tools, and business development is responsible for business logic decoupling. There are four ways of service decoupling: routing Scheme, servitization API, common component sinking and module merging.

Take a few typical decoupling scenarios.

Scenario 1: A sub-business in the product module is product recommendation, and the order module is also needed, so the order module reversely depends on the product module, forming a circular relationship. This scenario is decoupled in a way that separates the base components from the product modules, and the order modules depend on the base components.

Scenario 2: When the product module jumps to the order module, the model of the product is used as the input parameter of the API. The order module reversely relies on the product module to reference the model of the product. The way to decouple this scenario is to use the routing URL Scheme protocol to convert the Model into the input parameter of query in the URL.

5. Long-term guarantee mechanism

After architecture governance, issues such as module cyclic dependencies and modula specifications are resolved, but secondary corruption may occur in the future. Of course, we do not want to have to re-govern after a period of time, so we should start from the interface of the architecture design and r&d process, optimize the architecture and process, and prevent the subsequent secondary corruption.

1 Architecture Optimization

  • Systematically define and divide modules to increase the cohesion of module logic and avoid the need to develop multiple modules at the same time for one requirement. Convergence module number, reduce module maintenance cost;

  • ICBU service modules will eventually be integrated into the host and the guest, and the unification of version arbitration in the main project can reduce complexity and avoid conflicts in the module version declaration. Module dependency description only declares the module name, not the version number. When packaging, synchronize the module version of the main project as version arbitration.

2 Convergence module engineering

If modules maintain their own build projects, long-term maintenance will inevitably result in vastly different build configurations. On the one hand, this does not uniformly upgrade build configurations, and the cost of architecture governance and technology upgrades can be high. ; On the other hand, the cost of troubleshooting modules with build problems can be high.

Therefore, we built a package script to dynamically generate module projects each time we package. Modules no longer maintain separate projects, and build configurations converge uniformly to podSpec files.

When a module is packaged, build projects for the module are created dynamically

require 'cocoapods' require 'cocoapods-core' require 'xcodeproj' require 'rubygems' project_creater = ProjectCreater.new(ContextHelper.tempProjectPath, ContextHelper.projectName) project_creater.transform require 'pathname' class ProjectCreater def initialize(root, Name) @project_path = pathname.new (root). Realpath @project_name = name end def transform puts "ProjectCreater- start" Prepare puts "ProjectCreater- Starts to rename "Rename puts "ProjectCreater- completes" end private def prepare Xcodeproj_path = @project_path.join("#{@project_name}.xcodeproj").to_s if File.exist? (xcodeproj_path) `rm -rf #{xcodeproj_path}` end end def rename Dir.glob(File.join(@project_path.join("Podfile").to_s)).each do |file| content = File.read file content = content.gsub(/POD_NAME/, @project_name) File.open(file, 'w') { |f| f << content } end Dir.glob(@project_path.join('PROJECT.xcodeproj').to_s + '/**/*').each do |name| next if Dir.exist? name if File.extname(name) == '.xcuserstate' next end text = File.read name text = text.gsub("PROJECT",@project_name) File.open(name, "w") { |file| file.puts text } end scheme_path = @project_path.join("PROJECT.xcodeproj/xcshareddata/xcschemes/").to_s File.rename(scheme_path + "PROJECT.xcscheme", scheme_path + @project_name + ".xcscheme") File.rename(@project_path.join("PROJECT.xcodeproj").to_s, @project_path.join(@project_name + ".xcodeproj").to_s) end endCopy the code

3 Compile bayonet through CocoaPod and Xcode

  • The main project CocoaPods environment is upgraded to version 1.9.1. Cyclic dependencies will be detected during update.

  • Remove compatible Header search Path logic, the module must use the standard Header file reference method to compile;

  • Enable the XCode Modular compilation check, which compiles if the module’s header references are not canonical;

4 Devops build bayonet

  • Strictly follow the integration single process, integration single needs to be compiled to pass the integration;

  • Static scan plug-in was added in the construction process to detect module specifications;

conclusion

Architectural corruption is like a “flu virus”, its negative effects are hard to perceive and quantify.

For the technical team, to avoid architectural corruption, the technical team should have a higher respect for the technology, rather than waiting for the fire to spread, we should give more substantive support and encouragement to those who put out the fire in time.

For the architect, it is necessary for the architect to be skilled in developing tools. In the face of complex degree architecture problems, the first step is to conduct a comprehensive analysis, disassemble the system problems, find the governance path with the lowest complexity, and consciously seek data support to obtain the support of the team.

Finally, from the perspective of architecture governance. Client engineering is a naturally centralized architecture that is prone to compilation problems due to environmental conflicts. Therefore, when designing a componentized architecture, we want to ensure that the environment of the module is completely independent and avoid a centralized architecture. Structural governance is not the end. After the completion of governance, there should be a mechanism to prevent corruption and avoid secondary corruption.

reference

  • “The Clean Architecture book.douban.com/subject/269…

  • The Code Complete book.douban.com/subject/143…

  • The DOT Language graphviz.org/doc/info/la…

  • LLVM Module clang.llvm.org/docs/Module…