preface

Meituan Takeout was started in November 2013, and has been developing rapidly since then, constantly breaking many industry records. As of May 19, 2018, the peak daily order volume has exceeded 20 million, making it the largest delivery platform in the world. The rapid development of the business puts forward higher requirements on technical support: provide online users with high and stable service experience, guarantee the high availability link business and systems operate at the same time, to improve multiple entry business development speed, advancing the App reasonable system architecture evolution, further enhance cross-functional cross-regional collaboration between the team efficiency.

On the other hand, with the rapid growth of users and orders, Meituan Takeout has gradually acquired the characteristics of a traffic platform. Brother businesses have tried to access Meituan Takeout for promotion and release, hoping to provide a unified and standardized service platform. Therefore, the standardization of basic capabilities, the promotion of multi-terminal reuse, and the output of mature and stable technical service platform have always been the core goals pursued by our technical team.

Multiterminal multiplexing end

The word “end” here has two meanings:

• One is multiple entrances to the same business

Meituan Takeout has three business entrances in iOS, namely, “Meituan Takeout” App, “Meituan takeout Channel” App and “Dianping” App.

It is worth mentioning that: due to differences in user profiles and product strategies, although “Dianping” takeout channel, “Meituan” takeout channel and “Meituan takeout Channel” experience the integration of technology stack, their business forms are quite different, so the reuse of upper-layer business is not considered for the time being. Therefore, this article mainly introduces the reuse of meituan’s two main portals.

Before the c-terminal of Takeout was merged in 2015, the two portals of Meituan were developed by two different teams. Although the user perceived interface was almost the same, the code style and technology stack of function implementation were quite different, so it was obviously unreasonable to develop the same demand repeatedly at both ends. So, our goal is the same function, only need to write the code once, do one evaluation, other end only need to do a small amount of adaptation.

• The second refers to the various lines of business on the platform

Different business lines of takeout rely on basic services, including but not limited to: map positioning, login binding, network channel, exception handling, tool UI, etc. Given the scope of standardization, these basic capabilities also need to be multiplexed.

About componentization

When it comes to multi-terminal multiplexing, componentization is inevitably associated with it, and componentization is one of the necessary conditions for multi-terminal multiplexing. What most companies call “componentization” is simply a repository of code, managed using Cocoapods’ Podfile, and aggregating the version numbers of each sub-library in the main project. However, relatively few can design a reasonable layered architecture, clarify dependencies, and have a set of tool chains to support component release and integration. Otherwise, componentization will only lead to side effects such as package size increase, slow development efficiency and complex dependency relationship.

The overall train of thought

A. Concept map of multiplex

The goal of multiplex is to extract the code from the main project into separate components (Pods), which then use podfiles to rely on the desired independent components, which in turn rely on other independent components indirectly through PodSpecs.

B. Preparation

Make sure the base libraries that the multiple ends rely on are consistent, including the open source libraries and the technology stacks within the company.

The common open source libraries in iOS (network, image, layout) have a monopoly in the library industry for each function, which is the advantage of iOS over Android. The company also has some secondary development of open source libraries or self-developed base libraries, namely the technology stack. There may be some variation in the technology stack between different large groups. If there are differences between the reuse ends, refactoring is required to unify the technology stack. (Reconfiguration is recommended, not adaptation, because if it is not done thoroughly, it may need to be filled in later.)

As far as Meituan is concerned, as the company’s two major apps, Meituan platform and review platform have a profound history. Since the merger at the end of 2015, the underlying technology stacks of the two platforms have been constantly integrating in order to jointly build and deposit public services, reduce repetitive wheel construction, improve r&d efficiency, and provide unified and stable basic capabilities for the upper business parties. Meituan Takeaways, as an independent App in early practice, is also a big business side relying on apps from the two platforms. Within one year after the c-terminal of Takeaways was merged, we also did a lot of necessary work to unify the underlying technology stack.

C. Scheme selection

The choice between evolutionary design and planned design.

Evolutionary design refers to design changes as the system is developed, while planned design refers to the design that fully specifies the system architecture prior to development. Evolutionary design, which also follows the basic principles of architectural design, differs only from planned design in terms of design objectives. Evolutionary design promotes meeting the existing needs of the client; The design of the plan needs to consider future functional extensions. Evolutionary design advocates fast implementation, fast solution determination, fast coding and fast implementation; The design of the plan needs to consider the plan carefully, the integrity of the architecture and ensure that the development process is orderly.

Meituan take-out iOS client, at the beginning of the project of multiterminal reuse facing multiple key point: channel entrance and independent application of reuse, the construction of the delivery platform, brothers business access, comments on take-away collaboration, as well as the architecture migration does not affect the development of the existing business, etc., so the balance after we use “evolutionary framework, and secondly plan type architecture” of the design. Rather than forcing history code to achieve the ultimate perfect architecture, one step at a time satisfies existing requirements while retaining some extensibility.

An evolutionary architecture drives reuse

terms

• Waimai: Specifically refers to “Meituan Take-out” App, generally refers to those business portals in the form of independent apps, generally named project.
• Channel: refers to delivery channels in Meituan App, generally refers to those business portals integrated in the main App in the form of channels or tabs, usually Pods.
• Special: Refers to the separation of the business code in Waimai from the original project, making the business code a Pods form.
• Sink down: Sink down to the lower layer, where “lower” refers to the base of the architecture, usually a platform layer or common layer. “Sinking” refers to unifying and moving code from different superlibraries into the underlying base library.

The dynamic evolution of the architecture is first posted here to give you a general idea, and then the experience of different nodes is further described.

Original reuse architecture

As Figure 4 shows, over the past year or two, we’ve had to be conservative with code reuse because of the technology stack. Precipitate individual business or utility class code into “kits,” or smaller grained components. At this time, the concept of layering was still vague, and the previous projects were seriously coupled and logically complicated due to the historical burden. After stripping UGC business, I found that other business codes could not be easily extracted. (At this point, the code reuse rate is only 2.4%.)

Since the previous preparation work has been completed and the multi-terminal base has been consistent, we no longer adopt the conservative strategy, enrich some means of component-based communication, decoupling and transition, and start to make efforts on the layered architecture.

Exploration of business reuse

Under the background that the technology stack has been unified and the base layer has been aligned, we selected Store (i.e. merchant container), one of the core takeout businesses, and began to explore business reuse. As shown in Figure 5, it can be roughly understood as the idea of “two in one, one in three”. We aligned Store businesses on both sides from the perspective of code style and development thinking. In this process, the business class was separated from the code of technical (functional) class, and some general domains were also separated accordingly. As we split each component, our overall reuse was significantly improved, but our development efficiency was unexpectedly affected. Multi-library development adds a lot of manual work to the release and integration of versions: dependency conflicts, lock file conflicts, and other issues prevent us from further improving the development efficiency, which is the side effect mentioned in “About componentization”.

So we put automatic release and automatic integration on the agenda. Automatic integration is the automatic completion of a series of operations from the completion of component development to the integration of functions into the engineering main body and out of the test package. Some preparatory work must be completed before this – shell engineering separation.

Shell engineering separation

As shown in Figure 6, shell engineering, as its name implies, is to remove all the codes in the original project and get an empty shell, only retaining some project configuration options and dependency library management files.

Why shell engineering is one of the necessary conditions for automatic integration?

Because automatic integration involves version number increments, you need machines to modify the project configuration class files. If a new service PR merges during binary creation, the commit tree forks are likely to conflict, resulting in integration failure. After pulling out the shell project, our shell only cares about configuration options modification (rarely), with dependent version number changes. The normal PR flow of business code is moved to git, the respective business component, to eliminate human-machine conflict.

The significance of shell engineering separation is as follows:

• Let the functions more clear, the previous comprehensive layer of multiple jobs too heavy.
• Paves the way for automatic integration, avoiding business PR and machine conflicts.
• Improved efficiency. Subsequent Pods move code to Pods faster than PROJ moves code to Pods.
• “Meituan Takeout” relies on the development environment of “Meituan” to reduce adaptation costs.

From the first figure to the second figure in Figure 7, the shell engineering separation mentioned above is shown. All the business codes of “Waimai” are packed and extracted and moved to the transitional warehouse Special, so that the original “Waimai” becomes the shell.

Images 2 through 3 show the internal digestion of the Pods library.

The first stage is a simple and crude physical code movement, while the second stage is the combing and sorting of the entire code in the Pods.

Internal digestive alignment

As mentioned in the “Multi-reuse Concept Map” section, reuse is the idea of multiple projects accessing unified code as Pods. We consider preserving the integrity of one end of the code to reduce the cost of callback, and decide to achieve smooth migration by Subpods using phased integration.

Figure 8 illustrates exactly how the code is unified within the same module at multiple ends. At this point, because the shell engineering separation has been completed, the business code is in a transitional repository like “Special”.

The unification of modules at both ends of “Special” and “Channel” can be roughly divided into three steps: translation → sinking → tie-back. (The premise is that the business of this module has been determined to be completely consistent.)

The translation phase preserves the integrity of one end of the “Special” code and copies the code file to the other end of the “Channel” in a top-down translation mode. At this point, the former is not affected, and the latter’s code is duplicated because of new file copies and the original code. At this point, the old file is renamed, and the dependencies of the new file are depth-first traversed to complete the file, finally making the compilation pass. Then add part of the difference code in the old file to the new file to do a good job of differentiation management, and finally delete the old file.

The sinking phase decouples and isolates the code processed by “Channel” and moves it to the lower Pods or lower SubPods. The code here supports both “Special” and “Channel”.

The callback phase is to make the “Special” reference to the previously sunken module as a Pods-dependent reference, then delete the code file before the pan. (It is best if you do it between versions, otherwise you need to consider the diff of the code file before panning.)

In practice, it is difficult to process a complete module (such as an order module) in a limited amount of time and sink it to the Pods and reattach it. Therefore, we choose to divide the large module into a sub-module, and these sub-modules smoothly sink into the SubPods. Then “Special” only references the unified SubPods, and separate the Pods after the module is completely sunk.

Here’s how to keep the risk under control when a lot of code sinks:

• Cooperate with PM to sort out business first and mark out special differences.
• The use of OClint’s advance scan dependence, so as to know the precise time estimate.
• Based on the code style of “Special”, “Channel” only adds, not subtracts, when aligned.
• “Channel” alignment does not affect “Special”, and the callbacks are minimal.
• Divided into iteration packages, QA resources coordinated in advance.

Middleware hierarchy flattening

After the “internal digestion” above, the transition code in Channel and Special is gradually distributed to the appropriate components, as shown in Figure 9, leaving only AppOnly for Special and ChannelOnly for Channel. So Special died and Channel became a package project.

AppOnly and ChannelOnly are leveled with other business components. Only two packing works remain on the upper level.

Platform construction

As shown in Figure 10, the lower layer is takeout base library. WaimaiKit contains many subdivided platform capabilities, Domain is a general model, XunfeiKit is the secondary development of intelligent speech, and CTKit is the secondary development of CoreText rendering framework.

As for platform adaptation layer, it plays an important role in differentiation convergence and dependency sorting, which will be explained in detail in “Derivative problem solving” in the next question.

The takeout base library and platform adaptation layer constitute our takeout platform layer as a whole (this is a logical structure rather than a physical structure), which provides more than 60 general capabilities and supports undifferentiated calls.

Multiterminal generic architecture

At this point, we sort out and supplement the basic and open source components to achieve a multi-terminal general architecture, which can be said to truly achieve the goal of multi-terminal reuse.

The actual components are controlled by the upper level different packaging engineering. With the exception of the two packaging projects and the two Only components, the following components have been multiplexed. Compare the two black circles in the business architecture diagram for “Waimai” and “Channel”.

Derived problem solving

differences

A. Differences in demand itself

Three solutions:

• We use runtime macros (dynamically obtaining proj-Identifier) or pre-compiled macros (CUSTOME define) to determine if else directly in the method for the difference of one or two lines of code such as text, value, etc.
• Different Glue layers are used for different method implementations. Protocol provides the same method declarations for external calls and writes different method implementations in different carriers.
• For big differences such as different WebView containers, we build multiple files using file-level precompilation, which can precompile regular.m files or categories. (for example, WmWebViewManeger_wm. m&WMWebViewManeger_mt.m, UITableView+WMEstimated. M&UITableView +MTEstimated. M)

Further optimization strategy:

Although the above three strategies are used to complete the differential management, the differential codes scattered in different components are difficult to converge, making it difficult to manage. With the platform adaptation layer, we converge the differentiation judgment inside the adaptation layer to provide undifferentiated calls to the upper layer. Component developers do not care about host differences in development and directly call the common interface. The determination of differences or subsequent optimization within the interface deals with external perceptions.

Figure 14 shows an example of a modified platform adaptation layer providing a common interface.

B. Multi-terminal rhythm differences

In the actual scene, in addition to the differences in requirements, there may also be differences in the pace of multi-terminal version, which we use the branch management model to solve.

The premise is that since multiplex is going to be used, the general direction of requirements will still want multiplex unification. In most scenarios, the functions of terminal A are the least, and the functions of terminal B are the superset of terminal A. (There is no absolute superset, and the A-end will have fewer differences.) In the takeout business, “Channel” is the end with fewer functions, and “Waimai” is basically a superset of “Channel”.

The differences at both ends can be roughly divided into 5 categories and 9 sub-categories:

1. The requirements at both ends are the same (1.1. The commissioning time is basically the same; 1.2. “Waimai” was tested 3 days earlier than “Channel”; 1.3 “Waimai” was launched 3 days later than “Channel”).
2. Demand “Waimai” advanced version, “Channel” next version into (2.1, Channel next version on; 2.2 channel after the next two versions).
3. Need “Waimai” advanced version, “Channel” not required.
4. The advanced version of “Channel” is required, and the next version of “Waimai” is advanced (4.1. The general part needs to be changed; 4.2 change only the part of “ChannelOnly”).
5. Need “Channel” advanced version, “Waimai” does not need (only change “Channel only” part).

There is no need to tangle too much. Figure 15 is the most complex scenario, which is difficult to encounter in practical situations. At present, our business only encounters 1 and 2 categories, with 2 lines at most.

Compilation problems

In the past, the first full compilation takes about 5 minutes, and then the differential compilation is very fast. However, after the component is extracted, the number of pod install indirectly increases with the change of partial molecular library version. At this time, the high frequency of 3 minutes or 5 minutes will be unacceptable.

At this point we have adopted an all-binary dependency approach, with the goal of reducing compile time by referring directly to compiled artifacts in daily development.

A is the three subPods, divided into three configurations:

1. X64 armv7 arm64 is set by deubg.
2. Release/under release sets the compiled armv7 arm64.
3. Dailybuild/release + TEST=1 armv7 arm64
4. The default (outside the folder. A) is Debug x64 + release armv7 + release arm64.

One issue that needs to be addressed here is the downside of referencing binaries, which obviously brings compile-time problems to runtime. A macro has been modified, but the compiled binary is not aware of the change, and if the dependency version does not match, the original method is missing a compilation error and crashes at runtime. The solution to this kind of problem is also very simple, that is, in all packaging projects are configured with automatic switch source packaging. Binary is only used for efficiency in development and is recompiled using full source code whenever a test package is pulled or released. The cut source vs. cut binary is controlled by environment variables to pull different Podspec sources.

In addition, we support mixed source and binary development mode in development, we can tag a binary_pod modified dependency library, or use.patch files to control the specific library source. In general, developers will Debug the kullah source code associated with their current requirements, and skip compilation of unassociated kullah binaries.

Depend on the problem

As shown in Figure 17, takeout has multiple business components, and the company also has many basic kits. Different business components more or less rely on several kits, so it is easy to form a network dependency situation. In addition, the version number of dependencies may be inconsistent, and it is easy to have dependency conflicts. Once a dependency conflict is encountered, it is necessary to modify a component and re-issue the version to solve it, which affects efficiency very much. The solution is to use the platform adaptation layer to uniformly maintain a set of dependent library versions, and the upper-layer business components only care about the version of the platform adaptation layer.

Of course, in order to avoid the problem of adding too many useless dependencies by introducing platform adaptation layer, we pulled some of the more dependent and less frequently used Kit out of the subPods and supported the optional way to introduce, such as IM components.

Then there is the problem of slow dependency analysis during POD install. For shell engineering, it is the convergence of all dependency libraries. If the dependency relationship writing method is not scientific, it will easily spend a lot of time in analyzing dependency. Cocoapods uses the Molinillo algorithm for dependency analysis, which is implemented in the link as a backtracking algorithm with forward checks. This algorithm itself is no problem, as long as the dependency level is deep enough to write reasonable can also reach second open. However, if the version number of leaf dependent nodes is not strictly controlled, or there is cyclic dependence in the middle, it will lead to the backtracking algorithm repeatedly performing a lot of stack pressing and out operations and consuming time. Meituan’s solution to this problem is to maintain a “dedependent PodSpec source” with the dependency node cleared (middle). The complete set of actual dependencies is tiled in the shell project Podfile for unified maintenance. This has the advantage of flattening the previous tree dependency (left) into a single layer (right).

The efficiency problem

We mentioned automatic integration earlier. Here’s how it can be used. The Meituan distribution engineering team has developed its own HyperLoop distribution integration platform. When a component can select the integration target before creating the binary, if multiplex is used, it only needs to select multiple integration targets when creating the binary in the release. After the release, it will carry out a series of checks and tests by itself, and finally merge the code into the main project (modify the dependent version number of the corresponding shell project).

Above is a commit comparison of “Waimai”. The first diagram shows the old development style, where you can see the engineering configuration commit stacked with the business commit. The second figure shows the commit after shell engineering separation. You can see that each message changes the version number of a dependent library. The third figure is a COMMIT with automatic integration, showing that each message is delivered in a uniform style and in serial machine.

Another problem was that when we replaced project centralization with shell engineering and Pods, our code changes were scattered across different component libraries. If you want to see the diFF of the 6.5.0 and 6.4.0 versions of the main project, you can only see the DIFF of all the dependent library versions. If you want to see the COMMIT and code diff, you have to go to the component library one by one to see the diFF. During the three rounds of test, such similar operations will be repeated many times every day, which is very inefficient.

Therefore, we developed the atomic diff tool. The main principle is to tune the Git Stash interface to get the version number DIff, then deeply traverse the commit through the version number and the corresponding warehouse address, then deeply traverse the corresponding files of the commit, and finally summarize to get the overall code DIff.

The whole tool chain supports multiterminal reuse

Some of the automation tools have been mentioned above, so here’s the overview of our tool chain.

1. In the preparation phase, we will use the OClint tool to process the compile_command. Json file and scan the dependencies of the component to be modified.
2. In dependency repositories, we have binary_pod.rb scripts that use source control to achieve binary and de-dependency effects. The Us Group maintains a set of ios-re-sankuai.com sources to store podspec.json files of Remove Dependency.
3. During dependency synchronization, sync_podfile periodically synchronizes the latest Podfile of the main project to maintain the version number of the complete set of dependencies.
4. During development, we used the Podfile.patch tool to switch binary/source and remote/native code with one click.
5. When referencing native code development, we don’t care much about the version number of the sublibrary, only the version number of the main project, and we use beforePod and AfterPod scripts for dependency filtering to prevent dependency conflicts.
6. Git squash is used to squash multiple commits with the same message at code commit time.
7. Before creating PR, we used to need some manual operation on the web side, writing a lot of Reviewers, now we use MTPR tool to complete this with one click, or use Chrome plug-in according to personal preference.
8. Before the function is integrated into the master, there will be some Jenkins jobs to check.
9. In the release stage, using Hyperloop system, one click release operation is simple.
10. After the release, you can choose the way of automatic integration and joint integration to package, and the packaged products will be automatically uploaded to meituan’s “Grab Fresh” internal testing platform.
11. If you need to look at the Commit Message and code diff between the various versions of the main project during problem tracking, we have the Atomic Diff tool to go deep through the warehouses and summarize the results.

Thoughts and summarize

• After multi-terminal reuse, pM-RD-QA has great changes. Our code reuse rate has increased from 2.4% to 84.1%, allowing more PM to devote to the handling of new demands. However, the improvement of R&D efficiency has increased the workload of QA. A big attempt requires RD to keep in constant communication with PM and QA to choose the optimal solution acceptable to all three parties.

• Prioritizing, technical architecture, etc., is ultimately about supporting the business. If an architecture is designed to be perfect, but it doesn’t work as well as it should in your business, or it leads to complaints, then it’s a failure. And in the actual development of technical code modification as far as possible to choose the version gap, if there is a conflict with business development students, business students should give way, can not affect the original version of the iteration speed.

• Be sensitive to “irrationality” and “duplication of effort.” Is it too expensive to add a buried constant to change the platform and send another version? Why should the Kit on the home page be modified for the requirement of an order status? Put more thought into the actual development of awkward areas rather than just going through them, and think about automated alternatives for doing more than two manual repetitions.

• Once you decide to do it, you must not relent at some key points. For example, a node in order not to Block others, overtime is inevitable. You don’t have to worry too much about a lot of code changes, there are pre-estimates, Case self-testing, and three rounds of QA regression to ensure that you stay focused and just do it.

Author’s brief introduction

Shang Xian, senior engineer of Meituan. I joined Meituan in 2015. Currently, AS the leader of the virtual team of Meituan’s iOS platform for external sales, I am mainly responsible for business architecture, continuous integration and engineering, and committed to improving r&d efficiency and collaboration efficiency.

Recruitment information

Meituan takeout is looking for senior/senior engineers and technical experts in iOS, Android and FE based in Beijing, Shanghai and chengdu. Please send your resume to chenhang03#meituan.com.