Freeline is a fast incremental compiler for Android. This article introduces how to implement fast incremental compilation.

Android builds the packaging process

First look at the Android packaging flow chart, picture source Android development learning notes (ii) – compilation and operation principle

Paste_Image.png

  • The R file records the ID of each resource, and then participates in the Java compilation process. The R file is generated by the Android Asset Package Tool (AAPT).
  • We know that sometimes there will be cross-process communication in app development. In this case, the interface can be defined in the way of AIDL. The AIDL tool can generate the corresponding Java files from AIDL files. R files, aiDL-related Java files, and Java files in SRC are then compiled to generate.class files
  • Dex generated and compiled. Class is packaged into dex files by the dex tool. Freeline uses the dex tool extracted from Buck, and the data provided by Freeline is 40% faster than the original dex tool

  • Resource file compilation The Android Asset Package Tool (AAPT) packages resource files in apps. Its process is shown in figure (photo source)

    Paste_Image.png


    Android application resource compilation and packaging process analysisLuo shengyang’s article clearly analyzes the packaging process of applied resources.

  • Apk file generation and signature The apkBuild tool packages compiled resource files and dex files into dex files. Jarsigner completes the signing of apk, of course, you can use the apksigner tool to sign after Android7.0. Learn about APK packaging in Android Studio 2.2.

Incremental compilation principle

Android incremental compilation is divided into code incremental and resource incremental. Resource incremental is a highlight of Freeline. When launched, instant-Run is actually not incremental in resources, but pushes resources of the entire application into resource packages to the mobile phone.

  • After Google supports multidex, when the number of methods exceeds 65535, There will be multiple dex files after android packaging. When the runtime loads the class, it will search from a dexList in turn, and return if it finds it. Using this principle, you can package the incremental code into dex files. Insert it in front of the dexList so that the class replacement is complete. There is a problem here that there is compatibility problem on non-art phones, which is also the reason why the instant run only supports android5.0 or above. Freeline has made compatibility processing by using the pile insertion scheme proposed in the introduction of hot patch dynamic repair technology of android App before. This allows incremental compilation on non-ART phones.
  • # # #Resource increment

    Resource increments are one of the highlights of Freeline. In part 1 we learned that application resource files are packaged using the AAPT tool. Freeline developed its own incrementAapt tool (currently not open source). We know that aAPT generates when it compiles resourcesR file and resources.arsc fileThe R file is a mapping table of resource names and resource ids used to reference resources in Java files, while the resources.arsc file describes the configuration information for each resource ID, that is, how to find the corresponding resource based on a resource ID.
    • Pulbici. XML and ids. XML files aapt are compiled for resources. If the resource files are added or deleted between compilations, the compiled R file may change the resource ID value even if the resource name does not change. Therefore, app may have resource reference disorder during resource reference. XML and ids. XML files do this. Freeline developed id-gen-tool to generate public. XML and ids. XML from R files compiled in the first time.
    • Client-side processing

      Freeline uses the incrementAapt incremental tool to package incremental resource files, then the client places the files in the correct location, and after starting the application, the application resources are properly accessed.


      Paste_Image.png

Freeline implementation analysis

Freeline uses the ideas of Buck and layoutCast for reference in implementation, and constructs the whole process into multiple tasks, which are concurrent, and caches the generated files of each stage at the same time, so as to achieve the purpose of fast construction.

  • Multitask concurrency

    Let’s start with a picture (source)

    Paste_Image.png


    Freeline draws on Buck’s idea here. If there are multiple modules in a project, Freeline will establish task dependencies for each project. Multiple modules may be built at the same time during the build process, and then the files are merged at the appropriate time.

  • The cache

    During debugging, we may make several code changes and run the program to see the effect of the changes. In other words, we need to make several incremental compilations. Freeline caches the compilation process each time. For example, we have done three incremental compilations, and each freeline compilation is for the modified file. Compared with LayoutCast and instant-run, each incremental compilation is to compile the changed file after the first full compilation. Freeline is much faster. According to Freeline official data, this is 3-4 times faster, but it adds a lot of complexity to freeline incremental compilation. In addition, Freeline can be debugged after incremental compilation, which is a great advantage over instant-Run and LayoutCast. The lazy loading mentioned in freeline’s official introduction, I think it’s just icing on the cake, but it may not have much effect in practice.

    The code analysis

    Finally to the link of code analysis, or first posted freeline github address: Freeline, we have a look at its source code what content

Paste_Image.png

Android-studio -plugin is a freeline plugin in Android. Databinding – CLI is a support for Dababinding. Freeline_core is the focus of our analysis today Gradle supports freeline configuration. Release-tools is used during compilation. Aapt and runtime is used for incremental compilation

If you want to compile and debug the source code of freeline incremental compilation, you can clone the source code of Freeline first and then import the sample project. Notice that the sample actually contains the source code of Freeline_core. The IDE I used here is Pycharm.

Freeline’s compilation of Android is divided into two processes: full compilation and incremental compilation. Let’s look at full compilation first.

  • Full amount to compile

    1. Code entry

      The code entry, of course, is freeline.py,

      if sys.version_info > (3.0):
   print 'Freeline only support Python 2.7+ now. Please use the correct version of Python for freeline.'
   exit()
parser = get_parser()
args = parser.parse_args()
freeline = Freeline()
freeline.call(args=args)Copy the code

      Python2.7 freeline is based on Python2.7. Then parse the command:

      parser.add_argument('-v'.'--version', action='store_true', help='show version')
parser.add_argument('-f'.'--cleanBuild', action='store_true', help='force to execute a clean build')
parser.add_argument('-w'.'--wait', action='store_true', help='make application wait for debugger')
parser.add_argument('-a'.'--all', action='store_true',
                   help="together with '-f', freeline will force to clean build all projects.")
parser.add_argument('-c'.'--clean', action='store_true', help='clean cache directory and workspace')
parser.add_argument('-d'.'--debug', action='store_true', help='enable debug mode')
parser.add_argument('-i'.'--init', action='store_true', help='init freeline project')Copy the code

      The Freeline object is then created

      def __init__(self):
   self.dispatcher = Dispatcher()

def call(self, args=None):
   if 'init' in args and args.init:
       print('init freeline project... ')
       init()
       exit()

   self.dispatcher.call_command(args)Copy the code

      Freeline creates dispatcher, from the name can be seen is for command distribution, is in the Dispatcher to perform different compilation process. The checkBeforeCleanBuild command is executed in the init method before the Dispatcher executes the call method, which completes part of the initialization.

    2. Key Modules

      # # # # #dispatcher

      Distribute commands and execute different commands based on the result of command parsing in freeline.py

      # # # # #builder

      Execute various build commands

      Paste_Image.png


      Gradleincbuilder and GradlecleanBuilder are subclasses for incremental and full compilation, respectively.

      # # # # #command

      Paste_Image.png


      With build commands, you can organize multiple commands, and when you pass in builder when you create commands, you can perform different tasks.

      # # # # #task_engine

      Task_engine defines a thread pool, and TaskEngine executes tasks in multiple threads based on their dependencies.

      # # # # #task

      Freeline defines multiple tasks that perform different functions

      Paste_Image.png


      # # # # #gradle_tools

      Some public methods are defined:

      Paste_Image.png

    3. The command to distribute

      The command is parsed in the code entry and then the command is distributed in the Dispatcher:

         if 'cleanBuild' in args and args.cleanBuild:
       is_build_all_projects = args.all
       wait_for_debugger = args.wait
       self._setup_clean_build_command(is_build_all_projects, wait_for_debugger)
   elif 'version' in args and args.version:
       version()
   elif 'clean' in args and args.clean:
       self._command = CleanAllCacheCommand(self._config['build_cache_dir'])
   else:
       from freeline_build import FreelineBuildCommand
       self._command = FreelineBuildCommand(self._config, task_engine=self._task_engine)Copy the code

      Let’s focus on the last line, where FreelineBuildCommand is created, followed by full and incremental compilation.

    4. FreelineBuildCommand

      The first thing you need to decide is whether to do incremental or full compilation, with CleanBuildCommand executed for full compilation and IncrementalBuildCommand executed for incremental compilation

         if self._dispatch_policy.is_need_clean_build(self._config, file_changed_dict):
       self._setup_clean_builder(file_changed_dict)
       from build_commands import CleanBuildCommand
       self._build_command = CleanBuildCommand(self._builder)
   else:
       # only flush changed list when your project need a incremental build.
       Logger.debug('file changed list:')
       Logger.debug(file_changed_dict)
       self._setup_inc_builder(file_changed_dict)
       from build_commands import IncrementalBuildCommand
       self._build_command = IncrementalBuildCommand(self._builder)

   self._build_command.execute()Copy the code

      Let’s look at the is_need_clean_build method

      def is_need_clean_build(self, config, file_changed_dict):
   last_apk_build_time = file_changed_dict['build_info'] ['last_clean_build_time']

   if last_apk_build_time == 0:
       Logger.debug('final apk not found, need a clean build.')
       return True

   if file_changed_dict['build_info'] ['is_root_config_changed']:
       Logger.debug('find root build.gradle changed, need a clean build.')
       return True

   file_count = 0
   need_clean_build_projects = set()

   for dir_name, bundle_dict in file_changed_dict['projects'].iteritems():
       count = len(bundle_dict['src'])
       Logger.debug('find {} has {} java files modified.'.format(dir_name, count))
       file_count += count

       if len(bundle_dict['config') >0 or len(bundle_dict['manifest') >0:
           need_clean_build_projects.add(dir_name)
           Logger.debug('find {} has build.gradle or manifest file modified.'.format(dir_name))

   is_need_clean_build = file_count > 20 or len(need_clean_build_projects) > 0

   if is_need_clean_build:
       if file_count > 20:
           Logger.debug(
               'project has {}(>20) java files modified so that it need a clean build.'.format(file_count))
       else:
           Logger.debug('project need a clean build.')
   else:
       Logger.debug('project just need a incremental build.')

   return is_need_clean_buildCopy the code

      The policy for Freelined is as follows, which can be done by changing this section of code if required.

      2. Changes that cannot rely on incremental implementation: changes to Androidmanifest.xml, changes to third-party JAR references, dependencies on compile-time cuts, annotations, or other code preprocessing plugins. 3. Replace the debugging mobile phone or install an installation package that is inconsistent with the development environment on the same debugging mobile phone.

    5. CleanBuildCommand

         self.add_command(CheckBulidEnvironmentCommand(self._builder))
   self.add_command(FindDependenciesOfTasksCommand(self._builder))
   self.add_command(GenerateSortedBuildTasksCommand(self._builder))
   self.add_command(UpdateApkCreatedTimeCommand(self._builder))
   self.add_command(ExecuteCleanBuildCommand(self._builder))Copy the code

      As you can see, the full amount of the above a few command when actual execution at compile time, we focus on see GenerateSortedBuildTasksCommand, created here exist multiple dependencies task, in task_engine start implemented according to the dependencies, The other commands are similar.

      Paste_Image.png


      Its dependency is throughchildTaskGenerate_sorted_build_tasks in gradLE_clean_build module:

         build_task.add_child_task(clean_all_cache_task)
   build_task.add_child_task(install_task)
   clean_all_cache_task.add_child_task(build_base_resource_task)
   clean_all_cache_task.add_child_task(generate_project_info_task)
   clean_all_cache_task.add_child_task(append_stat_task)
   clean_all_cache_task.add_child_task(generate_apt_file_stat_task)
   read_project_info_task.add_child_task(build_task)Copy the code

      Finally, start task_engine in ExecuteCleanBuildCommand

      self._task_engine.add_root_task(self._root_task)
self._task_engine.start()Copy the code

  • Incremental compilation

    Incremental compilation takes the same steps as before full compilation, creating IncrementalBuildCommand in FreelineBuildCommand

    1. IncrementalBuildCommand

      self.add_command(CheckBulidEnvironmentCommand(self._builder))
self.add_command(GenerateSortedBuildTasksCommand(self._builder))
self.add_command(ExecuteIncrementalBuildCommand(self._builder))Copy the code

      Three commands have been created, so let’s focus on thatGenerateSortedBuildTasksCommandThis is a little more complicated than full compilation.

    2. GenerateSortedBuildTasksCommand

      
def generate_sorted_build_tasks(self):
   """
   sort build tasks according to the module's dependency
   :return: None
   """
   for module in self._all_modules:
       task = android_tools.AndroidIncrementalBuildTask(module, self.__setup_inc_command(module))
       self._tasks_dictionary[module] = task

   for module in self._all_modules:
       task = self._tasks_dictionary[module]
       for dep in self._module_dependencies[module]:
           task.add_parent_task(self._tasks_dictionary[dep])Copy the code

      You can see the first create AndroidIncrementalBuildTask traverse each module, after traversal mudle create task dependencies. When creating AndroidIncrementalBuildTask incoming GradleCompileCommand

    3. GradleCompileCommand

      self.add_command(GradleIncJavacCommand(self._module, self._invoker))
self.add_command(GradleIncDexCommand(self._module, self._invoker))Copy the code

      Take a look at GradleIncJavacCommand

         self._invoker.append_r_file()
   self._invoker.fill_classpaths()
   self._invoker.fill_extra_javac_args()
   self._invoker.clean_dex_cache()
   self._invoker.run_apt_only()
   self._invoker.run_javac_task()
   self._invoker.run_retrolambda()Copy the code

      Perform the above several functions, the specific content can view the source code. The depth of each task is defined according to the parent_task list in the task:

      def calculate_task_depth(task):
   depth = []
   parent_task_queue = Queue.Queue()
   parent_task_queue.put(task)
   while not parent_task_queue.empty():
       parent_task = parent_task_queue.get()

       if parent_task.name not in depth:
           depth.append(parent_task.name)

       for parent in parent_task.parent_tasks:
           if parent.name not in depth:
               parent_task_queue.put(parent)

   return len(depth)Copy the code

      Tasks are sorted by depth at execution time

         depth_array.sort()

   for depth in depth_array:
       tasks = self.tasks_depth_dict[depth]
       for task in tasks:
           self.debug("depth: {}, task: {}".format(depth, task))
           self.sorted_tasks.append(task)

   self._logger.set_sorted_tasks(self.sorted_tasks)

   for task in self.sorted_tasks:
       self.pool.add_task(ExecutableTask(task, self))Copy the code

      Each task execution then determines whether the parent has completed

      while not self.task.is_all_parent_finished(): # self.debug('{} waiting... '.format(self.task.name)) self.task.wait()Copy the code

      The task can be executed only after the parent task is completed. This paper simply analyzes the implementation of Freeline from the perspective of incremental compilation principle and code. The principle part mainly refers to the Chinese principle explanation, and the code part mainly analyzes the general framework without going into every detail, such as how Freeline supports APT and lambda, etc., so maybe we will continue to write the analysis later. I am uneducated, if there is an analysis of the wrong place, please point out.

Reference github.com/alibaba/fre # #… Yq.aliyun.com/articles/59… www.cnblogs.com/Pickuper/ar… Blog.csdn.net/luoshengyan…