The background,

With the development of mobile platforms, the scale of mobile users is getting bigger and bigger, and the demand for products is also growing. In order to solve many rapidly iterating business product lines and requirements and improve our development efficiency, industry peers have tried to explore many cross-platform solutions, and the mainstream solutions today are roughly as follows. Such as:

  1. The React Native.
  2. Weex;
  3. Hybrid App;
  4. Flutter.
  5. Small program;

More or less each of the above solutions has bottlenecks or usage scenarios that will not be discussed here. For your reference, here are the main comparisons:

Package name React Native Weex Hybrid App Flutter Small program
Platform to realize JS JS There is no bridge There is no bridge There is no bridge
engine JSCore JS V8 Native rendering Flutter engine –
The core of language React Vue Java/Obeject-C Dart WXML
Apk size (Release) About 4 to 6 m About 10 m – Around 8 to 10 m –
Bundle File size Default single, large Small, many pages can be many files Don’t need Don’t need Don’t need
Start difficulty (native Angle) easy general general easy easy
Degree of framework The heavier The lighter heavy The heavier The lighter
The characteristics of Suitable for overall App development Suitable for single page Suitable for overall App development Suitable for overall App development Suitable for overall App development
community Richness (FaceBook) General (Ali) general Richness (Google) General (wechat)
Cross-platform support Android, iOS, Android, iOS, Android, iOS, Android, iOS, Web, Fuchsia, etc Android, iOS,

As a cross-platform solution that has been developing rapidly in the last two years, Flutter has entered the scope of our research. We will mainly measure it in the following aspects:

  1. Access difficulty.
  2. Cost of learning.
  3. Performance.
  4. Package volume.
  5. Dynamic capability.

Second, we will explore a dynamic solution

I’m sure you already know some of the previous aspects of Flutter, so I won’t go into any more detail here. As a cross-platform solution, dynamism is one of the most important functions. Through research, documents, and technical group communication and discussion, we found that there are three implementations of Flutter:

  1. Similar to React Native framework.
  2. Replace the Flutter compilation product.
  3. Page dynamic component framework.

Next, we will briefly introduce the concrete implementation principles of these solutions.

1. Dynamic component solution

At present, most technical teams in the market are through the idea of dynamic components of the page to achieve dynamic, such as Xianyu, Vipshop, headlines, etc. The core principle of this scheme is that before application packaging, such as piling/embedding dynamic widgets into the code during compilation, and then dynamically delivering Json data, and dynamically replacing Widget content by matching the data in Json with agreed semantics to achieve dynamic (except UI, if dynamic logical code needs to be implemented, You can write the business logic in a more dynamic scripting language like Lua).

The summary features are as follows:

  1. There are many similar mature frameworks in the market, such as Tangram of Tmall, DinamicX of Taobao, etc. It achieves a relatively good balance between performance and dynamics and development costs. It can meet the dynamic requirements of common situations and solve practical problems to a certain extent.
  2. Supports dynamism on both Android and iOS.
  3. UI dynamic is relatively easy, business logic dynamic is more troublesome.
  4. Semantic parsers are relatively costly to develop and difficult to maintain.

1.1 About syntax trees

Tangram, DinamicX and other frameworks have one thing in common: they all use Xml or Html as DSL. But Flutter is React Style grammar, and the grammar of the Flutter itself is pretty good at expressing the page. Therefore, no custom syntax is required for this scenario. Use the Flutter source code as a DSL. This greatly eases the development and testing process without the need for additional tool support.

Analyze the source code to get ASTNode process:

As can be seen from the figure above, the plug-in or command initiates a request to analysis Server with the file path to be analyzed and the type of analysis. Analysis_server obtains commilationUnit (ASTNode) by using package: Analyzer, then analyzes ASTNode by computer and returns a list of analysis results.

According to the principle of Flutter, we can also use Package: Analyzer to convert source files to a commilationUnit (ASTNode), ASTNode is an abstract syntax tree (AST), which is a tree-like representation of the abstract syntax structure of the source code. It can be used to parse the Dart source code well.

Scheme defects:

You need to make rules about the format of the source code. For example, writing an if else is not supported and you need to use a logical WigET component instead of an if ELSE statement. Without rules, parsing from the AST Node to the Widget Node can be complicated. So you can introduce Lua to make your logical code dynamic.

1.2 Overall architecture planning of JSON + Lua solution:

Open source solution:

Github.com/dart-lang/s…

Github.com/dengyin2000…

Luakit_plugin:github.com/williamwen1…

References:

The dart. Dev/tools/darta…

Yq.aliyun.com/articles/67…

2. Rn-like scheme (JS Bundle)

According to the design idea of React Native, it can be summarized as replacing DartVM with JavasSriptCore and transforming XML DSL into atomic widget component of Flutter with JavaScript (JS for short). Then let the Flutter render. It’s technically possible, but it’s expensive, it’s a huge project.

To be specific, the first tree (WidgetTree) among the three trees (WidgetTree, Element and RenderObject) in the rendering logic of Flutter is generated in JS. JS is used to complete the encapsulation of the Flutter control layer, and JS can be used to develop the application of Flutter in a similar way to Dart. Using the JavaScript version of the lightweight Flutter Runtime, a UI description is generated and passed to the Dart UI engine, which then generates the UI description to produce the actual Flutter control.

Open source solution of mobile QQ Watch Team: high-performance Flutter Dynamic Framework Based on JS

Scheme pitfalls: No matter how fast JSWidget is created, there will always be cross-language execution and there will always be a performance impact. In addition, because iOS has built-in support for JS, it is fully dynamic on iOS, but Android requires the introduction of additional JS libraries. Currently, the MXFlutter scheme is only dynamic for iOS, and it is complicated to implement.

3. Replace the compilation product scheme

To make the compilation artifact dynamic, on The Android platform, it is limited to JIT code; On iOS, you’re limited to interpreting the code that executes. The Google Flutter team previously tried to provide an official solution, but gave up and rolled back the code. They are saying is with such a solution under the platform limitation, the performance of the iOS platform can achieve expected and not too much confidence (say simply is, will run on iOS CARDS had to bear, can’t let a person because he does not like android iOS, can be directly loaded dynamic library so, it needs to be loaded is static library) as a result, This compilation artifact replacement is currently only available on the Android side.

First of all, we need to know what the compilation product of Flutter is. As we know, the compilation product of Android is dex file. By changing the loading process of DEX file, we can achieve the purpose of dynamic. So, let’s take a look at the compilation of The Flutter. It’s important to note that the current iteration rate of the Flutter is very fast and that the compilation of the Flutter is not consistent across different versions.

3.1 Compilation instructions for Flutter

(1) Compile apK & AAR default engine

// Compile the pure Flutter APK, The default is the release version of the flutter build APK // the pure debug version of the flutter build APK The default is release version FLUTTER build AAR. The default is debug version FLUTTER build AAR --debugCopy the code

The instructions for compiling can be viewed with the Flutter build -h, as shown in the screenshot below:

(2) Compile apK & AAR to specify the local engine

  • For more information on how to build the Engine, check out this article [Building the Ubuntu 16.04 Flutter Engine](/home/lichaojian/ documentation /Ubuntu 16.04 Building the Flutter Engine. Md)

  • How do I reference the local engine

// specify a reference to the local engine to compile the APK, Flutter build APK --target-platform Android-arm64 --local-engine= Android_release_arm64 - local - engine - SRC - path = / home/lichaojian/engine/SRC / / specified reference local engine to compile the aar, Flutter build AAR --target-platform Android-arm64 --local-engine= Android_release_arm64 --local-engine-src-path=/home/lichaojian/engine/srcCopy the code

(3) View the source code of the Flutter compilation instruction

Whether we compile apK or AAR, we do so with the flutter directive, so take a look at what the source code for the flutter directive actually is. /your_flutter_sdk_path/bin/flutter

FLUTTER_TOOLS_DIR="$FLUTTER_ROOT/packages/flutter_tools"
SNAPSHOT_PATH="$FLUTTER_ROOT/bin/cache/flutter_tools.snapshot"
STAMP_PATH="$FLUTTER_ROOT/bin/cache/flutter_tools.stamp"
SCRIPT_PATH="$FLUTTER_TOOLS_DIR/bin/flutter_tools.dart"
DART_SDK_PATH="$FLUTTER_ROOT/bin/cache/dart-sdk"

DART="$DART_SDK_PATH/bin/dart"
PUB="$DART_SDK_PATH/bin/pub"

# FLUTTER_TOOL_ARGS isn't quoted below, because it is meant to be considered as
# separate space-separated args.
"$DART" --packages="$FLUTTER_TOOLS_DIR/.packages" $FLUTTER_TOOL_ARGS "$SNAPSHOT_PATH" "$@"
Copy the code
  • $DART: Starts a DART VM
  • $SNAPSHOT_PATH: specifies an executable snapshot file. The path is /your_flutter_sdk_path/bin/cache/flutter_tools.snapshot
  • $@: is the parameter you passed in (e.g. Build apk)
// As can be seen from the above, The flutter build apK is /your_flutter_sdk_path/bin/cache/dart-sdk/bin/dart /your_flutter_sdk_path/bin/cache/flutter_tools.snapshot build apkCopy the code

Apk is successfully compiled. The same is true with AAR:

  • The next check flutter_tools. The snapshot of the source code files, in the/your_flutter_sdk_path/flutter/package/flutter_tools/bin/flutter_tools. Dart

As you can see from the screenshot above, the executable. Main method was actually called, so let’s take a look at the executable

As you can see, the Runner is running a series of Command classes, and the one we’re familiar with is of course the flutter build Command, so we can look at the Command that the flutter build corresponds to is BuildCommand

As you can see from the figure above, BuildCommand is actually composed of many sub-commands, such as AAR, APK, aOT, etc.

To learn more about the packing and compilation process of the Flutter, it is recommended to check out [Read the Packing and Compilation Process of the Flutter]

3.2 Differences in compilation products under different versions of Flutter

(v1.5.4-hotfixes & v1.9.1)

  • V1.5.4 – hofixed

(1) Debug mode

(2) Release mode

  • v1.9.1

(1) Debug mode

(2) Release mode

As you can see from the screenshot above, in debug mode, v1.5.4-hofixes and V1.9.1 products don’t change much. We won’t discuss the debug version here, so we can ignore them, but we can see the differences as follows:

V1.5.4 – Hotfixes Release

  • isolate_snapshot_instr
  • isolate_snapshot_data
  • vm_snapshot_data
  • assets/vm_snapshot_instr

Product in V1.9.1 release mode

  • libapp.so

The products under v1.9.1 release mode are mainly divided into the following categories:

  • /lib/libapp.so builds Dart’s generated executable
  • /lib/libflutter. So contains the executable files of the Flutter Engine
  • /assets/flutter_assets mainly stores some resources files of flutter, such as fonts, pictures, etc.

As you can see, after v1.9.1, the code compilation product of Flutter has become much simpler, which is helpful for dynamic research. We know that libapp.so naturally supports dynamic linking. This means that we can replace the libapp.so file to make it dynamic. In the beginning, Lisson replaced the product directly by the root phone, and found it was supported, and then we had the follow-up.

Since libapp.so is supported for dynamic updates, how do we do this in code?

3.3 How does Flutter dynamically replace compiled products?

From the above, we know what the compilation product of the Flutter is, so we can dynamically specify the path to load the compilation product of the code to achieve the dynamic effect. How can we change the code? There are two ways:

(1) By modifying the Flutter Engine.

Advantages:

  • Familiarize yourself with Engine code.
  • Customizable extension Engine.

Disadvantages:

  • Intrusive code for Engine.
  • You need to maintain your own Engine and provide it externally.
  • Regular updates are required to synchronize the official Engine code.

(2) By Hook.

Advantages: Less intrusive to Engine code.

Disadvantages: When you need to maintain the SDK and Engine version update, you need to follow up whether the hook point needs to be replaced.

3.3.1 So file replacement process

  • (1) How Android loads so files.

    The release version of the flutter is compiled into a libapp.so file on Android. What are the methods of loading the so file on Android? It is mainly divided into the following two types:

// The default load path is ~/app/libs
System.loadLibrary("libname")
    
// Load by absolute path
System.load("/your_so_path/libupdate.so")
Copy the code

The main difference between these two methods is that loadLibrary loads the so file in the libs directory of the app, and load loads the absolute path of the file.

For Android, the principle of loading. So file, you can see Gityuan loadLibrary dynamic library loading process analysis, you can also take a look

Learn more about the System. LoadLibrary article.

In short, this is done by calling the function in the header file DLFCN. H, as follows:

void *dlopen(const char *filename, int flag);  // Open the dynamic link library
char *dlerror(void);   // Get the error message
void *dlsym(void *handle, const char *symbol);  // Get the method pointer
int dlclose(void *handle); // Close the dynamic link library
Copy the code

Having seen how Android loads so files, let’s take a look at how the Flutter loads so files.

  • (2) How the Flutter loads so files.

  1. By looking at the source code to initialize the Flutter, we know that there are two methods that must be called.

    FlutterMain.startInitialization(@NonNull Context applicationContext)
FlutterMain.ensureInitializationComplete(@NonNull Context applicationContext, @Nullable String[] args)
Copy the code

Omit a lot of……. Go straight to the key….

native_library_posix.cc

NativeLibrary::NativeLibrary(const char* path) {
  ::dlerror();

  FML_LOG(ERROR)<< "lichaojian-path = " << path;
  
  handle_ = ::dlopen(path, RTLD_NOW);
  if (handle_ == nullptr) {
    FML_DLOG(ERROR) << "Could not open library '" << path << "' due to error '"
                    << ::dlerror() << "'.";
  }
}

fml::RefPtr<NativeLibrary> NativeLibrary::Create(const char* path) {
  auto library = fml::AdoptRef(new NativeLibrary(path));
  FML_LOG(ERROR)<< "lichaojian-Create = " << path;
  returnlibrary->GetHandle() ! =nullptr ? library : nullptr;
}
Copy the code

As you can see from the above instructions, dlopen is actually called to load the so library. So with that in mind, we know what to do. I add the log to the two functions as follows:

After knowing the principle, the implementation of Flutter dynamic loading so file can be divided into two parts:

(1) the native layer

By changing the native layer code, the native layer can judge whether there is an updated file in a predetermined path. If there is, the updated file will be loaded; if not, the original libapp file will be loaded.Copy the code

(2) the Java layer

Just in FlutterMain# ensureInitializationComplete method, we can see libapp related parameters, there are two lines of code is important, let’s review the:

private static final String DEFAULT_AOT_SHARED_LIBRARY_NAME = "libapp.so";
private static String sAotSharedLibraryName = DEFAULT_AOT_SHARED_LIBRARY_NAME;

shellArgs.add("--" + AOT_SHARED_LIBRARY_NAME + "=" + sAotSharedLibraryName);

                // Most devices can load the AOT shared library based on the library name
                // with no directory path. Provide a fully qualified path to the library
                // as a workaround for devices where that fails.
                shellArgs.add("--" + AOT_SHARED_LIBRARY_NAME + "=" + applicationInfo.nativeLibraryDir + File.separator + sAotSharedLibraryName);
Copy the code

From the above code, we can see that aot_share_library_name and its path are loaded into shellArgs parameter, so we can change the path and name in Java layer to achieve the purpose of dynamically loading so. If you do not change the name, when you find the name of libapp.so, it will be directly mapped to the libapp.so file in the lib directory, so that the dynamic loading of so is invalid.

Dynamic loading is achieved by replacing a so file whose path has changed its name.

3.3.2 Resource Replacement

About Flutter Resources

Compared to the resource management of Android system, The resource management of Flutter is much simpler. The resources of Flutter are completely exposed in the form of source files without any processing of compilation, and resources are obtained by reading files. What is returned to the Flutter is the buffer of the resource in memory, identified by directory name + file name, as follows:

About the flutter AssetManager

There is also an AssetManager inside the Flutter Engine. The source path is Flutter /assets/asset_manager.h There is not much code in the AssetManager, but a queue of AssetResolvers is maintained internally. There are two core approaches

// Add an AssetResolver to the queue
void AssetManager::PushBack(std: :unique_ptr<AssetResolver> resolver) {
  if (resolver == nullptr| |! resolver->IsValid()) {return;
  }

  resolvers_.push_back(std::move(resolver));
}

// Retrieve the resource
std: :unique_ptr<fml::Mapping> AssetManager::GetAsMapping(
    const std: :string& asset_name) const {
  if (asset_name.size() == 0) {
    return nullptr;
  }
  TRACE_EVENT1("flutter"."AssetManager::GetAsMapping"."name",
               asset_name.c_str());
  for (const auto& resolver : resolvers_) {
    auto mapping = resolver->GetAsMapping(asset_name);
    if(mapping ! =nullptr) {
      return mapping;
    }
  }
  FML_DLOG(WARNING) << "Could not find asset: " << asset_name;
  return nullptr;
}
Copy the code

As you can see from the code, the real resources are provided by the AssetResolver.

About the flutter AssetResolver

AssetResolver is an interface class that the Flutter resource provider must implement under the source code flutter/assets/asset_resolver.h, which is loosely defined as follows

namespace flutter {

class AssetResolver {
 public:
  // What is not important is omitted...
  virtual std: :unique_ptr<fml::Mapping> GetAsMapping(
      const std: :string& asset_name) const = 0;

 private:
  FML_DISALLOW_COPY_AND_ASSIGN(AssetResolver);
};

}  // namespace flutter
Copy the code

The core method is GetAsMapping. This method returns a file Mapping. Mapping is also an interface class, and its definition is very simple

class Mapping {
 public:
  // What is not important is omitted...
  virtual size_t GetSize(a) const = 0;
  virtual const uint8_t* GetMapping(a) const = 0;
};
Copy the code

GetSize returns the size of the file, and GetMapping returns the address of the resource in memory. The management structure of the entire resource is roughly as follows:

About the flutter APKAssetProvider

APKAssetProvider implements the AssetResolver interface, which provides flutter with resource acquisition capabilities on the Android platform, The essence is to convert the Java layer AssetManager to the C++ layer via the AAssetManager_fromJava interface, AAssetManager_open AAsset_getBuffer AAsset_close Source code paths in the flutter/shell/platform/android/apk_asset_provider. H below, not the code, directly call process is given here

Several scenarios for the dynamic deployment of flutter resources

  • The Android platform

From the previous code analysis, we can clearly see that the resources for Flutter are also provided by AssetManager on the Android platform, so we can use the principle of hotfix (which is much easier than hotfix, because we don’t need to do full synthesis here, Do a semi-full synthesis, and you don’t need to go to the Replace system’s AssetManager. Just call addAssetPath.) Of course, this solution will have to address Android 9’s limitations on private apis.

  • Cross-platform common approach

The drawbacks of the above solution are obvious. First, it can only satisfy the Android platform; second, it needs to solve the system’s constraints on private API, etc. In fact, before the first solution, the author and I have first implemented a common cross-platform method based on c++ layer, and its process and principle are also very simple. All we need to do is implement our own AssetResolver and Mapping and then cram the AssetResolver into the AssetManager queue of our flutter.

  • Take advantage of the native support solution provided by Flutter

This approach was discovered last night when I was writing this article, so it hasn’t been tested yet, but it’s theoretically feasible and seems to be the simplest and most effective solution so far.

In RunConfiguration we can find the following code (source path flutter/shell/common/run_configuration.h)

RunConfiguration RunConfiguration::InferFromSettings(
    const Settings& settings,
    fml::RefPtr<fml::TaskRunner> io_worker) {
  // I have deleted the unimportant code below...
  if (fml::UniqueFD::traits_type::IsValid(settings.assets_dir)) {
    asset_manager->PushBack(std::make_unique<DirectoryAssetBundle>(
        fml::Duplicate(settings.assets_dir)));
  }
  asset_manager->PushBack(
      std::make_unique<DirectoryAssetBundle>(fml::OpenDirectory(
          settings.assets_path.c_str(), false, fml::FilePermission::kRead)));
}
Copy the code

In fact, InferFromSettings here is for Fuchsia (Flutter cross-platform, you can see all over the engine project fuchsia android ios Windows Linux Darwin directory structure), We can’t call this function directly, but DirectoryAssetBundle is public. (In fact, ios does not wrap an APKAssetProvider like Android does.) DirectoryAssetBundle is essentially an implementation of the AssetResolver, The source path is under the flutter/assets/directory_asset_bundle.h, which will not be analyzed here.

3.3.3 Implementation Process

General process of scheme

The principle is similar to that of Tinker and other Android hot update schemes, which compare the files between the old and new versions and generate a fix pack. The patch package is then placed on the server and distributed to users of the older VERSION of APK. After downloading the package, decompress it locally and merge it for full replacement.

Generation and merging of subcontractors

At the beginning of the search, bsdiff and BSPatch were recommended, but the official website only had C code. At this time, I was confused, so I directly ported the code to Android platform through THE NDK. When porting and compiling the dynamic library, I made some mistakes. However, it is mainly the novice pit caused by some unfamiliarity with CMake and c++.

Some reference links to BSDIFF:

bsdiff.pdf

Bsdiff algorithm

Google’s differential update actually uses Bsdiff as well

Tinker is based on Java code encapsulated in Bsdiff V4.2

As for the generation and merging of subpackages, it is all in the existing framework, so it is not very difficult.

conclusion

This paper mainly explores and explains the three mainstream dynamic implementation schemes of Flutter, dynamic component schemes and Js schemes such as RN, which are all implemented through AST parsing semantic tree in essence. And the dynamics of the compilation product, through the analysis of the source code found that can be achieved in the Android platform, iOS platform is not a good solution. Android’s product compiler dynamic solution is relatively easy to achieve at present, the difficulty is not too big. In the process of exploration, more or less stepped on some pits, if the article is not enough, we also hope that more corrections ~

Thanks for reading.

The author



xiaosongzeem