Introduce TensorFlow Lite

TensorFlow Lite is a lightweight solution customized to solve the problem of overloading TensorFlow on both mobile and embedded platforms. It is a completely separate project from TensorFlow and has little code sharing with TensorFlow. TensorFlow itself is designed for desktop and server applications and is not optimized for ARM mobile platforms, so it is not comfortable to use directly on mobile or embedded platforms. TensorFlow Lite implements a low-power, low-latency machine learning framework for mobile platforms and makes the compiled binary distribution smaller. TensorFlow Lite not only supports traditional ARM acceleration, but also provides support for the Android Neural Networks API, providing better performance on ann-enabled devices.

TensorFlow Lite not only uses ARM Neon instruction set acceleration, but also presets activation functions that provide quantization, speed up execution, and reduce model size.

TensorFlow Lite features

TensorFlow Lite has a number of features that enable it to perform well on mobile platforms. The following are the features of TensorFlow Lite:

1. Support for a set of core operators, all of which support floating-point input and quantized data, that are individually optimized and customized for mobile platforms. These operators also include preset activation functions to improve computing performance on mobile platforms while ensuring the accuracy of quantized calculations. We can use these operators to create and execute our own custom model, or we can write our own custom operator implementation if we need some special operators in the model.
2. A new model file format is defined for mobile platforms that is based on FlatBuffers. FlatBuffers is a high-performance open source cross-platform serialization library that is very similar to Protobuf, but the big difference between the two is that FlatBuffers do not require any parsing or reporting of the data before it is accessed (corresponding to Protobuf’s compression mechanism). Because the data format of FlatBuffers is generally memory aligned. In addition, the code for FlatBuffers is smaller than for Protobuf, making it easier to integrate with mobile platforms. Therefore, TensorFlow Lite and TensorFlow model file formats are different.
3. Provides a web interpreter optimized for mobile platforms, which makes the entire code much simpler and faster. The idea of optimizing the interpreter is to use static graph paths to speed up decision-making at runtime, while customizing the memory allocator to reduce dynamic memory allocation, ensuring reduced model loading and initialization time and resource consumption, and improving execution speed.
4. Hardware acceleration interface is provided, one is the traditional ARM instruction set acceleration, one is the Android Neural Networks API. If the target device is running Android 8.1 (API 27) or later, you can use Android’s built-in acceleration API to speed up the overall execution.
5. Model transformation tools are provided to convert training models generated by TensorFlow into TensorFlow Lite models. This solves the problem of the different TensorFlow and TensorFlow Lite model formats.
6. The compiled binary volume is very small. Under the optimized setting of O3 with ARM Clang, the whole library is less than 300KB after compilation, which can basically meet the operators required by most deep learning networks at present.
7. Both Java and C++ apis are provided to facilitate the integration of TensorFlow Lite into Android apps and embedded applications.

TensorFlow Lite architecture

To understand how TensorFlow Lite achieves these features, we must first understand the architectural design of TensorFlow Lite. The architecture design of TensorFlow Lite is shown in The following figure.

We can understand the difference between TensorFlow Lite and TensorFlow in this way. First, we need to train a TensorFlow model file. Then use the TensorFlow Lite model converter to convert the TensorFlow pattern into a TensorFlow Lite model file (.tflite format). The converted files can then be used in mobile applications. We can use TensorFlow Lite on Android and iOS to load the converted. Tflite model file. TensorFlow Lite provides the following invocation methods.

1. Java API: the Java API encapsulated by C++ API on Android is easy to call directly by the application layer of the Android App.
2. C++ API: can be used to load TensorFlow Lite model files to construct the call interpreter. The API is available on both Android and iOS platforms.
3. Interpreter: An accounting method that executes the model and invokes operators based on the network structure. Accounting method loading is optional. If we don’t use accelerated accounting, we only need 100KB of space; If all the accelerated accounting methods are linked, it is only 300KB, which is very small overall. On some Android devices, the interpreter directly calls the Android Neural Networks API for hardware acceleration.

Finally, we can implement our own accounting method through the C++ API and load it through the interpreter for deep web implementation.

TensorFlow Lite code structure

TensorFlow Lite and TensorFlow are two basically independent sets of code, the complete code can be found in TensorFlow’s GitHub repository illustration. Here is a brief introduction to the code structure of TensorFlow Lite, which will be of great help later in the game. There is a Lite directory under the TensorFlow code root, which is the complete TensorFlow Lite code. To make it easier for the reader to read the complete TensorFlow code, here is a brief introduction to the TensorFlow Lite code structure and some important files, as shown in the following table.