Author: Idle Fish technology – adjacent cloud

background

When developing pictures and video related functions, album is an unavoidable topic, because everyone basically needs to get pictures or videos from album. The most direct way is to call the system album interface, although the basic function is satisfied, but can not meet some advanced functions, such as custom UI, multi-selection of pictures, etc..

Design ideas

Xianyu this album component API is simple to use, rich and flexible functions, with high customization. The business side can choose to fully access the component or customize the UI on top of the component.

Flutter serves as the UI presentation layer, and the specific data is provided by each Native platform. This pattern creates a natural engineering separation between UI code and data code. We often use the MVC architecture when developing a native component. The Idea behind the Development of the Flutter component is similar. The overall structure is as follows:

It can be seen that the Flutter side is a typical MVC architecture, where the Widget is the View and the View is bound to the Model. When the Model changes, the View will be rebuilt to reflect the changes of the Model. The View event triggers the Controller to fetch data from Native and update the Model. Native and Flutter communicate through Method Channel. There is no strong dependence between the two layers and they only need to communicate according to the agreed protocol.

As part of the Native side, UIAdapter is mainly responsible for the adaptation of models, recognition of fringe screen and full screen. Permission Processes media read and write Permission applications. The Cache is mainly responsible for caching GPU textures to improve response speed during large image preview. Decoder is responsible for parsing Bitmap, OpenGL is responsible for Bitmap to texture.

It should be noted that our implementation of flutter relies on surrounding textures. Most of the images seen throughout the album component are a GPU texture, which gives the Java heap memory footprint a significant reduction compared to previous album implementations. If you use the native system album on the low-end machine, the app may be killed by the system due to memory problems. When you return from the system album, your app reboots. With the Flutter Albums component, the experience on low-end devices will be a bit different.

Some difficulties

Paging load

The album list requires a large number of images to load. The GridView component of Flutter has several constructors. The easy mistake to make with Flutter is to use the first one, which requires a large number of widgets to start with. The second constructor should be selected, and the GridView will call back to the IndexedWidgetBuilder to get the widget as it slides, equivalent to lazy loading.

GridView.builder({... List<Widget> children = const <Widget>[],... }) GridView.builder({... @required IndexedWidgetBuilder itemBuilder,int itemCount,... })Copy the code

In the process of sliding, after the image slides, that is, when it is not visible, resources should be recycled. The corresponding texture here and here is deleted. So if you keep sliding, the memory will be stable as it goes up, it won’t keep growing. If the texture is repeatedly created and deleted by sliding it back and forth quickly, it will shake the memory and the experience will not be very good.

So we maintain a picture state machine with states like None,Loading,Loaded,Wait_Dispose,Disposed. When Loading starts, the state changes from None to Loading. At this time, the user sees blank or placeholder map. When the data callback comes back, the state is set to Loaded and the widget tree is rebuilt to display the image icon. It will not Dispose immediately. If the user slides back, it will enter Loaded state from Wait_Dispose and will not continue to Dispose. If the user doesn’t slide back then they go from Wait_Dispose to Disposed. Now when you’re in the Disposed state, you need to reload when you want to display that image.

Photo album large picture display

When clicking on an image in the GridView, a large image of the image will be displayed for users to see more clearly. We knew that the camera’s images were at a very high resolution, and that it would be expensive to load them completely, so we scaled them up to 1080p when Decode the Bitmap. The Android native Bitmap Decode experience also applies, first Decode out the width and height of the Bitmap, and then calculate the zoom factor according to the size to be displayed, and then Decode out the required Bitmap.

Most of the pictures in Android album have rotation Angle. If they are not displayed directly, there will be a 90 degree rotation problem, so we need to rotate the Bitmap. It takes about 200ms to rotate a 1080p picture using Matrix on my test machine. If using OpenGL texture coordinates for rotation, it only takes about 10ms, so using OpenGL for texture rotation is a better choice.

A horizontal sliding PageView is entered when a Flutter preview is performed. Normally, the PageView of a Flutter will not actively load adjacent pages. The viewportFraction parameter for PageController can be set to 0.9999, as shown below:

PageController (viewportFraction = 0.9999)Copy the code

Another option is to do preloading on Native. For example, when loading the 5th image, the adjacent 4 and 6 image textures are loaded ahead of time, and when sliding to 4 and 6, the cached textures are used directly.

memory

The use of GPU texture for album images can greatly reduce the occupation of Java heap memory and improve the performance of the entire APP. It should be noted that the GPU memory is limited and should be deleted after use, otherwise there may be a risk of memory leakage. In addition, when deleting a texture on the Android platform, ensure that it is done on the GPU thread, otherwise the deletion will have no effect.

Comparison tests on huawei P8 and Android5.0 show that the total memory usage of Flutter album is basically the same as that of the original native album. In the GridView list page, the maximum memory added is about 13M. The difference is that the original Flutter albums use Java heap memory, while the Flutter albums use Native memory.

conclusion

The album component API is simple, easy to use and highly customizable. The Flutter side is well layered, and those with UI customization requirements can rewrite widgets to achieve this. In addition, this is an album component that does not depend on the system album. It is complete and can maintain the consistency of UI and interaction with the existing APP. It also lays the foundation for supporting more album related gameplay in the future.

The follow-up plan

Since we are using GPU textures, we can consider supporting the display of HD 4K images without too much stress on client memory. However, it takes more time to convert 4k images from Bitmap to texture, so we will consider supporting loading state in UI interaction in the future.

The component is feature-rich and will be open sourced to the community after running stably online for a period of time. Please pay attention to xianyu Technology official account.