Improve Android video performance on News Feed with Litho

Improving Android Video on News Feed with Litho

As video consumption on mobile devices grows rapidly, Facebook mobile engineers face new challenges in efficiently presenting content. Videos require more resources than simple UI elements such as text and photos. They use decoders that keep the CPU busy; They allocate a lot of memory for setup; They use more network bandwidth to download video data from servers. Playing video in scrollable containers like News Feed is particularly challenging — the pressure on device resources risks losing frames, which can lead to a shaky scrolling experience. Also, we don’t want people waiting for the video to load or buffering during playback, so the video player needs to start up quickly and run smoothly. All of these challenges are exacerbated by Android, given the wide variety of devices on the market.

We recently completed a News Feed migration for Android, powered by Litho, our open source UI rendering framework. Litho’s support for asynchronous layout and fine-grained recycling not only helped optimize the News Feed to render content more efficiently, but also made our code more robust and easier to scale. We want to bring these benefits to video to improve the playback experience for Facebook users and engineers designing new use cases.

Building video UI elements is more challenging than any other UI part we’ve dealt with. The video component takes advantage of advanced features not required by other Litho components and activates features that didn’t exist before. The new Litho Video component introduces several improvements, from the scrolling performance of News Feed to a more flexible design that can be easily repeated for various video functions. This article describes the challenges we faced in rewriting the video player in our Android app, and how Litho helped overcome those challenges.

Video in the news feed

The news feed supports multiple video story types. Some people behave and look different from others. A common type of story is video attachment, where regular video, Facebook live video, 360-degree video, GIF, or other formats are attached to regular posts.

Other video story types can play generated videos, sponsored messages, or short animations.

Supporting all of these variants makes code difficult to maintain and test. The main video view for our video attachment is called VideoAttachmentView. For some story types, we must extend this view to reuse and customize it to fit the design. In some cases, we cannot make all the changes in a derived class and must create a separate view class, which means moving the public logic to the helper class. This further complicates the code.

The scrolling performance of the News Feed was also affected. RecyclerView only from the same class type recycling view, this means that they don’t like SponsoredVideoAttachmentView conventional story video attachment recycling view, Even if VideoAttachmentView SponsoredVideoAttachmentView extension. Because of this inefficiency, RecyclerView allocates more views to suit different story types, resulting in a larger memory footprint. In addition, assignment is triggered before the video view needs to appear on the screen, increasing the chance that it won’t be ready in time and one or more frames will be lost. Even though VideoAttachmentView reuses the same content in the new layout to modify the look and feel of the VideoAttachmentView, we need to create a new view type. Because we are adding another layer, this also makes the view hierarchy deeper. The deeper the view hierarchy, the longer it takes to render.

Designing video components

Litho defines the UI using units called “components.” It supports two main types of components: MountSpec and LayoutSpec. MountSpec defines UI building blocks, such as text or image components. A LayoutSpec defines a layout that contains one or more parts. A common practice in Litho is to use composition of components to build more complex components. LayoutSpec can define a layout that contains other LayoutSpecs and MountSpec components, while MountSpec renders a specific view or drawable object.

The video component is a UI building block, which means we need a MountSpec to define it. One option is to create a render MountSpec VideoAttachmentView, but then we need to create a MountSpec for other extensions view, such as SponsoredVideoAttachmentView. While this works, we end up with the same reuse and maintainability issues. Another approach is to create a MountSpec to render a simple video view instead of VideoAttachmentView and add it to a LayoutSpec to adjust it for a particular story type. The following diagram illustrates the relationship between the new components:

The CoreVideoComponent is a MountSpec with the minimum functionality required for any video story.

@MountSpec public class CoreVideoComponentSpec { @OnCreateMountContent static SimpleVideoView onCreateMountContent(ComponentContext context) { return new SimpleVideoView(context); } @OnPrepare static void onPrepare( ComponentContext context, @Prop VideoParams videoParams) { prefetchVideo(videoParams); } @OnMount static void onMount( ComponentContext context, SimpleVideoView videoView, @Prop VideoParams videoParams) { initVideoPlayback(videoView, videoParams); } @OnUnmount static void onUnmount( ComponentContext context, SimpleVideoView videoView, @Prop VideoParams videoParams) { cleanupVideoPlayback(videoView, videoParams); }... }Copy the code

CoreVideoComponent as item added to the AutoplayVideoComponentLayoutSpec, the LayoutSpec registered to automatically play video in news headlines. All videos in the news Feed are registered with the autoplay manager, but not all videos in the application require autoplay (for example, videos in a full-screen video player).

@LayoutSpec public class AutoplayVideoComponentSpec { @OnCreateLayout static Component onCreateLayout( ComponentContext c, @Prop VideoParams videoParams) { registerVideoForAutoplay(videoParams); return CoreVideoComponent.create(c) .videoParams(videoParams) .build(); }... }Copy the code

Finally, we will automatically play video added to VideoAttachmentComponent components as a child. This component translates the video attachment data structure into properties understood by more general video components.

@LayoutSpec
public class VideoAttachmentComponentSpec {

  @OnCreateLayout
  static Component onCreateLayout(
      ComponentContext c,
      @Prop VideoAttachmentInfo videoAttachmentInfo) {
    final VideoParams videoParams = createVideoParams(videoAttachmentInfo);

    return AutoplayVideoComponent.create(c)
        .videoParams(videoParams)
        .build();
  }
  ...
}
Copy the code

This design gives us more flexibility VideoAttachmentView. Any one of these components can be added to another LayoutSpec to create a more complex component that extends its functionality and/or UI design. Litho encourages the use of nested components and combinations of components to build greater functionality. The layout tree was optimized by Litho for optimal rendering performance, creating a flatter view.

Here is an example of creating a video attachment component that displays a watermark at the bottom:

@LayoutSpec public class WatermarkVideoAttachmentComponentSpec { @OnCreateLayout static Component onCreateLayout( ComponentContext c, @Prop VideoAttachmentInfo videoAttachmentInfo) { return Column.create(c) .flexShrink(0) .alignContent(YogaAlign.FLEX_START) .child( VideoAttachmentComponent.create(c) .videoAttachmentInfo(videoAttachmentInfo)) .child( Text.create(c) .text("Powered by Litho") .textColorRes(android.R.color.holo_green_light) .textSizeDip(25) .paddingDip(YogaEdge.LEFT, 5) .positionType(YogaPositionType.ABSOLUTE) .positionDip(YogaEdge.BOTTOM, 0) .positionDip(YogaEdge.START, 0)) .build(); }}Copy the code

The new component reuses all the code and UI of the video attachment component by adding it as a child component.

Performance improvements

In addition to a more flexible design, Litho has properties and features that help optimize video playback in the news feed and overall performance across the application.

recycling

Android’s built-in RecyclerView stores views in different pools based on their type, which can be problematic for uIs with many different types.

Litho’s recycling system, by contrast, reuses smaller UI building blocks, such as text or images, rather than entire views. By using a core video component, you can reclaim the same view for all video story types. More efficient recycling reduces object allocation and improves rolling performance.

pre-allocated

The first video story in the News Feed could not reclaim pre-existing video views because there were no previous views. This is also important when two video stories appear on the screen: one video view can be recycled from the previous story, but the second view needs to be created at that time. When RecyclerView needs to allocate new view objects, especially for complex view objects like video views, there is a risk of frame loss. We wanted to optimize that, so we created pre-allocation in Litho.

By adding a few attributes to the MountSpec annotation, we instructed Litho to allocate some instances ahead of time. When scrolling through the first video story in the news feed, the pre-allocated video view can significantly improve scrolling performance.

@MountSpec(poolSize = 3, canPreallocate = true)
public class CoreVideoComponentSpec {
   ...
}
Copy the code

The life cycle

MountSpec has some useful and simple lifecycle callback methods. This allows us to encapsulate most of the video playback logic in components. Before Litho, this logic was distributed in different classes and triggered by a separate controller. The main callback methods in the video component include:

• onPrepare
• – Video prefetch starts. Fires on the background thread before the video component appears on the screen.
• onMount
• – Initializes the video player. Triggered when a component configures its view for the first time.
• onUnmount
• – Clean up the video player for next use. Triggered when the video scrolls away.

LayoutSpec has one main callback: onCreateLayout(). Its primary purpose is to build LayoutSpec layouts, but it can also prepare resources for its children. For example, cover photo LayoutSpec can create a layout with a video and cover photo, while also triggering a prefetch of the cover photo, all under the same callback method.

MountSpec supports another useful callback: shouldUpdate(). When RecyclerView’s adapter is updated, it can rebind all its child views and reinstall all visible components (trigger onUnmount, then onMount). For simpler components, there is no noticeable impact, but reconfiguring the video player is a onerous operation. Calling this callback before Litho reinstalls the component gives us a chance to skip it when we don’t have to (for example, while installing the same video).

@ShouldUpdate(onMount = true) static boolean shouldUpdate(@Prop Diff<VideoParams> videoParamsDiff) { return videoParamsDiff.getNext().videoId ! = videoParamsDiff.getPrevious().videoId; }Copy the code

testability

The modularity of the new components makes it easier to test video playback logic. Litho comes with a testing framework that simulates the life cycle of components in unit tests. By encapsulating more video playback logic in these components, we can test and verify complex scenarios that were not possible before. In addition, extending functionality by using composition rather than inheritance is safer and easier to maintain.

conclusion

Litho helps improve the performance, efficiency, scalability, and maintainability of video implementations compared to video implementations built using traditional Android views. The video component improved scrolling performance in our Facebook for Android application by 20%. It also shortened our cold startup time and reduced our total out-of-memory crashes by 2.5%, all as a result of more efficient memory management. The improvements in the code not only improve the Facebook experience, but also give engineers more leeway to create new video experiences across Facebook applications.

Litho allowed us to write highly optimized UIs in Android. It opened source in April 2017 and hasn’t stopped growing since. Late last year, we rolled out Sections, an API for writing highly optimized list surfaces, built on top of Litho. We saw some significant performance improvements for Litho and Sections conversions, such as scrolling improvements of up to 42%. Getting started with Litho is easy and well-documented. Check out the Litho website and GitHub page for code samples, tutorials, and advanced guides.