This article was originally published on the wechat official account — Interesting Things in the world, handling, reprint please note the source, otherwise will be held liable for copyright. Communication QQ group: 859640274

We haven’t seen each other for a long time. We haven’t posted any articles for more than a month. I don’t know if anyone else will remember my series of articles about creating a douyin App from scratch. It’s been a while since I updated this series. The last one was written when I started developing the video editing SDK. At that time, I stepped into a new field, and there were too many unknown things, so I didn’t update relevant articles for the next half a year. But don’t think I have given up. Today is a memorable day for me. On October 28, 2019, I finally completed the simplest version of the video editing SDK, which I named WSVideoEditor. Over the next few days I plan to update four articles on parsing the SDK, and the code in WsVideoEditor will be updated along with the articles. When the SDK is parsed, the most critical step will be to copy a Douyin App series from scratch.

This paper is divided into the following sections, which can be read as needed:

  • 1. Project introduction
  • 2.SDK function description
  • 3. Introduction of SDK architecture and operating mechanism
  • 4. VideoDecodeService parsing

I. Project introduction

In this chapter, I’ll cover the basic structure, organization, and operation of the WsVideoEditor project. I need you to clone the project and follow me step by step.

1. Basic structure

Let’s look at Figure 1 and say one by one:

  • 1. Android: As the name suggests, this directory is an Android project. Ignore the.gradle, build,.idea and so on.
    • 1. Ffmpeg-cpp: As shown in Figure 2, there are ffMPEG header files and.so files in this folder, we need to integrate this library into our SDK. Our EDITING SDK needs to have the ability to decode video. As for how to get these things, I have previously written an FFMPE food guide for interested readers.
    • 2. Protobuf – CPP: This folder is similar to ffmPEG-cpp, with Protobuf For CPP header and.a file, because Protobuf is our Native way of communicating with Android/iOS/Linux. So we also need to integrate the Cpp layer Protobuf into our SDK.
    • 3. Wsvideoeditor-sdk: As shown in Figure 3, this folder is an Android Library project. Our editing SDK will exist as a separate JAR package on The Android side. There’s a lot of stuff in this directory. For example, under the SRC directory is some wrapper code for the Java layer. In the JNI directory is some Cpp code that uses the Android Native Api. More detailed analysis will follow in later chapters.
    • 4. Wsvideoeditor-test: This folder is an Android Application project for writing testsEdit the SDKThe code.
  • 2. Buildtools: As shown in Figure 4, this directory mainly contains some tool scripts. For example, build_proto.sh is currently used to generate Protobuf code for Java and Cpp layers.
  • 3. Ios, Linux: Because I giveEdit the SDKIs defined asA cross-platform video editing SDKSo the idea in the future is that iOS and Linux will be able to access oursEdit the SDKFor now, there is nothing in these two directories: -d.
  • 4. Sharedcpp: As shown in Figure 5, this directory mainly stores platform-independent Cpp code. Since we are doing a cross-platform video editing SDK, it is a wise choice to share as much platform-independent code as possible. You can see that in the prebuilt_protobuf directory there is the Cpp file that we generated with build_proto.sh, and these files can be shared.
  • 5. Sharedproto: This is where the Protobuf file we defined is stored.
  • 6. Thirdparty: This is a repository for platformer independent third-party libraries that contain source code, such as libyuv.
  • 7. Cmakelists. TXT: The main purpose of this file is to enable Clion to identify our entire project.

2. How to run the project

  • 1.git clone https://github.com/TheGodsThemselves/WsVideoEditor.git
  • 2. The NDK environment must be ready
  • 3. Open the WsVideoEditor/ Android directory with Android Studio
  • 4. Prepare the /sdcard/test.mp4 video file in your phone
  • 5. Run the wsVideoEditor-test project

Two, SDK function introduction

In this chapter, we look at the current and future features of the editing SDK. The final form of the editing SDK will be similar to the video editing function of Douyin, and readers with other ideas can also leave comments or issue in the comments section.

1. Current features

  • 1. Start playing
  • 2. Pause
  • 3. Adjust the video volume
  • 4. Play a single video
  • 5. Play multiple videos
  • 6. Video the Seek
  • 7. Blur and fill the video edges

2. Planned functions

  • 1. Video:
    • 1. Add additional sounds by timeline
    • 2. Add filters on the timeline
    • 3. Add static stickers and dynamic stickers according to the timeline
    • 4. Switch between multiple videos
  • 2. 3. How to make a living
    • 1. Add sound
    • 2. Transitions between multiple pictures
    • 3. Photo movies
  • 3. Tools:
    • 1. Capture a video thumbnail
    • 2. Read the video metadata
  • 4. Code class:
    • 1. Export videos in different formats
    • 2. Change the video resolution and frame rate
    • 3. Transfer video to GIF
  • 5. Technology:
    • 1. Multi-process video codec
    • 2. Play videos in multiple processes
    • 3. Multi-process video thumbnail capture

Iii. Introduction of SDK architecture and operating mechanism

In this chapter, I will introduce the overall architecture and operation mechanism of the current editing SDK.

1. Edit the SDK architecture

Figure 6 is the architecture diagram for the editing SDK, which I will follow in this section.

(1). Basic API

Start at the bottom, which is the underlying API library that the entire SDK depends on.

  • 1.FFMPEG: As briefly described above, it is an open source video library, which is mainly used for software codec in our project.
  • 2.MediaCodec: Is a hardcodec API in Android, and iOS has its own hardcodec.
  • 3.OpenGL: OpenGL ES is an open source graphics library that is built into both Android and iOS as the default graphics library. In our project, it is mainly used to draw the decoded video frame to the screen. It is also possible to make changes to these images, such as filters, video/image transitions, etc.
  • 4.Libyuv: A Google open source library for conversion, rotation, and scaling between YUV and RGB.
  • Protobuf: Google’s open source, platform-independent, language-independent, extensible, lightweight and efficient protocol for serializing data structures. In our project, it is mainly used for data communication between Cpp and Java, OC and Dart.

(2). The SDK

Then let’s look at the main part of the picture. Because there is only the implementation of Android, I use Android to replace the upper implementation of the main part.

  • 1.Android layer architecture:
    • 1.WSMediaPlayerView: Inherits from TextureView, so it can provide a thread with an Open GL environment. For those who are not familiar with the Surface family, check out these two articles: Android Drawing Mechanism and Surface family source code Full analysis, Camera /OpenGL/ Video /Flutter and SurfaceView
    • 2.WSMediaPlayer: This is a Java class that represents Native NativeWSMediaPlayer. This class has various apis that a player should have, such as play, pause, seek, and so on. In fact, many Android system classes exist in this form, such as Bitmap, Surface, Canvas, and so on. In the final analysis, Java is only the upper level language of Android system to facilitate developers to develop apps. Most of the functions in the system will eventually go to Native, so readers need to get used to this kind of code logic
    • 3.AudioPlayer: this class is a Java class that plays audio frames based on the AudioTrack in Android. We also have an AudioPlayer in the Native layer. In contrast to WSMediaPlayer the Native AudioPlayer is a shell, the Java AudioPlayer is a reverse proxy for the Native AudioPlayer, Because in this case the AudioPlayer in the Java layer is the one that actually plays the audio.
  • 2.Native layer architectureHere we analyze the architecture of the Native layer from the bottom up
    • 1.AudioDecodeService: It uses FFMPEG/MediaCodec to decode audio frames from video/audio at a point in time and store them in an audio frame queue. Finally, the audio frame is taken out and handed to the audio player.
    • 2.VideoDecodeService: Similar to AudioDecodeService, it uses FFMPEG/MediaCodec to decode video frames from video at a point in time and store them in a video frame queue. Finally, the video frame is taken from the outside and given to OpenGL to be drawn on the screen.
    • VideoFramePool: It is responsible for responding to the external seek event, and then uses FFMPEG/MediaCodec to decode the current time frame from the video, and then stores it in a LruCache and returns the seek time frame.
    • 4.AudioPlayer: As mentioned earlier, this is a Java layer AudioPlayer proxy class that plays audio frames decoded by the AudioDecodeService.
    • 5. FrameRenderer: This is a renderer that is used to render the video frames decoded by VideoDecodeService while the video is playing, to send a seek request to VideoDecoderPool during the video seek, and then render the returned video frames.
    • 6.NativeWSMediaPlayer: Used to synchronize audio and video playback between AudioPlayer and FrameRenderer. What we think of as a video player entity is mediated by the Java layer WSMediaPlayer.

2. Edit the RUNNING mechanism of the SDK

The last section covered the architecture of the editing SDK, and this section will cover how the editing SDK works based on Figure 7.

  • 1. From the previous section, we all know that WSMediaPlayerView is the top class of the entire editing SDK. So let’s start with WSMediaPlayerView, and look at the top of the picture.
    • 1. Can see WSMediaPlayerView will maintain a 30 ms timing cycle, the cycle will continue to call the draw in frame to drive WSMediaPlayer/NativeWSMediaPlayer video/audio playback.
    • 2. At the same time, the user on the left will update the status of NativeWSMediaPlayer through apis such as Play, Pause, seek, etc.
    • 3. Note that the WSMediaPlayerView timing loop is not interrupted by the user’s play, pause, seek, etc.
  • 2. Take a look at the image on the left. This is the internal playback mechanism of WSMediaPlayer. The point isThree loops, two playAgain, let’s do it from the bottom up.
    • 1. VideoDecodeService: It maintains an internal blocking loop with a first-in, first-out queue — BlockingQueue. When we start playing a video or seek a video to a certain point in time, VideoDecodeService records the start time and then decodes every frame after the current point in time. Every time you decode a frame, you put that frame in BlockingQueue. When the number of elements in the queue reaches its maximum, the current loop will be blocked until the external BlockingQueue consumes the Top frame, and the loop will be started again. Note that VideoDecodeService only provides video frames when the video is playing, because in this case the order of video frames in BlockingQueue is the order in which the video is actually playing.
    • VideoFramePool: It internally maintains a blocking request loop with a LruCachePool. Normally VideoFramePool’s loop is blocked. When the external seek video is played, the loop receives a request and starts processing the request. If any Cache in the LruCachePool is hit, it returns to the Cache directly. Otherwise, the video frame at the time point in the request is immediately decoded from the video and stored in the LruCachePool and then returned. Note that VideoFramePool only provides video frames when the video is in seek. Since our seek operation is random, VideoDecodeService cannot be used in this case.
    • 3.AudioDecodeService: Similar to VideoDecodeService, it also maintains a blocking loop and fifO queue. Other internal behavior is similar except that video frames are replaced by audio frames.
    • 4.FrameRenderer:
      • 1. When the video seeks, it draws the video frame at the moment of seek from the VideoFramePool.
      • 2. When the video is playing, its drawFrame method is constantly called by WSMediaPlayerView through a timed loop and draws it from the current frame in The VideoDecodeService through Open GL.
    • 5. AudioPlayer: While the video is playing, it is also constantly driven by WSMediaPlayerView through a timed loop to fetch the current audio frame from the AudioDecodeService, The audio frame is then handed over to the AudioPlayer in the Java layer via a reverse proxy for playback.

Four, VideoDecodeService analysis

The last chapter talked about the overall architecture and operation mechanism of the entire editing SDK, but in fact, the details of every part of the internal editing SDK are very much, so this chapter I will first explain the internal details of VideoDecodeService. The other sections are covered in subsequent articles. In the meantime, the code in WsVideoEditor is constantly updated as the presentation goes on. The result is a usable editing SDK.

1. The API

-- -- -- -- -- code block1----- VideoDecodeService.java
  private native long newNative(int bufferCapacity);
  
  private native void releaseNative(long nativeAddress);
  
  private native void setProjectNative(long nativeAddress, double startTime, byte[] projectData);
  
  private native void startNative(long nativeAddress);
  
  private native String getRenderFrameNative(long nativeAddress, double renderTime);
  
  private native void updateProjectNative(long nativeAddress, byte[] projectData);
  
  private native void seekNative(long nativeAddress, double seekTime);
  
  private native void stopNative(long nativeAddress);
  
  private native boolean endedNative(long nativeAddress);
  
  private native boolean stoppedNative(long nativeAddress);
  
  private native int getBufferedFrameCountNative(long nativeAddress);
Copy the code

As shown in block 1, let’s start with the VideoDecodeService API

  • 1.newNativeAs we know from the previous chapters, VideoDecoderService has one insideFirst-in, first-out blocking queues, the input parameter to this methodbufferCapacityThat’s what it’s used to set upBlocking queueThe length of the. After this method is called, the Native layer creates a Videodecodeservice.cpp object with the same name as the Java layer. And then return alongRepresents the address of this Cpp object. We’ll record it in the Java layer, and we’ll need to use this address to find the corresponding object when we call other methods.
  • 2.releaseNative: Because Cpp does notThe garbage collectionThis method is used to release the Videodecodeservice.cpp object.
  • 3.setProjectNativeProtobuf is an efficient cross-platform communication protocol, so the communication between Java and Cpp layer is Protobuf, as we can seews_video_editor_sdk.protoThis file, in which EditorProject is defined, is the data structure that both sides use together. The input parameter to this methodnativeAddressThat’s the address of the object that we got in 1. The ginsengstartTimeRepresents the starting decoding point, in seconds. The ginsengprojectDataIs the byte stream after the Serialization of EditorProject.
  • 4.startNative: This method indicates the start of decoding.
  • 5.getRenderFrameNative: This method represents gettingrenderTimeThe frame data for this moment is currently returned to the Java layer by aStringIn the Cpp layer, what we’re going to do is basically use this method to get the frame data and then use OpenGL to draw it to the screen.
  • 6.updateProjectNative: This method andsetProjectNativeSimilarly, for updating EditorProject.
  • 7.seekNativeWhen we are watching a video, the action of dragging the progress bar to a certain point in time is called seek, which is reflected in VideoDecodeService. This method sets the current decoding point toseekTime.
  • 8.stopNative: This method means to pause decoding.
  • 9.endedNative: returns abooleanIndicates whether the decoding point of the video has reached the end of the video.
  • 10.stoppedNative: Returns onebooleanIndicates whether decoding is currently suspended.
  • 11.getBufferedFrameCountNative: returns aint, which indicates the currentBlocking queueNo more than we set in 1bufferCapacity.

2. Code analysis

In this section, I use a complete example to analyze the VideoDecodeService source code

  • In TestActivity, we run the project and see three buttons and two TextViews on the interface.
  • 2. We are ininitButtonThe following operations are performed in
    • 1. The UI is initialized.
    • 2. Create a VideoDecodeservice. Java class that internally calls what we talked about in the last sectionnewNativeMethods. This method will eventually go intovideo_decode_service.hCall the Videodecodeservice.cpp constructor, which creates a BlockingQueue.cpp objectdecoded_unit_queue_That’s what we’ve been talking aboutFirst-in, first-out blocking queues
    • 3. Build an EditorProject.java that passes a path to the video to decode/sdcard/test.mp4
  • 3. Let’s click the START button
    • 1.stringBuilder 和 timesIt’s used to record test data, not to mention
    • 2. Here is then calledsetProjectMethod, through a chain of calls through JNI into block 3
      • 1. ThebufferDeserialize to an EditorProject.cpp object.
      • 2.addressStrongly convert the Videodecodeservice. CPP object.
      • Use 3.LoadProjectMethod to parse some data, such as the video frame rate, width and height, etc. Interested readers can follow to have a look.
      • 4. CallSetProjectSet up EditorProject.cpp for VideoDecodeservice.cpp.
    • 3. CallstartIt ends up in block 3, callingStartMethods. We continue intoStartMethod, which is found to start a thread and then calledVideoDecodeService::DecodeThreadMain, this method has awhileLoop, every time a video frame is decoded using FFMPEG, it will be placed indecoded_unit_queue_In the. When there are no external consumers,decoded_unit_queue_The number of frames will soon reach the threshold (we set it to 10) and the thread will be blocked. Until the number of frames is reduced after external consumption, the thread will continue to decode video frames, and so on.
-- -- -- -- -- code block3----- com_whensunset_wsvideoeditorsdk_inner_VideoDecoderService.cc
JNIEXPORT void JNICALL
Java_com_whensunset_wsvideoeditorsdk_inner_VideoDecodeService_setProjectNative
        (JNIEnv *env, jobject, jlong address, jdouble render_pos, jbyteArray buffer) {
    VideoDecodeService *native_decode_service = reinterpret_cast<VideoDecodeService *>(address);
    model::EditorProject project;
    jbyte *buffer_elements = env->GetByteArrayElements(buffer, 0);
    project.ParseFromArray(buffer_elements, env->GetArrayLength(buffer));
    env->ReleaseByteArrayElements(buffer, buffer_elements, 0);
    LoadProject(&project);
    native_decode_service->SetProject(project, render_pos);
}

JNIEXPORT void JNICALL Java_com_whensunset_wsvideoeditorsdk_inner_VideoDecodeService_startNative
        (JNIEnv *, jobject, jlong address) {
    VideoDecodeService *native_decode_service = reinterpret_cast<VideoDecodeService *>(address);
    native_decode_service->Start();
}
Copy the code
  • 4. Moving on to block 2, you can see that I started an infinite loop thread in the Java layer, sleeping every 30ms. I’m going to call it every time I loopgetRenderFrameMethod to consume a video frame from VideoDecodeService. Then print the frame information to TextView. In fact, the code here can be likened to the video playback, VideoDecodeService is constantly decoding in the background thread to put the video frames in the queue in order, this thread will continue to take a frame from the queue to consume, as the video frame is rendered to the screen.
  • 5. At the bottom there is an infinite loop thread of Java layer, which will constantly read other information of VideoDecodeService and print it to TextView.
  • 6. This section is just a simple introduction to the VideoDecodeService running ideas, in fact, there are many implementation details in the code, these details can only be given to the reader, after all, I am relatively lazy 😀

Five, the tail

Finally from zero development copy write a douyin APP this series of articles began to update again, since this year the article published a lot longer interval, writing a lot less time. But I can’t just give up updating for the sake of all the readers who support and follow me. Set up a flag, in the future every month to update an article, I hope you can be a lot of support, thanks!!

Serial articles

  • 1. Copy a Douyin app from scratch — Start
  • 4. Copy a Douyin App from scratch — log and burial points and the initial architecture of the back end
  • 5. Copy a Douyin App from scratch — App architecture update and network layer customization
  • 6. Copy a Douyin App from scratch — start with audio and video
  • 7. Copy a Douyin App from scratch — a minimalist video player based on FFmpeg
  • 8. Copy a Douyin App from scratch — build a cross-platform video editing SDK project
  • 9. Copy a Douyin App from scratch — Android drawing mechanism and Surface family source code full analysis)

No anxiety peddling, no headlines. Share something interesting about the world. Topics include but are not limited to: science fiction, science, technology, Internet, Programmer, computer programming. Below is my wechat public number: the world’s interesting things, do a lot of goods waiting for you to see.