Cloud Clip is an online video production platform integrating video production, production and distribution, equipped with tools and services: In terms of tools, Cloud Clip has comprehensive video editing capabilities, which are more powerful than other online editing tools in the industry. The range of support includes multi-track audio and video synthesis, audio and video material cutting, cutting, dragging and dropping, patch word setting and audio editing, etc. In terms of services, the cloud clipping platform supports various services such as business invocation, video production and distribution, and video production status query.
Figure 1 IQiyi Cloud clip editing preview page
With the continuous development of Web technology, especially the release and implementation of HTML5 standard, the front-end can not only achieve friendly interface display and interaction, but also realize the operation and rendering of streaming media data. With the continuous enhancement of Web capabilities, Web applications have more and more complex functions and higher performance requirements. When using desktop software to edit audio and video data, it not only has high requirements on machine performance, but also consumes a lot of transcoding time. Iqiyi Cloud Clip realizes online audio and video production through Web technology optimization, and cloud service realizes transcoding, which effectively reduces PC performance requirements of users and improves transcoding performance and speed.
The main functions of cloud clipping are:
1. Video cutting, splicing and other basic functions
2. Basic interaction of video track zooming and dragging
3. Basic packaging ability of video stickers, stickers, etc
4. Real-time preview of video effects
5. Online transcoding ability of video
The previous four points need to be completed on the browser side. There are many technical difficulties in implementing large-scale desktop editing software on the browser side based on Web technology:
First of all, how to simulate playback in Web, wySIWYG is what you see;
Secondly, when complex operations are performed on the Web, browser resources are consumed greatly, so how to ensure good interactive experience and performance?
Moreover, in addition to cutting and splicing, video production, such as effect overlay of pasting words, maps and filters, also needs to be simulated by the browser, and it needs to be consistent with the server, which is also a technical difficulty.
Let’s take a look at how iQiyi technology team realizes online audio and video editing in cloud clip through Web technology.
1
Challenges in cloud clipping architecture design
Cloud clipping migrates PC desktop application functionality to the cloud. Programs that used to run on one computer are now done by multiple computers. Calculations that were previously performed by different modules of a program are now coordinated across different services connected to the network. This brings great challenges to the architectural design of cloud clipping.
As shown in Figure 2, the overall architecture of IQiyi Cloud Clip is divided into four parts:
Web client: responsible for specific interactive operations with users, including material import, video clips and real-time preview of effects. See Figure 3 for specific functions.
Node middle layer: the main functions are receiving, forwarding, and saving project and video data.
Creation center: provide cloud clip with video production, transcoding, distribution and other business processing and scheduling.
Cloud platform: provides basic support, such as audio and video transcoding, storage and distribution, etc.
Figure 3. Architecture diagram of Web products
2
Technical principle of real-time preview of cloud clip
Real-time preview of video clip effect becomes a technical difficulty of cloud clip tool. Iqiyi cloud clip player creatively realizes the real-time preview of multiple videos after cutting, cutting and splicing by combining relay mode of multiple player instances. The principle of clip preview player is shown in Figure 4:
Figure 4 schematic diagram of clip preview player
Multiple video player instances are combined into one video player, which uses Canvas projection to integrate video images. To solve the following problems:
2.1. Multiple groups of video players and the organization of playing order
* * * *
Put the video player instances into different video groups. Video group is a higher virtual form of video integration. In each video group, a different number of video instances are added. The video groups have different divisions of labor. Some groups are responsible for preloading, some for projection, and some for conversion. Video groups complement each other, relay each other, and restrict each other. In this way, the problem of smooth playback of video stitching will be effectively solved.
2.2. How to ensure that the resources to be played are properly prepared during video switching and jumping
When the video jump (seek) play, can not predict the user to play which video which segment, it is not to mention the preparation of resources in advance. But we’ve done a lot of real-world testing and found that creating a video team dedicated to caching historical data effectively increases the chance of hitting a preloaded video. Therefore, the temporarily unused resources in other video groups are aggregated to the cache video group. The cache video group can reuse the cache data to the maximum extent by virtue of its own advantages of flexible intervention, so as to solve the lag phenomenon when jumping to play to the maximum extent. The specific process is shown in Figure 5:
Multiple video instances are combined to play, and the video format has better compatibility and stability. Group playback resources to effectively improve resource utilization and reduce memory usage.
To solve the problem of video preview, we also need to add a variety of packaging customization and special effects, because the preview player picture of cloud clip is projected on the canvas element, which leaves a lot of room for our video packaging. The video packaging process is shown in Figure 6.
Based on the combination relay principle of the player and the ability of packaging customization, the cloud clip player realizes the real-time preview function of video clips, and the effect is consistent with the cloud transcoding effect. At the same time, from the perspective of high performance and maintainability, the architecture of the player is designed in layers according to the play control, core and packaging. Figure 7 shows the overall architecture of the clip Preview player.
Figure 7 Clipping player architecture diagram
3
How is the video clip track implemented
Video production itself is a strong interaction between users and tools. Cloud clip is an online video production tool. Compared with desktop programs, its disadvantages are mainly in performance and interaction. Among them, video clip track is the difficulty of web-based video production tools. In addition to implementing complex, customizable video production functions, it is necessary to ensure smooth operation. We solve these problems through various methods such as functional stratification and structural optimization. Figure 8 is the overall functional architecture diagram of clip track:
The requirement of clip track varies greatly among different video clip tasks. Some only need video track, some need video and audio track, and some need simple packaging or texturing track. Based on the diverse needs of video clips, we needed to make the track parts into customizable components. Many components and complex interactions make it necessary for the code structure to be reasonable, so we have sorted out the development mode and structure of components. Figure 9 shows the code structure after the clipping track is simplified:
Figure 9 code structure diagram
The clarity of the structure ensures the scalability and maintainability of large and complex programs, as well as the satisfaction of diversified business requirements.
4
Cloud clip middle layer:
Node is the bridge between the front end and the back end
We will solve the problems of interface integration, temporary data storage, and data consistency between front and back end interactions through the Node middle tier. As for the back end, it only needs to provide raw data without paying too much attention to the requirements of the front end on the interface data format, and focuses more on its own business. For the front end, the acquisition of customized interface data can also reduce the number of front-end requests. The architecture of the cloud clipping middle layer is shown in Figure 10 below:
Nginx forwards requests that do not need to be processed by the middle tier and provides static resource services. Node handles the business logic and is responsible for the interaction with the authoring console. The middle layer of cloud clipping adopts self-developed QiYI-Wings framework, and divides router, Controller, Service, Model, DAO and other layers on the basis of Koa2. Service modules are separated from each other and divided into different levels. The code architecture is shown in Figure 11 below:
Figure 11. Mid-tier code architecture diagram
5
Overall architecture of the Web
We solve the code maintenance and extensibility problems caused by the huge amount of front-end code through Web system and code architecture design.
5.1. Web side system architecture
In the cloud clipping tool, users need to perform a series of operations such as clipping, splicing, adjusting the entry and exit points, and changing the double speed of the material on the track, and finally preview the finished film in the player. During the entire operation, the platform periodically saves the video intermediate to the server database. When the page is refreshed, you can obtain the intermediate state recovery video from the server and continue editing.
Figure 12. Data flow chart
As shown in Figure 12, the cloud clipping tool provides a large amount of audio and video materials and a variety of text templates.
Cloud editing tools to track data as the center, once the orbital data change (material drag and drop to the rail, rail operations within the material, configuration area change material), will trigger the data distribution process, track their data distribution to the players, special packing layer, material configuration area, the module will be distributed through data processing, into their own need to format. The track data-centric mechanism enables tracks to support global undo redo, not just orbital modifications.
5.2. Web front-end code architecture
Cloud clipping front-end engineering is built based on Vue+QUI2.0, QUI2.0 is the component library incubated by iQiyi front-end team. It is a Web-based development suite based on Vue, aiming to provide a unified front-end solution for enterprise application development, including basic components, engineering framework and tool commands. Front-end engineering is a single page application with multiple modules. The code structure under the SRC directory is shown in Figure 13 below.
FIG. 13 Front-end code structure diagram
The editor module is divided into six parts according to the function, including project configuration area, material area, track area, player area, video, decorator special effects area, and video publishing area forms. Each part has a corresponding store to store data to achieve unified management and sharing of data.
The flow of front-end engineering requesting server data is shown in Figure 14 below.
FIG. 14 Front-end data flow chart
6
conclusion
* * * *
Through structured code layering, strict flow control, fine interaction design, the use of Web technology to achieve complex audio and video editing. With the continuous development and improvement of Web technology, its performance and interaction will be more and more close to or even beyond the desktop program. The Web architecture of cloud clipping will also make continuous progress with The Times. In the future, We will introduce Web Assembly to realize decoding and computation, and AI technology to enable video. At present, iQiyi Cloud clip is only available to iQiyi’s internal production team. In the future, it will be open to the outside world to provide efficient and convenient creation ability for external studios, small teams and individual producers, so that everyone can enjoy the convenience of Web+ cloud service +AI technology, effectively reduce the threshold of video creation and improve production efficiency.
Maybe you’d like to see more
What kind of video can iQiyi AI editor cut?
Let AI “read” short video, iQiyi content tag technology analysis
Scan the qr code below, more exciting content to accompany you!
\
