With the advent of the era of intelligent video production, professional video production will become smarter and easier. Intelligent video production mainly contains three modules, they are: video SDK underlying ability, AI ability and mass video preset export ability. In this LiveVideoStackCon 2021 Beijing station, Li Zhiqiang, CEO of Hangzhou Vision Technology Co., LTD., shared with us how to develop intelligent video preset tools to assist designers to intelligently export preset content and some successful cases.
Article | li chi keung,
Finishing | LiveVideoStack
Hello everyone, I am Li Zhiqiang (Jiezi), the founder and CEO of Hangzhou Shixing Technology Co., LTD. Shixing Technology is mainly concerned with the export of professional video data. Why I want to emphasize “professional”, this is in the whole process of my sharing you will gradually feel.
Let’s take a look at the topic we share today — the development of professional video data export tools in intelligent video content production scenarios. The concept of an intelligent video content production scenario is something you’ve probably already been exposed to. We try to solve some problems of perceptual design of designers, mainly for some scenes with a large demand for video content, high frequency and high production efficiency, which requires the use of technology to realize content production intelligently. The reason why we need to export some data in this kind of production is that at present, our solution, especially for professional video production, cannot exclude the role of designer, because professional video production itself still depends on the perceptual design ability of professional designers. What we can do is digitize these perceptual design capabilities, turn them into our code, and turn them into assets that can be reused. Today I would like to share with you our company in this aspect of the problem, how to do the operation.
1. Enter the current situation of video content production technology
1.1 Application Scenarios
1.1.1 Automated advertising
First, let’s take a look at the current situation. Here are a few application scenarios. First of all, a relatively large application scenario is automated information flow advertising. The emergence of this demand scenario is just like the hot discussion of “inner volume”, which is based on a new demand generated in this state.
Especially in the field of out some of our customers on time to the advertising, such as the face of the customer is likely to be some southeast Asian countries, each country has its own language, if we use the traditional low frequency times before advertising, may be updated once a week or a month AD content, so the use of artificial means to create video advertising is no problem, But now, when we need to increase users’ interest and attention to these ads, we have to provide customized video content for each user’s personal user profile. At this point, automated video AD content production has become an industry demand, and we mainly work with Facebook’s AD agency in China in this regard. Many of China’s overseas APP and foreign trade practitioners need to promote the information flow of overseas communities through agents. We mainly help to solve the problem of the intelligent information flow video AD production.
1.1.2 Template-based video making website APP
I believe that all of you are familiar with some template video production, which is also a professional video content generation method widely used at present. It is also a capability that many of our mobile terminal products have at present, and it is also to let ordinary users who are not designers, The ability to easily generate a professional video with one or two clicks. In this respect, there are more mobile clients, here is not a list.
One solution we chose was to incorporate the design capabilities of professional designers. The training of professional designers’ creative ability and proficiency in using a tool itself is costly and time-consuming. To be designer-friendly, we chose to design various complex templates that are directly compatible with the toolchain AE that designers are currently familiar with.
1.1.3 Live mixed cutting
We mainly cooperate with Dharma Academy for some live-streaming mixed cutting work. For livestreaming by e-commerce companies, one piece of livestreaming material may take 5 or 6 hours, and these livestreaming materials often need secondary processing. If I remove some highlights from the livestreaming and then make secondary editing, I will generate some short videos and put them on other short video platforms. It can be seen from here that the editing work has become labor-intensive, and manual participation has become a matter of low ROI. Therefore, we are also making use of our ability in video editing to achieve the editing and packaging work of live video.
1.1.4 Production of original materials in scenic spots
Scenic spot to visitors of the rich experience, or as some of the new energy vehicles according to the collected on-board camera video material to produce smart clips, they are the same type of demand, let our life experience to upgrade, using intelligent technology for our life something that need to be manual operation to provide a kind of intelligent solutions.
1.2 Difficulties in production of intelligent video technology
With a number of requirements scenarios in place, what are the problems we are facing now? Let’s share it with you.
The first difficulty is AI capability, which is the first step in intelligent production. We need to understand the user’s material, understanding is mainly the understanding of semantic, multi-modal identification and other technical means to analyze the user’s material. Some of the LVS partners have done an excellent job in this area, and we are mainly focused on the latter two difficulties.
The second difficulty is theme template, which mainly refers to the whole process of a relatively complete professional design itself: from the designer to design, form the idea and turn it into a visual work, and finally turn the work into an editable template for THE C-end users.
The third difficulty is video presetting, which mainly refers to that some video templates obtained in the process of editing are not complete. We need to disassemble the designer’s ability in a modular way, and then provide these modular abilities to the small white users with certain creative ability, so that they can make secondary use of them.
1.3 Current Status
We believe that there are still two major problems, and I would like to discuss them with you here.
First of all, the first problem is the poor effect, we are discussing an effect or a template itself contains what effect, especially from the technical side to discuss this problem, such as now more popular “out-of-body”, these words become some industry terminology. However, from the perspective of designers, they will not describe an effect itself in this way, because they will use a tool they are familiar with, such as AE, which will disassemble it into animation of layers, add some special effects on layers themselves, or layer styles, or interact with other layers to produce some track mattes, etc. Use a combination of tools to create a variety of effects, rather than being limited to how a particular effect is achieved. If we don’t support the designer’s workflow or his ability, there will be a problem that the designer’s creativity can’t be put to use.
I don’t know if any of you in the company have encountered this situation: one day your designer is talking to the technical staff and says that there is such an effect of the technology in the brain, and the technical staff hears the designer’s needs and thinks this idea is difficult to implement. In fact, from the designer’s point of view, this idea is very simple, because he is already familiar with the tool set of AE, he can quickly create the effects, but for these effects if there is no corresponding technical solution, often become very difficult.
The second problem is actually caused by the first problem. If we don’t give designers a tool to weigh, it becomes very difficult for designers to design and translate data. At this point the technician is forced to step in again: take a bunch of raw data generated by the designer, rework some code, and turn it into a template that the SDK can work with.
Our determination is to solve these two problems once and for all.
2. Preset export tools for the era of intelligent video production
2.1 Solution Introduction
We offer designers a design tool called VE Exporter, which is a tool for designers. Designers can use IT to generate data templates and design templates, which can be combined with our VE SDK to form various functional experiences. For example, template video, photo albums, dynamic models, camera templates, sticker filters, etc. And all the functions come from the designer’s ideas.
I’m a designer by background, so I’m obsessed with the use of designer skills in our tech industry. If the creative ability of professional designers can not be brought into play, then the technical staff can only do some poor copying work, so we care about this problem very much. I have a video here that I’ll show you.
After a designer has freely created a game using the AE design tool, VE Exporter can be launched, and VE Exporter can load its effects with a single click. It can set some default background music. When we start with the requirements scenario, we may also encounter a problem: how can the user modify the template once we give it to them? VE Exporter also offers design tools that allow you to maintain a template editing interface. After completion of design, such as in the AE has 100 material, but there are 1-10 we want users to be able to change material, the rest of the material does not want the user to touch, and this time the material editor and it in the interface of which the editorial team can appear, all can be put here by dragging and compare the good design. Finally, you can export it as a template package and put it into the SDK to edit and use.
Where exactly are we now? The short answer is that we are working with AE, but really all we have to do is work hard. Since AE itself is not an open source software, there is no technical specification, so what we have to do is work backwards to figure out what the algorithm behind each parameter is, and then work our way through it to make sure that the results we get are exactly the same as AE. In order to do this work, we have accumulated experience from the days of designers to our current understanding of graphics, and dug deeply, and finally got a good comprehensive experience.
We now have support for almost all after Effects layers except the light layer, such as video layer, image layer, sequence frame layer, text layer, empty layer, adjustment layer, solid layer, camera layer, etc. Key animations like layer animations, camera animations, mask shape animations, character animations, and Layer Style animations are fully supported. One of the reasons why AE is widely liked by everyone is that it has a rich special effects system. In order to allow designers to use AE freely, we have also spent a lot of hard work on the secondary research and development of special effects in AE. If we take an existing ability and make it into a new effect, it’s not really that complicated, but when it comes to keeping an existing black core effect and bringing it back perfectly, it starts to get tricky.
For example, when we implement 3D layer rendering in After Effects, interpreting the work from the perspective of game graphics and understanding the work from the perspective of designers are actually two completely different challenges. For example, we look at a water bottle, which itself is a 3D object and translucent, when you fold these water bottles into a line from front to back, there will be a number of translucent objects interspersed before and after the light rendering problem. One manifestation of this problem in AFTER Effects is how translucent 3D objects interweave with each other and apply various layer blending modes to get the correct result. This problem is usually found in game engines, where people sort objects by their specific camera position, rendering the farthest objects first and the nearest ones later, and then using blending mode. But this doesn’t work in AE, because the order of the layers in AE is determined by the designer and we can’t change the order, so in order to complete the entire rendering task, we have to study how to implement this kind of out-of-order 3D transparency algorithm. At present, I think we have solved this problem well. If you are interested, you can also download VE Exporter combined with AE to experience our effect.
AE has a lot of plugins, so we can’t make everything compatible in one step. The idea is this: we should first make plugins that are used a lot and are compatible with templates. When there are some special effects does not require the user to modify, we don’t need to real-time rendering it, we can even ahead of time to render it into some material to save the real time operation time of some amount of calculation, as particles, some light efficiency plugin is often of this type, then this plugin now we needn’t compatible. We are now compatible with those need to be applied to the user’s material, can not be rendered in advance of the plugin, such as color, deformation, blur, including some background processing, transition plugin.
In addition to the intervention of AE plug-ins, we also developed a large number of AE plug-ins. On the one hand, as a designer, you can download and install it directly into your OWN AE and use it normally in your daily design work. On the other hand, the effects made with these plug-ins can be perfectly restored directly into the SDK, SDK and AE functions are synchronized.
Our own plug-in development mainly has two directions. The first is that AE has some existing capabilities, but if it is used in accordance with the existing HABITS of AE, it may lead to low rendering efficiency. When we restore this effect on the mobile terminal, the product experience will be poor. So we tend to use our own set of algorithms to enhance the effect to optimize the entire rendering process.
Another direction is to add some capabilities that AE doesn’t have on its own. Take a common example, like my party at the meeting often have some golden characters, this kind of characters in our template production is very widely used, and this effect is very difficult to achieve in AE, and rendering efficiency is very low. So for this specific use scenario, we often provide some very convenient, one-click effect design tools to designers, so that he can make a very good effect in the daily template design. Now plug-ins we almost every month will send two or so, and are free, we can go to download and use.
So here’s a video for you to watch. Like the elements in the video, basically all can be modified, the user only needs to enter a new word on the keyboard, then the elements of the original text will be replaced with the same effect. In the back are some of the effects that our designers made with our plugin, you can take a look. The previous video was a mash-up of some of the effects the designer used as templates in his daily use with the plugin we published.
2.2 Advantages
VE, which WE have just introduced to you, and the technical solution we have chosen, have three major advantages.
First of all, VE Exporter’s low-level AE parsing and SDK rendering are developed by us from scratch, so we don’t have to stick to any existing technology.
The second advantage is that we are very focused on the designer experience. VE Exporter’s operation flow is perfectly aligned with AE’s operation habits, and IT can be installed directly into AE, which makes it very designer-friendly.
You should have known a technical solution, he used to be called BodyMovin, but now you should know more about his name is Lottie. This solution was written by a foreign developer friend of mine. It mainly solved a data export of UI animation, exported some UI animation in AE into JSON format. After being provided with some rendering framework by Airbnb, many manufacturers in China will use it to realize the technical solution of template rendering. In itself, it has a good user base, mainly because it is open source and free. In order to enable our customers including our designers to have a large number of materials, our SDK is also directly compatible with Lottie’s scheme, that is to say, all Lottie files on the market now can be directly put into our SDK, which can have perfect function restoration.
2.3 VE Leap
Since designers are so important to us, we must serve them well. In addition to serving them well with tools, we also provide a community of designers called VE Leap, at veleap.com.
There are three main functions of it. The first is to provide professional design tools, mainly to provide some AE plug-ins, synergistic tools to help solve all the pain points designers encounter in template design.
The second is to provide professional design tutorials.
The third is this template transaction, initially initiated from the requirements of some B-side clients. Because when the design tool itself is open and flexible enough, it will immediately generate a large amount of material demand, this time any designer in the market can become its material supplier. Because VE Exporter is good enough that a new designer can run a template in almost three hours, there are a lot of requests for template trading on our platform. At present, VE Exporter has not done any promotion, and its users are close to 10,000 designers. Many of us, who are still studying and majoring in design, can earn tens of thousands of yuan per month on the platform.
2.4 Ecology of technical solutions
VE Exporter, VE Leap, and the SDK. The SDK is not what I’m talking about today, but it’s an indispensable player. As I mentioned earlier, the main problem it solves is to provide a solution for cross-platform productization of our templates; VE Exporter’s main job is to export our design data; VE Leap is a community of designers who use our tools to communicate, learn and create. These three are mutually promote, influence each other a relation.
2.5 C-end User Solution
Let me briefly describe some of the solutions we currently offer on the C user side.
The first solution is the standard template. This is easier to understand, designers put a complete work of their own as a template for ordinary users to use. Here is a template of the designer used by our client, which shows what his editing interface looks like and what pictures are available for editing. After the designer designs this thing, it can be exported so that other users can edit the template itself on the Web and mobile terminal, as well as render output. There is also a mobile editor. Due to time constraints, I will not show it in a video. Later, I will send out the TWO-DIMENSIONAL code of this experience. If you are interested, you can directly scan the code and install the experience on the mobile phone.
Dynamic templates are mainly embodied in electronic photo albums, so what is the difference? We basically disassemble the template of a designer in a modular way. For example, when you finish a work, we help you disassemble it into different animation segments. When the user flows into any number of materials, we will disassemble the template to ensure its effect. Is to generate a professional video effect based on the amount of user material. Because of the use of electronic album, we did not know how many pictures users needed before, so we also put forward some new requirements for template creation.
The CAMERA template SDK is relatively easy to understand. It combines some capabilities of the camera. When designing the template, the designer can set one of the materials in his template as the camera input source, so that other special effects animations in his template can be retained, and at the same time, he can interact with the camera in real time. Like some basic beauty, beauty type ability is also very good, can be used with the template.
The editing SDK should be familiar to everyone, for example, it provides some tracks, transitions, special effects, etc. These elements seem to become the basic quality of some of the technical solutions we do video processing now. We’re still integrating the designer into the solution. For example, in the process of using a clip, its transitions, special effects and animation styles of characters are all derived from the designer’s modular disassembly after the design in AE and exported into preset packages for different scenarios. If you are interested, you can directly scan the QR code to download our demo for experience. You will see that there are about 200 or 300 sets of transitions, special effects, filters and presets provided in the demo, which basically took two designers about two weeks without any development involvement.
VE SDK has powerful cross-platform capabilities. In addition to mobile terminal, our desktop terminal includes servers and Web browsers, which are supported by Websambly technology. VE, which we have designed, will be moving toward the Web side as well, with a view to making our entire design experience easier and having a better interaction with assets in our community.
3. Customer cases
Next, I would like to share with you a typical customer case.
3.1 E-commerce intelligence
The first is in the field of e-commerce intelligence. We started to cooperate with The intelligent design department of Ali in May last year. One of the problems that Ali is facing now is to solve the change of their whole e-commerce advertising from picture to video. Intelligent design before a very good solution is their Luban, we open Taobao can see a print advertising, the print advertising is thousands of people, within 100 milliseconds you open, it automatically designed a picture for you. Their new task now is to start providing intelligent video generation. Since May last year, we have used our template solution to connect with their needs and helped them realize some products of smart e-commerce video production on mobile terminal and cloud.
In the aspect of AI, we have an in-depth cooperation with Dharma Academy. At present, we have cooperated with him on an intelligent editing product called “Orange Painting”. This product should be online now, you can go to experience. We mainly combined the AI recognition capability of Dharma Academy with our video processing and template capabilities to complete the one-click operation process of the whole intelligent editing.
3.2 operators
Operators are also an active group in the CURRENT 5G era, because everyone is exploring what new services operators can put forward in the 5G scene. One of the services is video RBT. In fact, the video RBT itself is just an upgrade of the audio RBT, upgrading the audio content to video, but a new requirement is where the video comes from. Before the audio era is often everyone to purchase some personality audio; Now in the era of video, manufacturers hope that users can make these video content by themselves, so the production of video content has become a new demand. At present, we maintain good cooperation with several domestic operators and provide them with technical support for video creation platform.
That’s all I have to share today. Thank you!
The lecturer to recruit
LiveVideoStackCon 2022 Shanghai station, is open to the public recruitment of lecturers, whether you are in the size of the company, title level, old bird or rookie, as long as your content is helpful to technical people, the other is secondary. You are welcome to submit your personal information and topic description via [email protected], and we will give feedback within 24 hours.