An overview of the

This article is mainly about the author recently encountered a difficult problem, from encounter, to try all kinds of schemes, step on all kinds of pits, and finally solve the heart of the journey, for reference.

above

Our product is a tool software (desktop application using NWJS). Users can submit links they are interested in, and we help collect videos inside.

A recent video on a site that a user uses a lot was watermarked very strictly (dynamic position, moving watermarks from left to right, and lots of watermarks)

Something like this:

This article comes after checking that a video link cannot be reversed to a watermarked video link (a video link is a UUID).

Author skills introduction

A front-end, thinks the programming foundation is ok, has written JAVA (CRUd Boy).

The first step

At the beginning, of course, I said to the product, no, you can’t do this, it’s not practical to watermark. However, there are pictures in the link. We combine the pictures into video by ourselves. What do you think of this demotion scheme? After a heated discussion, the product reluctantly agreed. However, I need to do some animation effects on pictures, such as animated pictures between pictures and so on. I can’t simply play a picture for a few seconds without moving.

In order to save money and rush development time (online urgent use), I first tried to see if there is a solution to directly generate video in the front end, the result is actually ~

Whammy is a library that converts your Canvas animation to WebP.

In fact, his principle is to save canvas frame by frame to WebP, and finally convert a group of WebP into webM format video)

I wrote the Canvas animation very quickly and saved it frame by frame. It was easy

The downside, though, is that the speed is too fucking slow and the video is too fucking big.

At least it worked out a bit and was deployed.

The second step

After the deployment of the plan in front of the line, I immediately began to optimize the road, after an afternoon of groping, I found ……………… There is no room for optimization of this thing, it is best to change canvas animation to WebGL rendering, rendering speed is faster.

However, the two drawbacks mentioned above have nothing to do with animation rendering speed. Converting a frame of animation to WebP is unbelievably slow, the final WebP to WebM is unbearably fast, and the most unbearable is that it eats too much CPU and memory.

So, I quickly abandoned the optimization and prepared to refactor! C++ is very fast, so I can write a C++ extension and package it into nodejs native module.

Emmm sounds feasible, but I’ve never written C++ before. (only seen C++ introduction in rookie tutorial)

It’s not practical to start from zero, so with the help of a search engine,

After looking at the C++ image-to-video solution (which took a lot of code from stackoverflow), some tutorials on nodejs native modules, and FFMPEG, it’s pretty powerful.

Some blood and tears, the final copy of the code to modify, using freeImg to open the picture, picture operation, frame by frame editing operation, and finally call FFMPEG synthesis MP4 export,

Of course, the final compilation of C++ into the NWJS client was a bit of a mistake.

To here, there is still a fatal problem, the students who know image processing to here found a problem, yes, because of my weak chicken, I did not use GPU!! The performance of this refactoring is not much better than the original solution, and the problem is still not solved.

Here, I have been quick to endure, functional performance is optimized at least some, first online.

The third step

In fact, I have been thinking about changing C++ to GPU, but I have no time. Besides, I have counted the configuration of our users’ computers, and most of them don’t have independent graphics cards…

And many users are not satisfied with the implementation of this downgrade scheme…

As early as the first step, I told the product about the possible solution of video watermarking. Deep learning marks the video watermarking position, and then uses the difference superposition of the original watermarking image to watermark. Well, it is theoretically feasible. However, our team only has a front end and a back end, how to give you a deep learning!

Under the pressure from products every day, I began to learn about some deep learning schemes. I was shocked to see what I didn’t know.

It seems that it is not as difficult as imagined, the deep learning framework has been encapsulated so completely, it seems that people who do not understand deep learning can also call the framework to train their models?

By the time I got here, I was already boasting about the product

According to a lot of tutorials and introductions on the Internet, I finally decided to choose the TensorFlow Object Detect API and use the official pre-training model. It took two or three days to download the Version of TensorFlow-GPU. It also took several days to run the official demo and open source trained model. Feasible!

The e most painful thing is that they marked dozens of pictures, it was too painful.

Company computer graphics card not quite line, I took his zhanshen notebook 1060 ran for several days, finally ！！！！

Use OpencV to call the trained data model, identify the watermark position, repair it, save it to a new video, and then use FFMPEG to merge the audio.

That’s a good solution.

subsequent

Due to product privacy issues, the code and examples are not provided

The thing that helped me the most in this whole process was Google, Github, and the ability to read technical articles in English. If you have any problems, please Google it (strongly recommend English search!!). , find a solution,

The best way to learn/use a technical framework/third-party library is to go to the official website to see the tutorial, go to Github to see the issue, can not find a look at the source code, maybe have ideas.

Don’t limit yourself to a certain technical range, dare to think and do

Due to the limitation of writing style and technical level, the writing may not be very good, if the mistake please point out.

Watch /star~~~ github.com/zuluoaaa/bl…