Mona Lisa raps, Sushi reads poetry, and Gollum sings Black Pink’s Ice Cream. Yes, you read that right. This AI algorithm will allow you to do the super Fancy effects in 5 minutes, and make your own super videos that have been viewed millions of times and top trending.
No more words, let’s see the effect first!
In addition, it is also the AI algorithm that enables digital people to speak, which is the core technology of digital people, such as the recently popular virtual beauty blogger Liu Yexi, the first AI sign language anchor of CCTV headquarters released by Baidu last week, and various virtual idols.
So what is this amazing technology?
The answer is the Wav2lip model, the latest darling of PaddleGAN
The project also provides support for popular GAN models such as AnimeGANv2, GauGAN, and First Order Motion.
Github.com/PaddlePaddl…
What’s more, the complete project code has been released to AI Studio, and you can experience it with your fingers:
Aistudio.baidu.com/aistudio/pr…
You can read poems by Su Shi, sing a Rap by Mona Lisa or say love words by any Idol you like.
This is a task of inputting a voice, making the lip shape of the target person’s picture and video automatically match and move according to the voice. Therefore, we need to prepare an audio and a portrait/animation character video, the audio and video input Wav2lip model, after Wav2lip model prediction, will output a target character/animation character say input audio video, so far, “ten million” dubbing video is completed ~
PaddleGAN’s lip transfer ability — Wav2lip
The key of Wav2lip model to achieve accurate synchronization breakthrough of lip shape and speech is:
- A lip sync discriminator is used to force the generator to continuously produce accurate and realistic lip movements.
- In addition, visual quality was improved by using multiple consecutive frames instead of a single frame in the discriminator, and by using visual quality loss (rather than just contrast loss) to account for temporal correlation.
The universal Wav2lip model is suitable for any face, any voice, any language! It can achieve high accuracy for any video, can achieve seamless fusion with the original video, both video effect and voice effect are very realistic.
At the same time, PaddleGAN optimized the Wav2Lip model in HD, making the lip fitting more delicate and realistic.
PaddleGAN Wav2lip usage method
With PaddleGAN’s help, this magical homemade “dub/lip sync” is just two steps away:
1. Download PaddleGAN and install the required package
‘# Download PaddlePaddle installation package
Clone PaddleGAN from Github (gitee source if download speed is too slow)
! git clone
Gitee.com/PaddlePaddl…
#! git clone
Github.com/PaddlePaddl…
# Install PaddleGAN locally
%cd /home/aistudio/PaddleGAN
! pip install -v -e .
! pip install -r requirements.txt
! pip install librosa! PIP install numba = = 0.53.1 `
2. Use the lip compositing command
%cd applications/
! python tools/wav2lip.py \
–face /home/aistudio/1.jpeg \
–audio /home/aistudio/2.m4a \
–outfile /home/aistudio/pp_put.mp4 \
–face_enhancement
Just change the face parameter and audio parameter in the following command to their own video and audio path respectively, and then run to generate and audio synchronization video, after running, will be generated in the current folder named outfile parameter specified video file, the file is the audio synchronization video file: \
- Lface: The original video, the lips of the characters in the video will be synthesized according to the audio
- Laudio: Audio that drives the lip-synth, from which the character in the video will be lip-synth
- Loutfile: name of the finished video
- Lface_enhancement: Adds a face to increase the effect
PaddleGAN
Think that’s where PaddleGAN ends? NoNoNo~
PaddleGAN, a free open source trove kit, certainly has more than lip transfer/generation techniques. It’s full of interesting image/video generation and processing capabilities. Hot cutting edge models such as AnimeGANv2, GauGAN, First Order Motion and others are waiting to be explored.
Such as image style migration, video repair, image super resolution, portrait animation, photo animation, face editing and so on.
PaddleGAN is a “playground” and you are welcome to play with all kinds of “game facilities”. No ticket is required. If you are having a good time, click on the Star to support you
Github.com/PaddlePaddl… 
