This is the 13th day of my participation in the August More Text Challenge. For details, see:August is more challenging
One, foreword
Make sure to see the end. Python has been great for a long time, but I didn’t expect it to be so great. Some time ago, I came into contact with a batch cutout model library, and then I found inspiration in some videos. I think we can change a different scene for the video by means of cutout, so I have today’s article.
Let’s see what we can achieve. Let’s take a look at the original scene.
Second, implementation steps
As we all know, a video is composed of frame by frame, and each frame is a picture. If we want to modify the video, we need to modify every frame in the video. So in the beginning, we need to capture every frame of the video.
After we get the frame, we need to pick out the person in the frame.
After we’ve picked out the characters, we need to read our scene image. In the example above, the background is static, so we only need to read the scene once. After reading the scene we switch scenes for each frame and write new videos.
At this point we’ve just generated a video, we need to add audio. And the audio is the audio from our original video, so we just read the audio and set the audio for the new video.
The specific steps are as follows:
- Read the video, get every frame
- Batch cutout
- Read scene images
- Scene switch for each frame
- Written to the video
- Read the audio of the original video
- Set the audio for the new video
Because the above steps are still time-consuming, I will send a notification via email after the video is finished, telling me that the video is finished.
Iii. Module installation
We need to use the following modules:
pillow
opencv
moviepy
paddlehub
Copy the codeWe can install them directly with PIP:
pip install pillow
pip install opencv-python
pip install moviepy
Copy the codeOpenCV has some adaptation problems. You are advised to select a version later than 3.0.
Before we can use PaddleHub, we need to install PaddlePaddle: see the official website for details. Reference: Stop pinching yourself, Python uses 5 lines of code for batch pinching. Here we install the CPU version directly with PIP:
# installation paddlepaddle
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
# installation paddlehub
pip install -i https://mirror.baidu.com/pypi/simple paddlehub
Copy the codeWith these preparations, we can begin the implementation of our functionality.
Iv. Concrete realization
We import the following package:
import cv2 # opencv
import mail # Custom package for sending emails
import math
import numpy as np
from PIL import Image # pillow
import paddlehub as hub
from moviepy.editor import *
Copy the codeThe name of Pillow and opencv import is not quite the same, and there is my custom mail module. In addition, we need to prepare some paths first:
The root directory of the current project. The system automatically obtains the current directory
BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "."))
# Each frame saved address
frame_path = BASE_DIR + '\\frames\\'
# The position of the picture
humanseg_path = BASE_DIR + '\\humanseg_output\\'
The path to save the final video
output_video = BASE_DIR + '\\result.mp4'
Copy the codeNext we follow the above steps to implement one by one.
(1) Read the video and obtain every frame
OpenCV provides a function to read frames. All we need to do is use the VideoCapture class to read the video, and then call the read function to read the frame. The read method returns two arguments, ret is whether there is a next frame, and frame is the Ndarray object of the current frame. The complete code is as follows:
def getFrame(video_name, save_path) :
""" read the video, save the video as a picture frame by frame, and return the resolution size and frame rate of the video FPS :param Video_name: name of the video :param save_path: path to save the video :return: FPS frame rate, size resolution """ "
# Read video
video = cv2.VideoCapture(video_name)
# Get video frame rate
fps = video.get(cv2.CAP_PROP_FPS)
# Get screen size
width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
size = (width, height)
# 🦍 Gets the number of frames used to name the image
frame_num = str(video.get(7))
name = int(math.pow(10.len(frame_num)))
# read frame, ret is whether there is a next frame, frame is the ndarray object of the current frame
ret, frame = video.read()
while ret:
cv2.imwrite(save_path + str(name) + '.jpg', frame)
ret, frame = video.read()
name += 1
video.release()
return fps, size
Copy the codeAt 🦍, I get the total number of frames, and then use the following formula to get the total number of frames larger than ten or hundreds:
frame_name = math.pow(10.len(frame_num))
Copy the codeThis is done so that the screen is sorted frame by frame so that it doesn’t mess up when reading. In addition, we get the video’s frame rate and resolution, which we need to use when creating the video. It is important to note that the following versions of Opencv3.0 have a slightly different way of getting frame rate and screen size.
(2) Batch cutout
Batch matting needs to use the model library in PaddleHub, the code is very simple, I won’t go into it here:
def getHumanseg(frames) :
Param frames: Path of the frame :return: """
# Load model library
humanseg = hub.Module(name='deeplabv3p_xception65_humanseg')
Prepare a list of files
files = [frames + i for i in os.listdir(frames)]
# cutout
humanseg.segmentation(data={'image': files})
Copy the codeHumanseg_output (humanSEG_output) humanSeg_output (humanSeg_output)
(3) Read scene pictures
This is also a simple Image reading, we use the Image object in pillow:
def readBg(bgname, size) :
Param bgName: background Image name: param size: video resolution :return: Image Object """
im = Image.open(bgname)
return im.resize(size)
Copy the codeThe object returned here is not an Ndarray object, but a class object defined in the Pillow.
(4) Switch scenes for each frame
We know that all the images are in the humanseg_output directory, which is why we need to prepare the corresponding variable to store this directory:
def setImageBg(humanseg, bg_im) :
""" param humanseg: param bg_im: background image. This is the same type returned by readBg() :return: ndarray object of the composite image. """
# Read transparent images
im = Image.open(humanseg)
# Separate color channels
r, g, b, a = im.split()
# 🦍 Copy the background in case the source background is modified
bg_im = bg_im.copy()
# Merge images
bg_im.paste(im, (0.0), mask=a)
return np.array(bg_im.convert('RGB')) [: : : : -1]
Copy the codeAt 🦍, we duplicated the background, if this step was missing, we would have generated the “thousand-hand Guanyin effect” above.
The other steps are easy to understand, except that the return value is longer. Let’s look at them in detail:
# Convert composite image to RGB, so channel A is gone
bg_im = bg_im.convert('RGB')
# Transform Image object into Ndarray object for opencV to read
im_array = np.array(bg_im)
When im_array is in RGB mode and OpenCV is in BGR mode, we convert RGB to BGR
bgr_im_array = im_array[:, :, ::-1]
Copy the codeFinally, bgr_im_array is our final return.
(5) Write video
To save space, instead of waiting to put the write image behind the merged scene, I wrote the video while merging the scene:
def writeVideo(humanseg, bg_im, fps, size) :
""" :param humanseg: PNG image path :param bgName: background image :param FPS :param size: resolution :return: """
# Write video
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter('green.mp4', fourcc, fps, size)
Set the background for each frame
files = [humanseg + i for i in os.listdir(humanseg)]
for file in files:
# Loop merge image
im_array = setImageBg(file, bg_im)
# Write video frame by frame
out.write(im_array)
out.release()
Copy the codeThe above code is also very simple, after the execution of the project will generate a green.mp4, this is a video without audio, then we need to get the audio and mixed stream.
(6) Read the audio of the original video
Because you don’t find audio processing in OpencV, moviepy is also very easy to use:
def getMusic(video_name) :
""" "get the audio of the specified video :param Video_name: video name: return: audio object """ "
# Read the video file
video = VideoFileClip(video_name)
# Return audio
return video.audio
Copy the codeThen there’s mixed flow.
(7) Set the audio for the new video
Here we use Moviepy again, passing in the video name and the audio object for mixed streaming:
def addMusic(video_name, audio) :
""" Implement mixed stream, add audio to Video_name """
# Read video
video = VideoFileClip(video_name)
Set the audio of the video
video = video.set_audio(audio)
# Save the new video file
video.write_videofile(output_video)
Copy the codeWhere output_video is the variable that we defined at the beginning.
(8) Delete transition files
When we produce a video, we produce a number of transition files, which we delete after the video is synthesized:
def deleteTransitionalFiles() :
""" "Delete transition file """
frames = [frame_path + i for i in os.listdir(frame_path)]
humansegs = [humanseg_path + i for i in os.listdir(humanseg_path)]
for frame in frames:
os.remove(frame)
for humanseg in humansegs:
os.remove(humanseg)
Copy the codeThe final step is to integrate the entire process.
(8) Integration
Let’s combine the entire process above into one function:
def changeVideoScene(video_name, bgname) :
""" :param video_name: file of the video :param bgname: background image :return: """
Read every frame in the video
fps, size = getFrame(video_name, frame_path)
# Batch cutout
getHumanseg(frame_path)
# Read background image
bg_im = readBg(bgname, size)
# Write frame 1 frame to video
writeVideo(humanseg_path, bg_im, fps, size)
# mixed flow
addMusic('green.mp4', getMusic(video_name))
Delete transition files
deleteTransitionalFiles()
Copy the code(9) Called in main
We can also put the previously defined path into:
if __name__ == '__main__':
The root directory of the current project
BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "."))
# Each frame saved address
frame_path = BASE_DIR + '\\frames\\'
# The position of the picture
humanseg_path = BASE_DIR + '\\humanseg_output\\'
The path to save the final video
output_video = BASE_DIR + '\\result.mp4'
if not os.path.exists(frame_path):
os.makedirs(frame_path)
try:
# Call a function to make a video
changeVideoScene('jljt.mp4'.'bg.jpg')
# When the creation is complete send mailbox
mail.sendMail('Your video is done.')
except Exception as e:
When an error occurs, send an error message
mail.sendMail('There was a problem with the production.' + e.__str__())
Copy the codeThat completes the process.
5. Send emails
Sending emails is another matter. I define a mail.py file with the following code:
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart # An email
def sendMail(msg) :
#
sender = 'Sender'
to_list = [
'To'
]
subject = 'Video Production Status'
# Create mailbox
em = MIMEMultipart()
em['subject'] = subject
em['From'] = sender
em['To'] = ",".join(to_list)
The content of the email
content = MIMEText(msg)
em.attach(content)
# Send email
# 1. Connect to the server
smtp = smtplib.SMTP()
smtp.connect('smtp.163.com')
# 2. Login
smtp.login(sender, 'Your password or authorization code')
# 3. Email
smtp.send_message(em)
# 4. Close the connection
smtp.close()
Copy the codeI wrote the mailbox inside directly dead, you can play freely. For convenience, the sender is recommended to use email 163 and the recipient QQ. In addition, it is convenient to use the password directly when logging in, but there are hidden dangers of security. Finally, SEND you a set of 2020 latest Pyhon project combat video tutorial, click here to get follow the practice, I hope you make progress together!
Six, summarized
To be honest, the above procedure is very inefficient, not only takes up space, but also takes a long time. In the beginning I chose to switch scenes by traversing every pixel of the image, and then found a more efficient way to replace it. But saving frames, and PNG images, is a lot of space.
In addition, there are many unreasonable places in the program design, such as the ndarray object and Image distinction is not high, and some functions choose to pass in the path, and some functions choose to pass in the file object is also easy to confuse. And finally, what we’re going to do is we’re going to be able to do not only static scene switching, but also dynamic scene switching in this way, so that we can make richer videos. Of course, efficiency remains an issue. Interested readers can follow my official account: ZackSock. The full code has been submitted to GitHub: github.com/IronSpiderM… .
Finally, an Easter egg: