preface

Due to the recent project requirements related to computer vision, it is necessary to realize the function of recording video in Python and playing back the video in the front end of the browser after recording. Take a close look at this article to document the overall implementation.

The 2019-08-02 update

I have been busy with other things and did not further explore, so this article has been shelved temporarily. However, it was recently found that the previous implementation method (converting AVI video to MP4 by calling FFmpeg from Java) would affect the performance of the system. The reason is that the CPU consumption of FFmpeg conversion video is high QAQ, so WE continued to look for a solution based on the previous method.

Reduce FFmpeg CPU usage

If the CPU usage of FFmpeg is higher than that of FFmpeg, we can add -threads to the FFmpeg command to specify CPU usage

FFmpeg conversion test

This test used the same AVI video file with a size of 113

1. Original conversion command
ffmpeg -i test.avi -vcodec libx264 -f mp4 test.mp4
The conversion takes 30 seconds to 31 seconds
CPU usage is 950% to 1000%
Copy the code
2. Add-threads 6parameter
ffmpeg -i test.avi -threads 6 -vcodec libx264 -f mp4 test.mp4
The conversion time is 45s~46s
CPU usage is 490% to 550%
Copy the code
3. Add-threads 2parameter
ffmpeg -i test.avi -threads 2 -vcodec libx264 -f mp4 test.mp4
# Conversion time 87s~88s
CPU usage is 205%~230%
Copy the code

We can see that adding the -threads parameter does reduce CPU usage, but the corresponding video conversion time also increases, which is obviously not the desired effect; So there is no escaping the problem of recording H264 video

Compile and install OpenCV to record video

The reason why WE have been unable to record MP4 video encoded in H264 is because we use Opencv-Python installed for PIP. This library comes with FFmpeg, so no matter how much we play with the system FFmpeg, it doesn’t matter. If we want to call the system FFmpeg, we need to manually compile and install OpenCV. For specific reasons, please refer to the following figure:

How to compile and install OpenCV won’t go into detail, nor will it be the focus of this article, but there is a link for lazy cancer sufferers.Ubuntu16.04 install OpenCV with ffmpeg

After compiling and installingimport cv2The code is available in both the original and online versions of fourcc.

So far in Python call OpenCV to record H264 encoded MP4 video has been realized, no special needs students see here can be ~ spread!

Use the VidGear library to record video

For project reasons we can’t use manually compiled OpenCV (WTF!!) So had to continue looking for a solution QAQ

Vidgear – Github official linkThis scheme has been removed from the main theme, and is only adopted for project reasons. It will not be described here, and interested students can have a look.

The following is the original text

Python-opencv records videos

The environment

Python 3.7.1
Opencv – python 3.4.4.19

Import library support

import cv2
Copy the code

Calling the Camera

The input parameter contains digits such as 0, 1, and 2 as the camera index. 0 indicates the built-in camera. You can call the camera in sequence or pass the video file path

cap = cv2.VideoCapture(0)
Copy the code

Obtain the camera width and height

width = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
Copy the code

FPS = cap.get(cv2.cv_cap_prop_fps) FPS = Cap.get (cv2.cv_CAP_prop_fps) FPS = Cap.get (cv2.cv_CAP_prop_fps) FPS = Cap.get (cv2.cv_CAP_prop_fps)

Specifies the video codec

You need to pass in fourCC (Four Character Code) codec code: FourCC reference

encode = cv2.VideoWriter_fourcc(*'mp4v')
Copy the code

Initialize VideoWriter

Reference: official documentation

out = cv2.VideoWriter( './test.mp4', encode, 10, (width, height), True)
Copy the code

Gets the image frame and writes it to the video file

  • Loop a single frame image from the camera/video
  • Open a new window to display the image frame, play the next frame every 25 milliseconds, and type “Q” to break out of the loop
  • Writes image frames to a video file
while True:
    if cv2.waitKey(25) & 0xFF == ord('q') :break
    ret, frame = cap.read()
    cv2.imshow('test', frame)
    out.write(frame)
Copy the code

Release resources

  • Release VideoWriter
  • Release the camera
  • Close the window
out.release()
cap.release()
cv2.destroyAllWindows()
Copy the code

The complete code

This code is demo, for reference only

#! /usr/bin/env python3
# -*- coding: utf-8 -*-

import cv2

# call camera
cap = cv2.VideoCapture(0)
Get the camera width and height
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
Get the camera frame rate
#fps = cap.get(cv2.CAP_PROP_FPS)
# specify fourcc codec
encode = cv2.VideoWriter_fourcc(*'mp4v')
Initialize VideoWriter
out = cv2.VideoWriter('./test.mp4', encode, 10, (width, height), True)
while True:
    # Play the next frame every 25ms, if you type "q" out of the loop
    if cv2.waitKey(25) & 0xFF == ord('q') :break
    # Fetch the next frame from the camera
    ret, frame = cap.read()
    # Open a new window to show the image
    cv2.imshow('test', frame)
    Write the current frame to the video file
    out.write(frame)
# release VideoWriter
out.release()
# Release the camera
cap.release()
# Close window
cv2.destroyAllWindows()
Copy the code

Play the video in the browser

The environment

MacOS Mojave 10.14.3
Ubuntu 16.04
Vue 2.9.6
Nginx 1.15.5
The front end is a VUE project, which is packaged and deployed in NGINx. The server block/Location block is configured to provide static resource access such as images/videos

H5 Video Cannot play videos

Troubleshoot problems
  1. Code error
    • Check whether python records videos successfully
    • Check whether the SRC of video in the front end is correct
  2. Network request
    • Whether the browser console reports an error
    • Whether the nginx service is started
    • The request path is correct
    • Whether is a cross-domain problem
  3. Browser support
format IE Firefox Opera Chrome Safari
Ogg – 3.5 + 10.5 + 5.0 + –
MPEG 4 9.0 + – – 5.0 + 3.0 +
WebM – 4.0 + 10.6 + 6.0 + –
  1. Video codec
format Video coding Audio coding
Ogg Theora Vorbis
MPEG 4 H.264 AAC
WebM VP8 Vorbis

Problem orientation

After troubleshooting the code and network request problems, the problem can be located in the browser. The browser I use is Chrome, and the version problem is excluded. Therefore, it can be determined that the video codec problem is caused by the fact that H.264 codec is not used when recording videos in Python:

encode = cv2.VideoWriter_fourcc(*'mp4v')
Copy the code

It can be found from the introduction of the video that the video is indeed non-H. 264 codec, so the video can be played normally in the video player software but cannot be played in the VIDEO of H5, as shown below:

Try changing FourCC to re-record the video

encode = cv2.VideoWriter_fourcc(*'X264')
Copy the code

$sudo apt-get install ffmpeg x264 libx264-dev
Copy the code

Ubuntu cannot record after installation (video files cannot be generated), but it does not affect recording on my own computer:

Java uses FFmpeg to convert video

Avi format video is recorded in Python, the front end requests the background, and FFmpeg is used to convert the. Avi format video to. MP4 format video in Java

Install FFmpeg first. MacOS install FFmpeg Ubuntu install FFmpeg Java side will not detail, directly on the code ~ (also for the demo, for reference only)

// FFmpeg conversion command String transferCommand ="ffmpeg -i filePath/fileName.avi -vcodec libx264 -f mp4 filePath/fileName.mp4";
    Process process = Runtime.getRuntime().exec("/bin/bash");
    printWriter = new PrintWriter(new BufferedWriter(new 
    OutputStreamWriter(process.getOutputStream())), true);
    printWriter.println(transferCommand); // This command must be executed otherwiseinThe flow never ends.printWriter.println("exit");
    printWriter.close();
    process.waitFor();
Copy the code

The conversion process takes some time. The solution is to start a thread to complete the video conversion without affecting the response time of the current interface and complete the video conversion without the user’s perception.

conclusion

The above content is the record of the implementation process, the code is demo, not the actual application code, if necessary, can be adjusted according to the actual demand. I didn’t put much effort into recording H.264 videos due to time constraints, so I may try again in the future

If you have any questions or similar needs can leave a message or private letter to me ~
If you can achieve.264 video recording or better solution, please feel free to comment ~

Refer to the link

Video Cannot play mp4

What is the Codec for MP4 video in Python OpenCV

FFmpeg limits CPU usage

opencv-python-issues

vidgear-github