This is the 22nd day of my participation in the August Wen Challenge.More challenges in August
background
In the previous article, we elaborated the steps of soft decoding using FFMPEG in Windows environment, and gave the complete code. FFmpeg 4.x from the beginning to master (a) – QT how to use FFmpeg software decoding
This article will continue to learn how to use FFmpeg to implement hard decoding under Windows.
FFmpeg download address (fans as a free download) : download.csdn.net/download/u0…
The language environment of this paper is based on C++, and the interface part is QT.
Process analysis
As always, before we start looking at the code, we must first understand the general flow of ffMPEG hard solution:
Although some of these functions were explained in the first article, to make the article complete, they will be covered again in this article.
1, avformat_open_input
Allocate space to AVFormatContext, open the input video data and probe the format of the video. This function contains complex format parsing and probe algorithms, including: video stream, audio stream, video stream parameters, audio stream parameters, video frame index, etc. In Thor’s words, FFmpeg is the “soul.”
2, avformat_find_stream_info
Obtain information about multimedia streams, including bit stream, frame rate, and duration. But some of the early format or raw stream data is not indexed in the header, so it needs to be probed later. Notice A video file may contain multiple media streams, such as video files, audio files, and subtitle files.
3, av_find_best_stream
When the video is unsealed, the audio and video needs to be processed separately. The corresponding audio stream and video stream need to be found and the stream_index corresponding to the audio and video needs to be obtained.
*4, enum AVHWDeviceType AV_hwDEVICe_finD_type_by_name (const char name)
For hard decoders, the avCOdec_find_decoder method cannot be used to find the decoder. Instead, use the av_hwdevice_find_type_by_name method. To know avCODEC_find_decoder according to AVCodecID search, such as we want to decode H264, that entry is AV_CODEC_ID_H264, find the first registered decoder to exit, The resulting decoder cannot decode AV_CODEC_ID_VP6 files. Most importantly, the same AVCodecID may correspond to several different avCodecs (AVCodec) with different AVCodec.names.
But for example, if you want to use DXVA decoder on Windows, regardless of the video source is H264 or H265, use av_hwdevice_find_type_by_name with dxVA as the input parameter. The resulting decoder can decode BOTH H264 and H265 (provided, of course, that your decoder supports decoding both encodings).
So these are two completely different approaches that may look similar, but don’t be fooled by it.
**5, const AVCodecHWConfig avcodec_get_hw_config(const AVCodec codec, int index);
Get the hardware properties of the decoder codec, such as the target pixel format that can be supported.
6, avcodec_alloc_context3
Create AVCodecContext and allocate space.
7, avcodec_parameters_to_context
This function is used to copy the parameters of the stream, i.e. the parameters of AVStream, directly into the context of the AVCodecContext, performing a true copy of the content. Avcodec_parameters_to_context () is the new API that replaces the old version of avcodec_copy_context().
8 av_hwdevice_ctx_create.
Initialize hardware, enable hardware, and create context information related to the hardware device AVHWDeviceContext, including allocating memory resources and initializing hardware devices. Once the hardware device is initialized, it is also necessary to bind the obtained hardware information to the hw_device_CTx pointer of the AVCodecContext.
If soft decoding is used there is a soft decoding buffer (to get AVFrame) by default, while hard decoding requires the creation of additional hardware-decoded buffers. The buffer variable is hw_frames_ctx. If you don’t manually create one, you automatically create one inside the avcodec_send_packet() function, but you must manually assign the hardware decoding buffer to reference hw_device_CTx (which is an AVBufferRef variable), Otherwise an error will be returned when avcodec_send_packet is called.
9 avcodec_open2.
Initialize AVCodecContext with the given AVCodec.
At this point, the initialization of the decoder is complete. Now we can begin the actual decoding operation.
10, av_read_frame
The av_read_frame() function is a new ffMPEG usage that encapsulates av_read_packet. The old usage was abandoned because the data acquired before may not be complete. Av_read_frame () ensures the integrity of a frame of video data, so that the data read is always a complete frame.
11, avcodec_send_packet
Sends data to the background decoding queue.
It can be NULL (or an AVPacket with data set to NULL and size set to 0); in this case, it is considered a flush packet, which signals the end of the stream. Sending the first flush packet will return success. Subsequent ones are unnecessary and will return AVERROR_EOF. If the decoder still has frames buffered, it will return them after sendingCopy the codePacket. Data = NULL; if av_read_frame can’t read the data, you need to send the packet. packet.size = 0; Send a packet of empty data to FFMPEG, i.e., avCOdec_send_packet once, to flush out all cached frames in FFMPEG and solve the problem that the last few frames were not decoded.
12, avcodec_receive_frame
Read the frame data from the decoder. Once this function is done, it’s ready to get our frame data, which is stored in AVFrame. Note that one avCOdec_send_packet () corresponds to one AVCOdec_receive_frame (), but one avcodec_receive_frame() corresponds to more than one avcodec_receive_frame(). This depends on the specific stream, and is common in audio streams. There may be a situation where one AVPacket corresponds to multiple Avframes. So if you look at my flowchart above there are two while loops.
13, av_hwframe_transfer_data
- This function is to copy data from GPU to CPU after hard decoding. Of course, the premise is that you need to perform data operations in CPU, such as save data or convert it to RGB, YUV, etc., and then render it out through Opengl
TexImage2DTechnical efficiency is low, so this method is not optimal. - So if you want to render the data directly without copying it to the CPU, there are several options, such as the data format decoded by D3D11
AV_PIX_FMT_D3D11VA_VLDCan be converted directly to a D3D texture and then mapped via IDirect3DTexture. Another option is to use EGL+OpenglES, or simply use the open source project ANGLE (the open source EGL+OpenglES implementation). - This function converts hard-decoded data format, that is, converts gPU-mapped data format to CPU-mapped data format, such as DXVA2 decoded data format
AV_PIX_FMT_DXVA2_VLDconvertAV_PIX_FMT_NV12.
Code sample
// header file #ifndef MAINWINDOW_H #define MAINWINDOW_H #include< QMainWindow> #include<thread> extern "C" {#include "libavcodec/avcodec.h" #include "libavformat/avformat.h" #include "libavutil/pixfmt.h" #include "libswscale/swscale.h" #include "libavdevice/avdevice.h" #include <libavutil/pixdesc.h> #include <libavutil/hwcontext.h> #include <libavutil/opt.h> #include <libavutil/avassert.h> #include <libavutil/imgutils.h> } typedef struct DecodeContext { AVBufferRef *hw_device_ref; } DecodeContext; namespace Ui { class MainWindow; } class MainWindow : public QMainWindow { Q_OBJECT public: explicit MainWindow(QWidget *parent = nullptr); ~MainWindow(); void init(); void play(); private: AVBufferRef *hw_device_ctx = NULL; static enum AVPixelFormat hw_pix_fmt; int ret; enum AVHWDeviceType type; std::thread m_decodecThread; Ui::MainWindow *ui; AVFormatContext *pAVFormatCtx; AVCodecContext *pAVCodecCtx; SwsContext *pSwsCtx = nullptr; uint8_t *pRgbBuffer = nullptr; AVPacket packet; AVFrame *pAVFrameRGB = nullptr; int iVideoIndex = -1; QImage m_image; bool isFinish =false; void decodec(); signals: void signalDraw(); public slots: void slotDraw(); protected: void paintEvent(QPaintEvent *event) override; private: int hw_decoder_init(AVCodecContext *ctx, const enum AVHWDeviceType type); static enum AVPixelFormat get_hw_format(AVCodecContext *ctx,const enum AVPixelFormat *pix_fmts); }; #endif // MAINWINDOW_HCopy the code//CPP文件
#include "mainwindow.h"
#include "ui_mainwindow.h"
#include <QDebug>
#include <QPainter>
#include<thread>
#include <QDateTime>
enum AVPixelFormat MainWindow::hw_pix_fmt = AV_PIX_FMT_NONE;
MainWindow::MainWindow(QWidget *parent) :
QMainWindow(parent),
ui(new Ui::MainWindow)
{
ui->setupUi(this);
connect(this,&MainWindow::signalDraw,this,&MainWindow::slotDraw);
}
MainWindow::~MainWindow()
{
delete ui;
}
int MainWindow::hw_decoder_init(AVCodecContext *ctx, const enum AVHWDeviceType type)
{
int err = 0;
//初始化硬件，打开硬件，绑定到具体硬件的指针函数上
//创建硬件设备相关的上下文信息AVHWDeviceContext，包括分配内存资源、对硬件设备进行初始化
if ((err = av_hwdevice_ctx_create(&hw_device_ctx, type,
NULL, NULL, 0)) < 0) {
fprintf(stderr, "Failed to create specified HW device.\n");
return err;
}
/* 需要把这个信息绑定到AVCodecContext
* 如果使用软解码则默认有一个软解码的缓冲区(获取AVFrame的)，而硬解码则需要额外创建硬件解码的缓冲区
* 这个缓冲区变量为hw_frames_ctx，不手动创建，则在调用avcodec_send_packet()函数内部自动创建一个
* 但是必须手动赋值硬件解码缓冲区引用hw_device_ctx(它是一个AVBufferRef变量)
*/
ctx->hw_device_ctx = av_buffer_ref(hw_device_ctx);
return err;
}
enum AVPixelFormat MainWindow::get_hw_format(AVCodecContext *ctx,
const enum AVPixelFormat *pix_fmts)
{
const enum AVPixelFormat *p;
for (p = pix_fmts; *p != -1; p++) {
if (*p == hw_pix_fmt)
return *p;
}
fprintf(stderr, "Failed to get HW surface format.\n");
return AV_PIX_FMT_NONE;
}
void MainWindow::init()
{
std::string file = "E:/Video/bb.mp4";
//描述多媒体文件的构成及其基本信息
if (avformat_open_input(&pAVFormatCtx, file.data(), NULL, NULL) != 0)
{
qDebug() <<"open file fail";
avformat_free_context(pAVFormatCtx);
return;
}
//读取一部分视音频数据并且获得一些相关的信息
if (avformat_find_stream_info(pAVFormatCtx, NULL) < 0)
{
qDebug() <<"vformat find stream fail";
avformat_close_input(&pAVFormatCtx);
return;
}
// 根据解码器枚举类型找到解码器
AVCodec *pAVCodec;
int ret = av_find_best_stream(pAVFormatCtx, AVMEDIA_TYPE_VIDEO, -1, -1, &pAVCodec, 0);
if (ret < 0) {
qDebug()<< "av_find_best_stream faliture";
avformat_close_input(&pAVFormatCtx);
return;
}
iVideoIndex = ret;
type = av_hwdevice_find_type_by_name("dxva2");
// type = av_hwdevice_find_type_by_name("d3d11va");
if (type == AV_HWDEVICE_TYPE_NONE) {
fprintf(stderr, "Device type %s is not supported.\n", "dxva2");
fprintf(stderr, "Available device types:");
while((type = av_hwdevice_iterate_types(type)) != AV_HWDEVICE_TYPE_NONE)
fprintf(stderr, " %s", av_hwdevice_get_type_name(type));
fprintf(stderr, "\n");
}
// 所有支持的硬件解码器保存在AVCodec的hw_configs变量中。对于硬件编码器来说又是单独的AVCodec
for (int i = 0;; i++) {
//获取到该解码器codec的硬件属性，比如可以支持的目标像素格式等
const AVCodecHWConfig *config = avcodec_get_hw_config(pAVCodec, i);
if (!config) {
fprintf(stderr, "Decoder %s does not support device type %s.\n",
pAVCodec->name, av_hwdevice_get_type_name(type));
}
if (config->methods & AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX &&config->device_type == type) {
hw_pix_fmt = config->pix_fmt;
break;
}
}
pAVCodecCtx = avcodec_alloc_context3(pAVCodec);
if (pAVCodecCtx == NULL)
{
qDebug() <<"get pAVCodecCtx fail";
avformat_close_input(&pAVFormatCtx);
return;
}
ret = avcodec_parameters_to_context(pAVCodecCtx,pAVFormatCtx->streams[iVideoIndex]->codecpar);
if (ret < 0)
{
qDebug() <<"avcodec_parameters_to_context fail";
avformat_close_input(&pAVFormatCtx);
return;
}
// 配置获取硬件加速器像素格式的函数；该函数实际上就是将AVCodec中AVHWCodecConfig中的pix_fmt返回
pAVCodecCtx->get_format = get_hw_format;
if (hw_decoder_init(pAVCodecCtx, type) < 0)
return ;
if (avcodec_open2(pAVCodecCtx, pAVCodec, NULL) < 0)
{
qDebug()<<"avcodec_open2 fail";
return;
}
//为解码帧分配内存
pAVFrameRGB = av_frame_alloc();
qDebug()<<"pAVCodecCtx->width:" << pAVCodecCtx->width<<"pAVCodecCtx->height:" << pAVCodecCtx->height;
int size = av_image_get_buffer_size(AVPixelFormat(AV_PIX_FMT_RGB32), pAVCodecCtx->width, pAVCodecCtx->height, 1);
pRgbBuffer = (uint8_t *)(av_malloc(size));
//旧版本avpicture_fill
av_image_fill_arrays(pAVFrameRGB->data, pAVFrameRGB->linesize, pRgbBuffer, AV_PIX_FMT_RGB32,
pAVCodecCtx->width, pAVCodecCtx->height, 1);
// //AVpacket 用来存放解码数据
av_new_packet(&packet, pAVCodecCtx->width * pAVCodecCtx->height);
qDebug()<<"pAVCodecCtx->pix_fmt:" << pAVCodecCtx->pix_fmt;
}
void MainWindow::play()
{
m_decodecThread = std::thread([this]()
{
decodec();
});
m_decodecThread.detach();
}
void MainWindow::decodec()
{
//读取码流中视频帧
while (true)
{
AVFrame *frame = NULL, *sw_frame = NULL;
AVFrame *tmp_frame = NULL;
int ret = av_read_frame(pAVFormatCtx, &packet);
if(ret != 0)
{
qDebug()<<"file end";
isFinish = !isFinish;
return;
}
if (packet.stream_index != iVideoIndex)
{
av_packet_unref(&packet);
continue;
}
int iGotPic = AVERROR(EAGAIN);
// //解码一帧视频数据
iGotPic = avcodec_send_packet(pAVCodecCtx, &packet);
if(iGotPic!=0){
qDebug()<<"avcodec_send_packet error";
continue;
}
if (!(frame = av_frame_alloc()) || !(sw_frame = av_frame_alloc())) {
fprintf(stderr, "Can not alloc frame\n");
ret = AVERROR(ENOMEM);
continue;
}
while (0 == avcodec_receive_frame(pAVCodecCtx, frame))
{
qDebug()<<"frame->format:" << frame->format;//53 AV_PIX_FMT_DXVA2_VLD
if (frame->format == hw_pix_fmt) {
int64_t time = QDateTime::currentDateTime().toMSecsSinceEpoch();
qDebug() << "qhttime1:" << time;
/* retrieve data from GPU to CPU */
if ((ret = av_hwframe_transfer_data(sw_frame, frame, 0)) < 0) {
fprintf(stderr, "Error transferring the data to system memory\n");
break;
}
tmp_frame = sw_frame;
} else
{
tmp_frame = frame;
qDebug()<<"frame img";
}
if(!pSwsCtx)
{
pSwsCtx = sws_getContext(pAVCodecCtx->width, pAVCodecCtx->height, (AVPixelFormat)tmp_frame->format,
pAVCodecCtx->width, pAVCodecCtx->height, AV_PIX_FMT_RGB32,
SWS_BILINEAR, NULL, NULL, NULL);
}
int ret = sws_scale(pSwsCtx, (uint8_t const * const *) tmp_frame->data, tmp_frame->linesize, 0,
tmp_frame->height, pAVFrameRGB->data, pAVFrameRGB->linesize);
qDebug()<<"ret:" << ret;
QImage img((uint8_t *)pAVFrameRGB->data[0], tmp_frame->width, tmp_frame->height, QImage::Format_RGB32);
qDebug()<<"decode img";
m_image = img;
emit signalDraw();
std::this_thread::sleep_for(std::chrono::milliseconds(25));
}
//资源回收
av_free(pAVFrameRGB);
sws_freeContext(pSwsCtx);
avcodec_close(pAVCodecCtx);
avformat_close_input(&pAVFormatCtx);
}
void MainWindow::slotDraw()
{
update();
}
void MainWindow::paintEvent(QPaintEvent *event)
{
QPainter painter(this);
painter.setBrush(Qt::black);
painter.drawRect(0, 0, this->width(), this->height());
if (m_image.size().width() <= 0)
return;
//比例缩放
QImage img = m_image.scaled(this->size(),Qt::KeepAspectRatio);
int x = this->width() - img.width();
int y = this->height() - img.height();
x /= 2;
y /= 2;
//QPoint(x,y)为中心绘制图像
painter.drawImage(QPoint(x,y),img);
}
Copy the code