1. This paper is mainly based on C++ language, using hongsoft face recognition SDK, to achieve face tracking of local video stream or RTSP video stream.
The realization content includes the real-time detection of the camera image frame, to achieve multi-face tracking or the camera image frame of each face feature and face information features in the face database are compared, to achieve the specified target face tracking.
2. Key technologies 2.1 C++
C++ is an intermediate language designed and developed by Bjarne Stroustrup at bell LABS in 1979. C++ further expands and improves C language, is an object-oriented programming language, and C++ can be well used in a variety of platforms, such as: Windows, Mac and Unix versions.
2.2 OpenCV
OpenCV is an open-source, cross-platform computer vision library that runs on Linux, Windows, Android, and Mac OS operating systems. By a series of C functions and a small amount of C++ class composition, image processing and computer vision to achieve a lot of general algorithm, in the image processing has a very high performance!
2.3 ArcFace
Hongsoft provides offline face recognition library, version 4.1. Hongsoft has the world’s leading visual intelligence technology, and has good performance in multi-platform and multi-language face detection, face tracking, face comparison, richer face attribute detection, IR/RGB living detection, and image quality detection.
Second, the body
We use the Windows version, download hongsoft SDK directly to get a compressed package, after decompressing, we can get the files we need (structure is as follows) :
Note: Our project is carried out in a 32-bit (x86) environment!
1.1 New Project Solution
Use VS2017 to create a new console project on disk E (not familiar with MFC, just use the console as a demo) called myFaceDemo.
We create a precompiled header (or empty project) solution here
Then run it once in the Debug x86 environment first, causing the solution to generate the Debug folder directory.
1.2 Copying required files to workspace:
1. Copy the Inc and lib folders of the SDK decompression package to myFaceDemo solution
Myfacedemo. vcxproj file directory:
2. Copy the OpencV249 folder in Demo to the same directory.
3. Copy the following two files to the debug directory generated by the solution:
4. Copy the. DLL file from the OpencV249 folder to the debug folder.
5. Copy the LIB folder in the SDK decompression package (copied before, but need to copy the DLL files in it to the debug directory) (as shown below) :
6. Copy the OpencV lib file to the myFaceDemo solution lib folder:
7. In addition, copy all lib files from the path shown on the right in 6 to the lib folder instead of the lib/win32 folder (of course, you can also put them in the win32 folder, but select the win32 folder when adding the library later).
8. This ensures that we copy the required DLL files to the DEBUG folder, organize the lib folder to the Lib folder, and copy OpencV to our workspace (i.e. the solution) so that we can add paths later.
1.3 VS2017 Property Page Configuring paths 1. Add the following paths in the additional include directory
E:\myFaceDemo\myFaceDemo\opencv249\include
E:\myFaceDemo\myFaceDemo\opencv249\include\opencv
E:\myFaceDemo\myFaceDemo\opencv249\include\opencv2
Copy the code
2. Add the following path to the additional library directory in the linker
E:\myFaceDemo\myFaceDemo\lib
E:\myFaceDemo\Debug
Copy the code
3. In the linker -> Input -> Attach dependencies add the following:
opencv_imgproc249d.lib
opencv_highgui249d.lib
opencv_core249d.lib
Copy the code
At this point, our environment is configured successfully, let’s add a header file to verify:
1.4 Adding header files
#include "pch.h" #include <stdio.h> #include <stdlib.h> #include "arcsoft_face_sdk.h" #include "amcomdef.h" #include "asvloffscreen.h" #include "merror.h" #include <direct.h> #include <iostream> #include <stdarg.h> #include <string> #include "opencv2/opencv. HPP "#include <fstream>// for file operations, save device information used to generate an offline activation file. DatCopy the codeAfter adding the above header file, we will generate and run again, no error, indicating that our configuration is successful!!
2. The flow chart of all faces in the tracking video is as follows:
2.1 Obtaining Device Information (Used to Generate an Offline Activation File)
using namespace std; using namespace cv; #pragma comment(lib, "libarcsoft_face_engine.lib") ofstream fout; <fstream> void GetDeviceInfo() {MRESULT res = MOK; char* deviceInfo = NULL; // save deviceInfo res = ASFGetActiveDeviceInfo(&deviceInfo); if (res ! = MOK) { printf("ASFGetActiveDeviceInfo: %d\n", res); } fout.open("device.txt"); fout << deviceInfo; } int main() {GetDeviceInfo(); }Copy the codeWe will create a new TXT file named devinfo.txt in myfaceDemo.vcxproj.
In the above code, we mainly wrapped a function to generate device information and store it locally: GetDeviceInfo()
This function calls the ASFGetActiveDeviceInfo interface, saves device information in deviceInfo, and then outputs its value to the file devInfo.txt through file operations.
Run the above program and open the devinfo.txt file to find something saved:
Then we need to use this file to generate an offline activation file (.dat) on the official website of Hongsoft. 2.2 Generating an offline activation file this step should be performed on the official website of Hongsoft.
1. Enter the Developer Center, find the place to download the SDK package, and click the following figure to view the trial code:
Then we find one to activate and click offline activation:
2. Select the SDK version (4.1 in my case) and proceed to the next step:
3. Click Upload device information, and then find the devinfo. TXT file we generated to upload:
Click generate offline authorization file after uploading:
4. Download the license file:
5. Copy the file to the same directory as the device information file devinfo. TXT and rename it decactive.dat
2.3 Activating a Device using an Authorization File
We wrap another function devActive() for offline device activation and call it in the main function:
void devActive() { char ActiveFileName[] = "decActive.dat"; MRESULT res = ASFOfflineActivation(ActiveFileName); If (MOK! = res && MERR_ASF_ALREADY_ACTIVATED ! = res && MERR_ASF_LOCAL_EXIST_USEFUL_ACTIVE_FILE ! = res) { cout << "ASFOfflineActivation failed: " << res << endl; } else cout << "ASFOfflineActivation success" << endl; // Activate successfully print}Copy the codeThen we call this function in the main function and get the following result, indicating that our device has been successfully activated.
2.4 Activating the Engine The interface functions to activate the engine are ASFInitEngine().
MRESULT InitEngine(ASF_DetectMode detectMode) {m_hEngine = NULL; MInt32 mask = 0; if (ASF_DETECT_MODE_IMAGE == detectMode) { mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_AGE | ASF_GENDER | ASF_LIVENESS | ASF_IR_LIVENESS; } else { mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_LIVENESS | ASF_IR_LIVENESS; } MRESULT res = ASFInitEngine(detectMode, ASF_OP_ALL_OUT, FACENUM, mask, &m_hEngine); return res; }Copy the codeMRESULT InitEngine(ASF_DetectMode, detectMode)
DetectMode == ASF_DETECT_MODE_IMAGE: Image detectMode == ASF_DETECT_MODE_VIDEO: video
This way we can activate m_hEngine in the main function with the following statement. Note here that m_hEngine is defined first (see full program)
MHandle m_hEngine; / / engine handleCopy the codeAnd then call
InitEngine(ASF_DETECT_MODE_VIDEO);
Copy the codeIn this way, we have completed the engine activation, and when we do the identification trace later, the rainbow soft SDK function needs to use our activated engine: m_hEngine
1. PicCutOut()
Void PicCutOut(IplImage* SRC, IplImage* DST, int x, int y) {if (! src || ! dst) { return; } CvSize size = cvSize(dst->width, dst->height); CvSetImageROI (SRC, cvRect(x, y, sie.width, sie.height)); CvCopy (SRC, DST); CvResetImageROI (SRC); // After the source image is used, clear the ROI}Copy the code2. ColorSpaceConversion function ColorSpaceConversion()
The processed image information is saved in offScreen
Int ColorSpaceConversion(IplImage* image, MInt32 format, ASVLOFFSCREEN& offscreen) { switch (format) { case ASVL_PAF_RGB24_B8G8R8: offscreen.u32PixelArrayFormat = (unsigned int)format; offscreen.i32Width = image->width; offscreen.i32Height = image->height; offscreen.pi32Pitch[0] = image->widthStep; offscreen.ppu8Plane[0] = (MUInt8*)image->imageData; break; case ASVL_PAF_GRAY: offscreen.u32PixelArrayFormat = (unsigned int)format; offscreen.i32Width = image->width; offscreen.i32Height = image->height; offscreen.pi32Pitch[0] = image->widthStep; offscreen.ppu8Plane[0] = (MUInt8*)image->imageData; break; default: return 0; } return 1; }Copy the codePreDetectFace() : PreDetectFace()
Image data image uses tempEngine. If this parameter is not passed, the default value is m_hEngine, which is used for video recognition and tracking, or m_hImgEngine, which is passed for image extraction, which is used to extract face information from image frames. The return value is:
DetectedFaces for all face information
// If the engine is not passed in, then the video engine is used by default, MRESULT PreDetectFace(IplImage* image, ASF_MultiFaceInfo& MultiFaces, MHandle& tempEngine = m_hEngine) { if (! Image) {cout << "image empty" << endl; return -1; } IplImage* cutImg = NULL; MRESULT res = MOK; MultiFaces = { 0 }; CvCreateImage (cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels) PicCutOut(image, cutImg, 0, 0); ASVLOFFSCREEN offscreen = { 0 }; ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen); res = ASFDetectFacesEx(tempEngine, &offscreen, &MultiFaces); if (res ! = MOK || MultiFaces.faceNum < 1) { cvReleaseImage(&cutImg); return -1; } return res; }Copy the code4. Extracting face feature function using the function, the input is:
Image data image Single face information faceRect Use tempEngine, the default value is m_hEngine, that is, use video tracking engine
Note: We use the same engine to extract face features, so the extracted features should be copied (MEMCPY), because feature extraction by the same engine for many times will cover the previous features, so when feature comparison is carried out, if the features are not copied, the comparison result is always 1.
// Extract features, if not input engine, then the default selection of video engine, face tracking, MRESULT PreExtractFeature(IplImage* image, ASF_FaceFeature& feature, ASF_SingleFaceInfo& faceRect, MHandle& tempEngine = m_hEngine) { if (! image || image->imageData == NULL) return -1; IplImage* cutImg = cvCreateImage(cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels); PicCutOut(image, cutImg, 0, 0); if (! cutImg) { cvReleaseImage(&cutImg); return -1; } MRESULT res = MOK; ASF_FaceFeature detectFaceFeature = { 0 }; // ASVLOFFSCREEN offscreen = {0}; ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen); if (tempEngine == m_hEngine) res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_RECOGNITION, 0, &detectFaceFeature); Else res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_REGISTER, 0, &detectFaceFeature); if (MOK ! = res) { cvReleaseImage(&cutImg); return res; } if (! detectFaceFeature.feature) return -1; feature.featureSize = detectFaceFeature.featureSize; feature.feature = (MByte *)malloc(detectFaceFeature.featureSize); memset(feature.feature, 0, detectFaceFeature.featureSize); memcpy(feature.feature, detectFaceFeature.feature, detectFaceFeature.featureSize); cvReleaseImage(&cutImg); return res; }Copy the code2.6 Obtaining local Stream Image Frames Then in main() function we obtain image frames through local camera or local video for detection:
cv::Mat rgbFrame; CV ::VideoCapture rgbCapture; if (! rgbCapture.isOpened()) { bool res = rgbCapture.open("testVideo.avi"); // Can use local video path //bool res = rgbCapture. Open (0); If (res) cout << "Get local video successfully!" << endl; } // If the camera is used, that is, rGBcapture.open (0) is 0, then you can set the length and width of the camera using the following content: // CV_CAP_PROP_FRAME_WIDTH and CV_CAP_PROP_FRAME_HEIGHT are defined in the program, see the macro definition key value in the complete program. (rgbCapture.set(CV_CAP_PROP_FRAME_WIDTH, VIDEO_FRAME_DEFAULT_WIDTH) && // rgbCapture.set(CV_CAP_PROP_FRAME_HEIGHT, VIDEO_FRAME_DEFAULT_HEIGHT)) // Set the width and height of the camera to set values (here is 640×480)// {// cout << "RGB camera initialization failed!" << endl; // return 1; // if the setup fails, return 1.Copy the code2.7 Video frames are processed in the loop
while (true) { rgbCapture >> rgbFrame; if (! Rgbframe.empty ()) {ASF_SingleFaceInfo faceInfo = {0}; ASF_MultiFaceInfo multiFaceInfo = {0}; IplImage rgbImage(rgbFrame); MRESULT detectRes = PreDetectFace(&rgbImage, faceInfo, multiFaceInfo, true); DetectRes (MOK == detectRes) {for (int I = 0; i < multiFaceInfo.faceNum; i++) { cvRectangle(&rgbImage, cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top), cvPoint(multiFaceInfo.faceRect[i].right, multiFaceInfo.faceRect[i].bottom), cvScalar(0, 0, 255), 2); cvPutText(&rgbImage, to_string(multiFaceInfo.faceID[i]).c_str(), cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top), &font, CV_RGB(255, 0, 0)); IplImage* m_curVideoImage; IplImage* m_curVideoImage; // Save the image m_curVideoImage = cvCloneImage(&rgbImage); CvNamedWindow ("show image"); cvShowImage("show image", m_curVideoImage); if (waitKey(1) >= 0) break; } else// If the video frame is empty, end {cout << "Video ends" << endl; break; } } rgbCapture.release(); // Release rgbCapture after breaking out of the while loopCopy the codeBelow is the complete procedure!!
2.8 Complete Program
#include "pch.h"
#include <stdio.h>
#include <stdlib.h>
#include "arcsoft_face_sdk.h"
#include "amcomdef.h"
#include "asvloffscreen.h"
#include "merror.h"
#include <direct.h>
#include <iostream>
#include <stdarg.h>
#include <string>
#include "opencv2/opencv.hpp"
#include <windows.h>
#include <thread>
#include <time.h>
#include <mutex>
#include <atlstr.h>
#include <future>
#include <fstream> //进行文件操作
//使用命名空间std和cv
using namespace std;
using namespace cv;
//宏定义关键量
#define NSCALE 32 //支持最小的人脸
#define FACENUM 10 //可识别的人脸数目最大为10
#define VIDEO_FRAME_DEFAULT_WIDTH 640//定义摄像头宽
#define VIDEO_FRAME_DEFAULT_HEIGHT 480//定义摄像头高
//进行全局变量的定义
MHandle m_hEngine;//引擎handle
//加载人脸识别库
#pragma comment(lib, "libarcsoft_face_engine.lib")
void GetDevInfo()
{
ofstream fout;
fout.open("devInfo.txt");//将设备信息保存在该文件中
MRESULT res = MOK;
char* deviceInfo = NULL;
res = ASFGetActiveDeviceInfo(&deviceInfo);
if (res != MOK)
{
printf("ASFGetActiveDeviceInfo: %d\n", res);
}
fout << deviceInfo; //进行保存
}
void devActive()
{
char ActiveFileName[] = "decActive.dat"; //创建一个字符数组来保存激活文件的文件名
MRESULT res = ASFOfflineActivation(ActiveFileName); //离线激活
if (MOK != res && MERR_ASF_ALREADY_ACTIVATED != res &&
MERR_ASF_LOCAL_EXIST_USEFUL_ACTIVE_FILE != res)
{
cout << "ASFOfflineActivation failed: " << res << endl; //激活失败打印
}
else cout << "ASFOfflineActivation success" << endl; //激活成功打印
}
//裁剪图片
void PicCutOut(IplImage* src, IplImage* dst, int x, int y)
{
if (!src || !dst)
{
return;
}
CvSize size = cvSize(dst->width, dst->height);//区域大小
cvSetImageROI(src, cvRect(x, y, size.width, size.height));//设置源图像ROI
cvCopy(src, dst); //复制图像
cvResetImageROI(src);//源图像用完后，清空ROI
}
//颜色空间转换
int ColorSpaceConversion(IplImage* image, MInt32 format, ASVLOFFSCREEN& offscreen)
{
switch (format)
{
case ASVL_PAF_RGB24_B8G8R8:
offscreen.u32PixelArrayFormat = (unsigned int)format;
offscreen.i32Width = image->width;
offscreen.i32Height = image->height;
offscreen.pi32Pitch[0] = image->widthStep;
offscreen.ppu8Plane[0] = (MUInt8*)image->imageData;
break;
case ASVL_PAF_GRAY:
offscreen.u32PixelArrayFormat = (unsigned int)format;
offscreen.i32Width = image->width;
offscreen.i32Height = image->height;
offscreen.pi32Pitch[0] = image->widthStep;
offscreen.ppu8Plane[0] = (MUInt8*)image->imageData;
break;
default:
return 0;
}
return 1;
}
MRESULT InitEngine(ASF_DetectMode detectMode)//初始化引擎
{
m_hEngine = NULL;
MInt32 mask = 0;
if (ASF_DETECT_MODE_IMAGE == detectMode)
{
mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_AGE | ASF_GENDER | ASF_LIVENESS | ASF_IR_LIVENESS;
}
else
{
mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_LIVENESS | ASF_IR_LIVENESS;
}
MRESULT res = ASFInitEngine(detectMode, ASF_OP_ALL_OUT, FACENUM, mask, &m_hEngine);
return res;
}
//提取人脸信息
//如果不传入使用的引擎，那么默认使用视频引擎，返回为保存所有人脸的多信息
MRESULT PreDetectFace(IplImage* image, ASF_MultiFaceInfo& MultiFaces, bool isRGB = true, MHandle& tempEngine = m_hEngine)
{
if (!image) { cout << "图片为空" << endl; return -1; }
IplImage* cutImg = NULL;
MRESULT res = MOK;
MultiFaces = { 0 };//人脸检测
cutImg = cvCreateImage(cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels);
PicCutOut(image, cutImg, 0, 0);
ASVLOFFSCREEN offscreen = { 0 };
ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen);
res = ASFDetectFacesEx(tempEngine, &offscreen, &MultiFaces);
if (res != MOK || MultiFaces.faceNum < 1)
{
cvReleaseImage(&cutImg);
return -1;
}
return res;
}
//提取特征，如果不输入引擎，那么默认选用视频引擎，进行人脸人脸追踪，否则进行图片特征提取
MRESULT PreExtractFeature(IplImage* image, ASF_FaceFeature& feature, ASF_SingleFaceInfo& faceRect, MHandle& tempEngine = m_hEngine)
{
if (!image || image->imageData == NULL)
return -1;
IplImage* cutImg = cvCreateImage(cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels);
PicCutOut(image, cutImg, 0, 0);
if (!cutImg) { cvReleaseImage(&cutImg); return -1; }
MRESULT res = MOK;
ASF_FaceFeature detectFaceFeature = { 0 };//特征值
ASVLOFFSCREEN offscreen = { 0 };
ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen);
if (tempEngine == m_hEngine)
res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_RECOGNITION, 0, &detectFaceFeature);//这里相比于3.0版本做了修改
else
res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_REGISTER, 0, &detectFaceFeature);
if (MOK != res) { cvReleaseImage(&cutImg); return res; }
if (!detectFaceFeature.feature)
return -1;
feature.featureSize = detectFaceFeature.featureSize;
feature.feature = (MByte *)malloc(detectFaceFeature.featureSize);
memset(feature.feature, 0, detectFaceFeature.featureSize);
memcpy(feature.feature, detectFaceFeature.feature, detectFaceFeature.featureSize);
cvReleaseImage(&cutImg);
return res;
}
int main()
{
//获取设备信息
GetDevInfo();//调用获取设备信息的函数
//离线激活设备
devActive();//调用激活设备的函数
InitEngine(ASF_DETECT_MODE_VIDEO);
//捕获图像并开始处理
cv::Mat rgbFrame;//保存图像帧
cv::VideoCapture rgbCapture;//获取本地摄像头视频
if (!rgbCapture.isOpened())
{
bool res = rgbCapture.open("testVideo.avi");//可以使用本地视频路径
//bool res = rgbCapture.open("rtsp://127.0.0.1:8554/demo");//可以使用rtsp视频流
//bool res = rgbCapture.open(0);//可以使用本地摄像头
if (res)
cout << "打开本地摄像头成功！" << endl;
}
if (!(rgbCapture.set(CV_CAP_PROP_FRAME_WIDTH, VIDEO_FRAME_DEFAULT_WIDTH) &&
rgbCapture.set(CV_CAP_PROP_FRAME_HEIGHT, VIDEO_FRAME_DEFAULT_HEIGHT)))//设置摄像头的宽和高分别为设定值（这里为640×480）
{
cout << "RGB摄像头初始化失败！" << endl;
return 1;//设置失败的话就返回1，程序结束
}
CvFont font;
cvInitFont(&font, CV_FONT_HERSHEY_COMPLEX, 0.5, 0.5, 1, 1, 8);
while (true)
{
rgbCapture >> rgbFrame;
if (!rgbFrame.empty())
{
ASF_MultiFaceInfo multiFaceInfo = { 0 };
IplImage rgbImage(rgbFrame);
MRESULT detectRes = PreDetectFace(&rgbImage, multiFaceInfo);//faceInfo保存的是视频帧上识别到的最大的单人脸信息
if (MOK == detectRes)
{
//对检测到的所有人脸信息画框
int i = multiFaceInfo.faceNum;
for (int i = 0; i < multiFaceInfo.faceNum; i++)
{
cvRectangle(&rgbImage, cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top),
cvPoint(multiFaceInfo.faceRect[i].right, multiFaceInfo.faceRect[i].bottom), cvScalar(0, 0, 255), 2);
cvPutText(&rgbImage, to_string(multiFaceInfo.faceID[i]).c_str(), cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top), &font, CV_RGB(255, 0, 0));
}
}
//显示图像
IplImage* m_curVideoImage;//保存拷贝后的图像
m_curVideoImage = cvCloneImage(&rgbImage);//复制一份图像
cvNamedWindow("show image");
cvShowImage("show image", m_curVideoImage);
if (waitKey(1) >= 0)
break;
}
else
{
cout << "视频播放结束" << endl;
break;
}
}
rgbCapture.release();
return 0;
}
Copy the code2.9 running effect intercept several local video for face tracking effect, you can see the effect is very good!!
If we have a link to push the RTSP video stream, then it is easy to change the following code:
bool res = rgbCapture.open("testVideo.avi");
Copy the codeIs amended as:
Bool RES = rgbCapture. Open (" RTSP video stream link ");Copy the codeIn this way, you can get the corresponding screen.
If we don’t have an existing link, then we can use a VLC software to generate an RTSP video stream link to test it out!
1. Download VLC software — > Download link
2. Open the software after downloading and installing, select media -> Stream in the upper left corner, then select capture device, select our laptop camera and audio input, and finally select Stream to go to the next step.
3. Click next:
4. According to below operation, then select RTSP, click add after input a pathname casually, then generate the RTSP url is: RTSP: / / 127.0.0.1:8554 / custom path name \ my address is here: RTSP: / / 127.0.0.1:8554 / demo
5. Then select a profile and click next:
6. Finally click on the stream
Then the RTSP stream will be pushed through the software, which we used to test:
3. Specify the face tracking identification To specify the face tracking recognition, we at the beginning of the main program to join the specified image face feature extracting, joined in video frame processing all face feature extraction and a are compared to determine whether a confidence value is greater than the set THRESHOLD THRESHOLD (here set 0.82, can be adjusted according to actual situation)
The following mainly introduces the face comparison and tracking part:
3.1 Obtain the characteristic information of the specified face
string img_name; Cout << "Please enter the name in the image library :" << endl; cin >> img_name; string fimgPath = filePath + img_name + ".jpg"; Mat tempImg = imread(fimgpath.c_str ())); IplImage getImg(tempImg); ASF_MultiFaceInfo getMultiFaceInfo = { 0 }; PreDetectFace(&getImg, getMultiFaceInfo, true, m_hImgEngine); ASF_SingleFaceInfo getFaceInfo; ASF_SingleFaceInfo getFaceInfo; getFaceInfo.faceRect = getMultiFaceInfo.faceRect[0]; getFaceInfo.faceOrient = getMultiFaceInfo.faceOrient[0]; getFaceInfo.faceDataInfo = getMultiFaceInfo.faceDataInfoList[0]; FaceFeature = {0}; // ASF_FaceFeature = {0}; PreExtractFeature(&getImg, faceFeature, getFaceInfo, m_hImgEngine);Copy the codeThe main steps of the above code are:
First, we save the image path and load the image based on the entered image name
Face recognition is performed on the loaded picture, and the recognized face information getFaceInfo is used for the next step of feature extraction. Since we get multi-face information, feature extraction requires single face information. So here’s what you do: Save the first of the multiple faces in getFaceInfo for the single face (you can also find the largest face, adjust as needed)
Feature extraction is carried out, and the extraction results are saved in faceFeature for feature comparison later
3.2 Feature Comparison
for (int i = 0; i < multiFaceInfo.faceNum; i++)
{
tempFaceInfo.faceDataInfo = multiFaceInfo.faceDataInfoList[i];
tempFaceInfo.faceOrient = multiFaceInfo.faceOrient[i];
tempFaceInfo.faceRect = multiFaceInfo.faceRect[i];
cvRectangle(&rgbImage, cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top),
cvPoint(multiFaceInfo.faceRect[i].right, multiFaceInfo.faceRect[i].bottom), cvScalar(0, 0, 255), 2);
PreExtractFeature(&rgbImage, tempFeature, tempFaceInfo);
ASFFaceFeatureCompare(m_hEngine, &tempFeature, &faceFeature, &confidenceLeve);
if (confidenceLeve
>= THRESHOLD)
{
string text = to_string(multiFaceInfo.faceID[i]) + " level:" + to_string(confidenceLeve);
cvPutText(&rgbImage, text.c_str(), cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top), &font, CV_RGB(255, 0, 0));
}
Copy the codeDuring the loop processing of video frames, each face in the multi-face information is saved in tempFaceInfo one by one.
Feature extraction is then performed according to tempFaceInfo.
The extracted eigenvalues were compared with the previously obtained image eigenvalues
The confidence value after comparison is stored in confidenceLeve, and then the judgment between threshold value and threshold value is performed. If the judgment is successful, the confidence result will be displayed in the video.
3.3 Complete Program
#include "pch.h"
#include <stdio.h>
#include <iostream>
#include <stdlib.h>
#include "arcsoft_face_sdk.h"
#include "amcomdef.h"
#include "asvloffscreen.h"
#include "merror.h"
#include <string>
#include "opencv2/opencv.hpp"
#include <fstream> //进行文件操作
//使用命名空间std和cv
using namespace std;
using namespace cv;
//宏定义关键量
#define NSCALE 32 //支持最小的人脸
#define FACENUM 10 //可识别的人脸数目最大为10
#define VIDEO_FRAME_DEFAULT_WIDTH 640//定义摄像头宽
#define VIDEO_FRAME_DEFAULT_HEIGHT 480//定义摄像头高
#define filePath "./face_lib/"
#define THRESHOLD 0.82
//进行全局变量的定义
MHandle m_hEngine;//引擎handle
MHandle m_hImgEngine;
//加载人脸识别库
#pragma comment(lib, "libarcsoft_face_engine.lib")
void GetDevInfo()
{
ofstream fout;
fout.open("devInfo.txt");//将设备信息保存在该文件中
MRESULT res = MOK;
char* deviceInfo = NULL;
res = ASFGetActiveDeviceInfo(&deviceInfo);
if (res != MOK)
{
printf("ASFGetActiveDeviceInfo: %d\n", res);
}
fout << deviceInfo; //进行保存
}
void devActive()
{
char ActiveFileName[] = "decActive.dat"; //创建一个字符数组来存储激活文件的文件名
MRESULT res = ASFOfflineActivation(ActiveFileName); //离线激活
if (MOK != res && MERR_ASF_ALREADY_ACTIVATED != res &&
MERR_ASF_LOCAL_EXIST_USEFUL_ACTIVE_FILE != res)
{
cout << "ASFOfflineActivation failed: " << res << endl; //激活失败打印
}
else cout << "ASFOfflineActivation success" << endl; //激活成功打印
}
//裁剪图片
void PicCutOut(IplImage* src, IplImage* dst, int x, int y)
{
if (!src || !dst)
{
return;
}
CvSize size = cvSize(dst->width, dst->height);//区域大小
cvSetImageROI(src, cvRect(x, y, size.width, size.height));//设置源图像ROI
cvCopy(src, dst); //复制图像
cvResetImageROI(src);//源图像用完后，清空ROI
}
//颜色空间转换
int ColorSpaceConversion(IplImage* image, MInt32 format, ASVLOFFSCREEN& offscreen)
{
switch (format)
{
case ASVL_PAF_RGB24_B8G8R8:
offscreen.u32PixelArrayFormat = (unsigned int)format;
offscreen.i32Width = image->width;
offscreen.i32Height = image->height;
offscreen.pi32Pitch[0] = image->widthStep;
offscreen.ppu8Plane[0] = (MUInt8*)image->imageData;
break;
case ASVL_PAF_GRAY:
offscreen.u32PixelArrayFormat = (unsigned int)format;
offscreen.i32Width = image->width;
offscreen.i32Height = image->height;
offscreen.pi32Pitch[0] = image->widthStep;
offscreen.ppu8Plane[0] = (MUInt8*)image->imageData;
break;
default:
return 0;
}
return 1;
}
//初始化引擎
//如果是图片模式，那么激活引擎m_hImgEngine
//如果是视频模式，那么激活引擎m_hEngine
MRESULT InitEngine(ASF_DetectMode detectMode)
{
MInt32 mask = 0;
if (ASF_DETECT_MODE_IMAGE == detectMode)
{
mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_LIVENESS;
MRESULT res = ASFInitEngine(detectMode, ASF_OP_ALL_OUT, FACENUM, mask, &m_hImgEngine);
return res;
}
else
{
mask = ASF_FACE_DETECT | ASF_FACERECOGNITION | ASF_LIVENESS | ASF_UPDATE_FACEDATA;
MRESULT res = ASFInitEngine(detectMode, ASF_OP_ALL_OUT, FACENUM, mask, &m_hEngine);
return res;
}
}
//提取人脸信息
//如果不传入使用的引擎，那么默认使用视频引擎，返回为保存所有人脸的多信息
MRESULT PreDetectFace(IplImage* image, ASF_MultiFaceInfo& MultiFaces, MHandle& tempEngine = m_hEngine)
{
if (!image) { cout << "图片为空" << endl; return -1; }
IplImage* cutImg = NULL;
MRESULT res = MOK;
MultiFaces = { 0 };//人脸检测
cutImg = cvCreateImage(cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels);
PicCutOut(image, cutImg, 0, 0);
ASVLOFFSCREEN offscreen = { 0 };
ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen);
res = ASFDetectFacesEx(tempEngine, &offscreen, &MultiFaces);
if (res != MOK || MultiFaces.faceNum < 1)
{
cvReleaseImage(&cutImg);
return -1;
}
return res;
}
//提取特征，如果不输入引擎，那么默认选用视频引擎，进行人脸人脸追踪，否则进行图片特征提取
MRESULT PreExtractFeature(IplImage* image, ASF_FaceFeature& feature, ASF_SingleFaceInfo& faceRect, MHandle& tempEngine = m_hEngine)
{
if (!image || image->imageData == NULL)
return -1;
IplImage* cutImg = cvCreateImage(cvSize(image->width - (image->width % 4), image->height), IPL_DEPTH_8U, image->nChannels);
PicCutOut(image, cutImg, 0, 0);
if (!cutImg) { cvReleaseImage(&cutImg); return -1; }
MRESULT res = MOK;
ASF_FaceFeature detectFaceFeature = { 0 };//特征值
ASVLOFFSCREEN offscreen = { 0 };
ColorSpaceConversion(cutImg, ASVL_PAF_RGB24_B8G8R8, offscreen);
if (tempEngine == m_hEngine)
res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_RECOGNITION, 0, &detectFaceFeature);//这里相比于3.0版本做了修改
else
res = ASFFaceFeatureExtractEx(tempEngine, &offscreen, &faceRect, ASF_REGISTER, 0, &detectFaceFeature);
if (MOK != res) { cvReleaseImage(&cutImg); return res; }
if (!detectFaceFeature.feature)
return -1;
feature.featureSize = detectFaceFeature.featureSize;
feature.feature = (MByte *)malloc(detectFaceFeature.featureSize);
memset(feature.feature, 0, detectFaceFeature.featureSize);
memcpy(feature.feature, detectFaceFeature.feature, detectFaceFeature.featureSize);
cvReleaseImage(&cutImg);
return res;
}
int main()
{
//离线激活设备
devActive();//调用激活设备的函数
InitEngine(ASF_DETECT_MODE_VIDEO);
InitEngine(ASF_DETECT_MODE_IMAGE);
//输入名字，从图片库中加载图片提取特征
string img_name;
cout << "请输入图片库中姓名:" << endl;
cin >> img_name;
string fimgPath = filePath + img_name + ".jpg"; //简单直接的做法，如果想要更准确正式的获取路径或文件，可以使用c++的文件操作
Mat tempImg = imread(fimgPath.c_str());
IplImage getImg(tempImg);
ASF_MultiFaceInfo getMultiFaceInfo = { 0 };
PreDetectFace(&getImg, getMultiFaceInfo, m_hImgEngine);//图片中的人脸信息保存在getMultiFaceInfo中
//将多人脸信息提取第一个保存在getFaceInfo，用于下一步的特征提取
ASF_SingleFaceInfo getFaceInfo;
getFaceInfo.faceRect = getMultiFaceInfo.faceRect[0];
getFaceInfo.faceOrient = getMultiFaceInfo.faceOrient[0];
getFaceInfo.faceDataInfo = getMultiFaceInfo.faceDataInfoList[0];//相比于3.1版本新加入的
ASF_FaceFeature faceFeature = { 0 };
PreExtractFeature(&getImg, faceFeature, getFaceInfo, m_hImgEngine);
//捕获图像并开始处理
cv::Mat rgbFrame;//保存图像帧
cv::VideoCapture rgbCapture;//获取本地摄像头视频
if (!rgbCapture.isOpened())
{
//bool res = rgbCapture.open("rtsp://127.0.0.1:8554/demo");//可以使用本地视频路径
bool res = rgbCapture.open(0);//可以使用本地摄像头
if (res)
cout << "打开本地摄像头成功！" << endl;
else
{
cout << "打开本地摄像头失败！" << endl;
return -1;
}
}
if (!(rgbCapture.set(CV_CAP_PROP_FRAME_WIDTH, VIDEO_FRAME_DEFAULT_WIDTH) &&
rgbCapture.set(CV_CAP_PROP_FRAME_HEIGHT, VIDEO_FRAME_DEFAULT_HEIGHT)))//设置摄像头的宽和高分别为设定值（这里为640×480）
{
cout << "RGB摄像头初始化失败！" << endl;
return 1;//设置失败的话就返回1，程序结束
}
CvFont font;
cvInitFont(&font, CV_FONT_HERSHEY_COMPLEX, 0.5, 0.5, 1, 1, 8);
ASF_SingleFaceInfo tempFaceInfo = { 0 };//临时保存从视频帧上识别到的单人脸信息
ASF_MultiFaceInfo multiFaceInfo = { 0 };
ASF_FaceFeature tempFeature = { 0 };
MFloat confidenceLeve;
cout << "开始识别！" << endl;
while (true)
{
rgbCapture >> rgbFrame;
if (!rgbFrame.empty())//判断不为空时再处理
{
tempFaceInfo = { 0 };//临时保存从视频帧上识别到的单人脸信息
multiFaceInfo = { 0 };
tempFeature = { 0 };
IplImage rgbImage(rgbFrame);
MRESULT detectRes = PreDetectFace(&rgbImage, multiFaceInfo);//faceInfo保存的是视频帧上识别到的最大的单人脸信息
if (MOK == detectRes)
{
//对检测到的所有人脸信息画框
for (int i = 0; i < multiFaceInfo.faceNum; i++)
{
tempFaceInfo.faceDataInfo = multiFaceInfo.faceDataInfoList[i];
tempFaceInfo.faceOrient = multiFaceInfo.faceOrient[i];
tempFaceInfo.faceRect = multiFaceInfo.faceRect[i];
cvRectangle(&rgbImage, cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top),
cvPoint(multiFaceInfo.faceRect[i].right, multiFaceInfo.faceRect[i].bottom), cvScalar(0, 0, 255), 2);
PreExtractFeature(&rgbImage, tempFeature, tempFaceInfo);
ASFFaceFeatureCompare(m_hEngine, &tempFeature, &faceFeature, &confidenceLeve);
if (confidenceLeve >= THRESHOLD)
{
string text = to_string(multiFaceInfo.faceID[i]) + " level:" + to_string(confidenceLeve);
cvPutText(&rgbImage, text.c_str(), cvPoint(multiFaceInfo.faceRect[i].left, multiFaceInfo.faceRect[i].top), &font, CV_RGB(255, 0, 0));
}
}
}
//显示图像
IplImage* m_curVideoImage;//保存拷贝后的图像
m_curVideoImage = cvCloneImage(&rgbImage);//复制一份图像
cvNamedWindow("show image");
cvShowImage("show image", m_curVideoImage);
if (waitKey(1) >= 0)
break;
}
else
{
cout << "视频获取结束" << endl;
break;
}
}
rgbCapture.release();
cout << "识别结束" << endl;
return 0;
}
/***************************
//第一次运行时使用
int main()
{
//获取设备信息
GetDevInfo();//调用获取设备信息的函数
}
***************************/
Copy the codeCreate a face_lib folder in the project folder and place the picture of the specified face
When running the program, you can enter the name of the picture library, and the tracking effect of the specified face is as follows:
Third, summary
This is the first time for me to use SDK4.1 version, and I used 3.1 version before. Compared with 3.1 version, there are some changes in this version. When I was developing demo, I did it according to my previous experience, and I did not read the changes, which led to many detours in this development. In the end, I find the problem bit by bit by debugging, and then read the development document again to solve it. When developing a project using an SDK or other technology, it’s important to understand the development documentation and change instructions to save time and detours.
For more information about face recognition products, please visit the Open platform of Rainbow Soft vision