Using C++ design cache queue to realize efficient transmission of camera data
Requirements:
Doing live function, for example, sometimes we may have to the camera to capture the image data to do some additional operations (such as Crop, Scale, beauty) itself is very time consuming, but due to some operation algorithm of 30 FPS, for example, if a frame processing slower will probably drop frames, so to design a data buffer queue to capture onto a camera in the spare time in the queue, The camera data is then fetched from the work queue if needed in the program.
applicable
- Timing operation on each frame of image in camera callback (Crop, Scale…)
- Improve the efficiency of image processing
- Efficiently handle other large data volume work
Note: This example is designed to cache camera SampleBuffer using a C++ queue, using objective-C and C++.
GitHub address (with code)C++ cache queue
Letter Address:C++ cache queue
Blog Address:C++ cache queue
Address of nuggets:C++ cache queue
Overall process:
- Set always landscape and initialize the camera parameter setting agent
- The SampleBuffer is put into an idle queue in the callback that captures the camera data
- Start a thread to fetch sampleBuffer from the work queue every 10ms to process the data here, and then put the node back to the idle queue
Queue implementation and parsing
Principle 1.
A fixed number of nodes are initialized and loaded into the idle queue. When the camera callback generates data, a node is taken out from the head of the idle queue to load the image buffer of each frame, and then queued to the tail of the work queue. The buffer processing thread takes out the buffer of one node from the head of the work queue for processing. After processing, data in the node containing the secondary buffer will be empty and put back into the head of the idle queue for next use.
parsing
- We design the free queue as head in and head out, which doesn’t have much impact, because we only need to fetch one empty node at a time from the free queue for loading the camera data, so there is no need to guarantee the order of nodes in the way of tail in and head out.
- We design the work queue to be tail in and head out, because we want to make sure that the data captured from the camera is continuous so that we can play the image continuously later, so the work queue must be tail in and head out.
- In this way, we can use the idle queue as the buffer queue. Under normal conditions (FPS =30, i.e. 30 frames per second, approximately one frame every 33ms), if the data can be processed within 33ms, the work queue will always be 0 or 1. However, if we work for a long time or encounter slow data processing of a certain frame (i.e., the processing time is more than 33ms), the length of the work queue will increase, and just because we use such a queue, the data of the slow data processing of that frame can still be processed normally.
Note: this scenario only applies to a short period of time when only a few frames of data are processed slowly. If, for example, more than 20 frames of data are processed slowly within 1s, the work queue may be too long and the advantages of this queue cannot be taken into account.
Structure of 2.
- node
typedef struct XDXCustomQueueNode {
void *data;
size_t size; // data size
long index;
struct XDXCustomQueueNode *next;
} XDXCustomQueueNode;
Copy the codeData of type void * is used in the node to store the sampleBuffer we need. Index is used to record the index of the sampleBuffer currently loaded into the node, so that we can compare whether the node is taken out in order when we take out the node. The element of the next node of the same type is also installed in the node.
- Queue type
typedef struct XDXCustomQueue {
int size;
XDXCustomQueueType type;
XDXCustomQueueNode *front;
XDXCustomQueueNode *rear;
} XDXCustomQueue;
Copy the codeThe number of nodes in the queue is the number of nodes loaded for us. Because we are using pre-allocated fixed memory, the sum of the working queue and the free queue is always the same (because the element in the node is not in the working queue, but in the free queue).
- The design of the class
class XDXCustomQueueProcess {
private:
pthread_mutex_t free_queue_mutex;
pthread_mutex_t work_queue_mutex;
public:
XDXCustomQueue *m_free_queue;
XDXCustomQueue *m_work_queue;
XDXCustomQueueProcess();
~XDXCustomQueueProcess();
// Queue Operation
void InitQueue(XDXCustomQueue *queue,
XDXCustomQueueType type);
void EnQueue(XDXCustomQueue *queue,
XDXCustomQueueNode *node);
XDXCustomQueueNode *DeQueue(XDXCustomQueue *queue);
void ClearXDXCustomQueue(XDXCustomQueue *queue);
void FreeNode(XDXCustomQueueNode* node);
void ResetFreeQueue(XDXCustomQueue *workQueue, XDXCustomQueue *FreeQueue);
};
Copy the codeBecause it involves asynchronous operation, it is necessary to lock the operation of the node. When using it, it is necessary to initialize the queue first, and then define the queuing, queuing, clearing the elements in the queue, freeing the node, resetting the idle queue and other operations.
3. The implementation
- Initializing the queue
const int XDXCustomQueueSize = 3;
XDXCustomQueueProcess::XDXCustomQueueProcess() {
m_free_queue = (XDXCustomQueue *)malloc(sizeof(struct XDXCustomQueue));
m_work_queue = (XDXCustomQueue *)malloc(sizeof(struct XDXCustomQueue));
InitQueue(m_free_queue, XDXCustomFreeQueue);
InitQueue(m_work_queue, XDXCustomWorkQueue);
for (int i = 0; i < XDXCustomQueueSize; i++) {
XDXCustomQueueNode *node = (XDXCustomQueueNode *)malloc(sizeof(struct XDXCustomQueueNode));
node->data = NULL;
node->size = 0;
node->index= 0;
this->EnQueue(m_free_queue, node);
}
pthread_mutex_init(&free_queue_mutex, NULL);
pthread_mutex_init(&work_queue_mutex, NULL);
NSLog(@"XDXCustomQueueProcess Init finish !");
}
Copy the codeAssume that the total number of idle queue nodes is 3. First, allocate memory for the work queue and idle queue, and then initialize them respectively. For the specific process, refer to Demo.
Note: Nodes of reuse, we initialize only a few fixed number of nodes, for processing the data amount is larger, there is no need to let the program always do malloc and free, in order to optimize our queue is equivalent to a static linked list, the nodes of reuse, because when the node is used in the work queue will be after the completion of the data in blank and team again to the free queue, So the total number of nodes is always the same.
- The team the Enqueue
void XDXCustomQueueProcess::EnQueue(XDXCustomQueue *queue, XDXCustomQueueNode *node) {
if (queue == NULL) {
NSLog(@"XDXCustomQueueProcess Enqueue : current queue is NULL");
return;
}
if (node==NULL) {
NSLog(@"XDXCustomQueueProcess Enqueue : current node is NULL");
return;
}
node->next = NULL;
if (XDXCustomFreeQueue == queue->type) {
pthread_mutex_lock(&free_queue_mutex);
if (queue->front == NULL) {
queue->front = node;
queue->rear = node;
}else {
/*
// tail in,head out
freeQueue->rear->next = node;
freeQueue->rear = node;
*/
// head in,head out
node->next = queue->front;
queue->front = node;
}
queue->size += 1;
NSLog(@"XDXCustomQueueProcess Enqueue : free queue size=%d",queue->size);
pthread_mutex_unlock(&free_queue_mutex);
}
if (XDXCustomWorkQueue == queue->type) {
pthread_mutex_lock(&work_queue_mutex);
//TODO
static long nodeIndex = 0;
node->index=(++nodeIndex);
if (queue->front == NULL) {
queue->front = node;
queue->rear = node;
}else {
queue->rear->next = node;
queue->rear = node;
}
queue->size += 1;
NSLog(@"XDXCustomQueueProcess Enqueue : work queue size=%d",queue->size); pthread_mutex_unlock(&work_queue_mutex); }}Copy the codeAs mentioned above, if the enqueue operation is an idle queue, the head is used, that is, the enqueue node is always at the head of the queue. The specific code implementation is to make the next of the current node point to the head of the idle queue, and then change the current node into the head of the idle queue. If the enqueue operation is a work queue, we use the tail-end method and assign the index value to the node, so that when we take out the node, we can print whether the index is continuous. If it is continuous, it means that the enqueue is always in the order.
Here the use of simple data structure in the knowledge, such as do not understand the Internet can be simple access
- Out of the team
XDXCustomQueueNode* XDXCustomQueueProcess::DeQueue(XDXCustomQueue *queue) {
if (queue == NULL) {
NSLog(@"XDXCustomQueueProcess DeQueue : current queue is NULL");
return NULL;
}
const char *type = queue->type == XDXCustomWorkQueue ? "work queue" : "free queue";
pthread_mutex_t *queue_mutex = ((queue->type == XDXCustomWorkQueue) ? &work_queue_mutex : &free_queue_mutex);
XDXCustomQueueNode *element = NULL;
pthread_mutex_lock(queue_mutex);
element = queue->front;
if(element == NULL) {
pthread_mutex_unlock(queue_mutex);
NSLog(@"XDXCustomQueueProcess DeQueue : The node is NULL");
return NULL;
}
queue->front = queue->front->next;
queue->size -= 1;
pthread_mutex_unlock(queue_mutex);
NSLog(@"XDXCustomQueueProcess DeQueue : %s size=%d".type,queue->size);
return element;
}
Copy the codeThe dequeue operation, whether idle queue or working queue, is dequeue from the head node of the current queue.
Note: The null node is not confused with the null data in the node. If the node is null, it means that the node is not removed from the queue, i.e. the null node has no memory address, and the data in the node is node->data. In this Demo, it is the sampleBuffer data generated by the camera for every frame.
- Reset idle queue data
void XDXCustomQueueProcess::ResetFreeQueue(XDXCustomQueue *workQueue, XDXCustomQueue *freeQueue) {
if (workQueue == NULL) {
NSLog(@"XDXCustomQueueProcess ResetFreeQueue : The WorkQueue is NULL");
return;
}
if (freeQueue == NULL) {
NSLog(@"XDXCustomQueueProcess ResetFreeQueue : The FreeQueue is NULL");
return;
}
int workQueueSize = workQueue->size;
if (workQueueSize > 0) {
for (int i = 0; i < workQueueSize; i++) {
XDXCustomQueueNode *node = DeQueue(workQueue);
CFRelease(node->data);
node->data = NULL;
EnQueue(freeQueue, node);
}
}
NSLog(@"XDXCustomQueueProcess ResetFreeQueue : The work queue size is %d, free queue size is %d",workQueue->size, freeQueue->size);
}
Copy the codeWhen we are going to perform some interrupt operations, such as jumping from this View to other views or entering the background, we need to empty all the nodes in the work queue and put them back to the idle queue, so as to ensure that the nodes we applied for initially are still available and will not be lost.
——————————————————–
process
1. Initialize camera parameters
General process, Demo has been implemented, not repeated here
2. Put sampleBuffer into the idle queue
After setting up the camera agent, Void captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer The fromConnection:(AVCaptureConnection *)connection method loads the samplebuffer into an idle queue
- (void)addBufferToWorkQueueWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {
XDXCustomQueueNode *node = _captureBufferQueue->DeQueue(_captureBufferQueue->m_free_queue);
if (node == NULL) {
NSLog(@"XDXCustomQueueProcess addBufferToWorkQueueWithSampleBuffer : Data in , the node is NULL !");
return;
}
CFRetain(sampleBuffer);
node->data = sampleBuffer;
_captureBufferQueue->EnQueue(_captureBufferQueue->m_work_queue, node);
NSLog(@"XDXCustomQueueProcess addBufferToWorkQueueWithSampleBuffer : Data in , work size = %d, free size = %d !",_captureBufferQueue->m_work_queue->size, _captureBufferQueue->m_free_queue->size);
}
Copy the codeNote: Since the sampleBuffer captured in the camera callback has a lifetime, we need to manually CFRetain it to the nodes in our queue.
3. Start a thread to process the Buffer in the queue
Create a thread with pThread and fetch data every 10ms, where we can do whatever we want with the data, and then empty the sampleBuffer and put it into an idle queue for recycling.
- (void)handleCacheThread {
while (trueXDXCustomQueueNode *node = _captureBufferQueue->DeQueue(_captureBufferQueue->m_work_queue);if (node == NULL) {
NSLog(@"Crop handleCropThread : Data node is NULL");
usleep(10*1000);
continue; } CMSampleBufferRef sampleBuffer = (CMSampleBufferRef)node->data; // Prints the index of the node. If it is continuous, the sampleBuffer placed in the camera callback is continuous NSLog(@)"Test index : %ld",node->index); /* The Buffer from the queue can be processed here. When it is used up, remember to free memory and put the node back into the idle queue *........ */ CFRelease(sampleBuffer); node->data = NULL; _captureBufferQueue->EnQueue(_captureBufferQueue->m_free_queue, node); }}Copy the code