The last article briefly introduced the use of AVFoundation. This article will look at how to use VideoToolbox and AudioToolbox to achieve audio and video codec. The video stream format studied is H264 and the audio stream format is AAC. If you are not familiar with these two formats, click here for a quick understanding.
VideoToolbox
VideoToolbox is a low-level framework that provides direct access to hardware codecs. Can provide video compression and decompression services, as well as provide format conversion of images stored in the CoreVideo pixel buffer. First, let’s take a look at the CMSampleBuffer data structure. In the process of video collection, there are the following two data structures
As you can see, the difference is the CMBlockBuffer and CVPixelBuffer.
CVPixelBuffer is unencoded original data, while CMBlockBuffer is encoded H264 raw stream data, as shown below:
Let’s implement this process in code
The coding part
Initialize the
#pragma mark - Code
- (void)encodeInit{
OSStatus status = VTCompressionSessionCreate(kCFAllocatorDefault, (uint32_t)_width, (uint32_t)_height, kCMVideoCodecType_H264, NULL, NULL, NULL, encodeCallBack, (__bridge void * _Nullable)(self), &_encoderSession);
if(status ! = noErr) {return; } / / set the real-time encoding status = VTSessionSetProperty (_encoderSession kVTCompressionPropertyKey_RealTime, kCFBooleanTrue); / / set the filter frames status = B VTSessionSetProperty (_encoderSession kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel); CFNumberRef bitRate = (__bridge CFNumberRef)(@(_height*1000)); status = VTSessionSetProperty(_encoderSession, kVTCompressionPropertyKey_AverageBitRate, bitRate); CFArrayRef = (__bridge CFArrayRef)(@[@(_height*1000/4),@(_height*1000*4)]); status = VTSessionSetProperty(_encoderSession, kVTCompressionPropertyKey_DataRateLimits, limits); // set FPS CFNumberRef FPS = (__bridge CFNumberRef)(@(25)); status = VTSessionSetProperty(_encoderSession, kVTCompressionPropertyKey_ExpectedFrameRate, fps); CFNumberRef maxKeyFrameInterval = (__bridge CFNumberRef)(@(25*2)); status = VTSessionSetProperty(_encoderSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, maxKeyFrameInterval); / / ready to code status = VTCompressionSessionPrepareToEncodeFrames (_encoderSession); }Copy the code
VTCompressionSessionCreate argument parsing
/* * allocator: memory allocator: NULL * * width, height: video frame pixel width, if the encoder does not support this width may change * * codecType: encoding type, enumeration * encoderSpecification: Specify a specific encoder. If NULL is filled, the VideoToolBox will automatically select *sourceImageBufferAttributes: Attributes of the source pixel buffer used to create a pixel buffer pool for the source frame. If this parameter has a value, VideoToolBox will create a buffer pool, no buffer pool can be set to NULL. Using VideoToolbox with unallocated pixel buffers may increase the chances of copying image data. * compressedDataAllocator: a memory allocator for compressed data, NULL using the default allocator * outputCallback: a callback function for compressed video output data. This function was on the call VTCompressionSessionEncodeFrame thread asynchronous calls. NULL, call when VTCompressionSessionEncodeFrameWithOutputHandler calls for coding frame. * compressionSessionOut: param outputCallbackRefCon: a custom pointer to a callback function, in which we normally pass self, to retrieve the current class's methods and properties. Encoder handle, Incoming encoder pointer * / VT_EXPORT OSStatus VTCompressionSessionCreate (CM_NULLABLE CFAllocatorRef allocator, int32_t width, int32_t height, CMVideoCodecType codecType, CM_NULLABLE CFDictionaryRef encoderSpecification, CM_NULLABLE CFDictionaryRefsourceImageBufferAttributes, CM_NULLABLE CFAllocatorRef compressedDataAllocator, CM_NULLABLE VTCompressionOutputCallback outputCallback, void * CM_NULLABLE outputCallbackRefCon, CM_RETURNS_RETAINED_PARAMETER CM_NULLABLE VTCompressionSessionRef * CM_NONNULL compressionSessionOut) API_AVAILABLE (macosx (10.8), the ios (8.0), tvos (10.2));Copy the code
void encodeCallBack (
void * CM_NULLABLE outputCallbackRefCon,
void * CM_NULLABLE sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CM_NULLABLE CMSampleBufferRef sampleBuffer ){
if(status ! = noErr) { NSLog(@"encodeVideoCallBack: encode error, status = %d",(int)status);
return;
}
if(! CMSampleBufferDataIsReady(sampleBuffer)) { NSLog(@"encodeVideoCallBack: data is not ready");
return; } Demo2ViewController *VC = (__bridge Demo2ViewController *)(outputCallbackRefCon); BOOL isKeyFrame = NO; CFArrayRef attachArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer,true); isKeyFrame = ! CFDictionaryContainsKey(CFArrayGetValueAtIndex(attachArray, 0), kCMSampleAttachmentKey_NotSync); //(note the inverse sign)if(isKeyFrame && ! VC->hasSpsPps) { size_t spsSize, spsCount; size_t ppsSize, ppsCount; const uint8_t *spsData, *ppsData; / / get the image source format CMFormatDescriptionRef formatDesc = CMSampleBufferGetFormatDescription (sampleBuffer); OSStatus status1 = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDesc, 0, &spsData, &spsSize, &spsCount, 0); OSStatus status2 = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDesc, 1, &ppsData, &ppsSize, &ppsCount, 0);if (status1 == noErr && status2 == noErr) {
VC->hasSpsPps = true; //sps data VC->sps = [NSMutableData dataWithCapacity:4 + spsSize]; [VC->sps appendBytes:StartCode length:4]; [VC->sps appendBytes:spsData length:spsSize]; //pps data VC->pps = [NSMutableData dataWithCapacity:4 + ppsSize]; [VC->pps appendBytes:StartCode length:4]; [VC->pps appendBytes:ppsData length:ppsSize]; [VC decodeH264Data:VC->sps]; [VC decodeH264Data:VC->pps]; } // get NALU data size_t lengthAtOffset, totalLength; char *dataPoint; / / copy the data to the former dataPoint CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer (sampleBuffer); OSStatus error = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPoint);if(error ! = kCMBlockBufferNoErr) { NSLog(@"VideoEncodeCallback: get datapoint failed, status = %d", (int)error);
return; } // loop to get nalu data size_t offet = 0; Const int lengthInfoSize = 4;while(offet < totalLength - lengthInfoSize) { uint32_t naluLength = 0; // Obtain the length of nALU data memcpy(&naluLength, dataPoint + offet, lengthInfoSize); NaluLength = CFSwapInt32BigToHost(naluLength); NSMutableData *data = [NSMutableData dataWithCapacity:4 + naluLength]; [data appendBytes:StartCode length:4]; [data appendBytes:dataPoint + offet + lengthInfoSize length:naluLength]; dispatch_async(VC.encoderCallbackQueue, ^{ [VC decodeH264Data:data]; }); Offet += lengthInfoSize + naluLength; }}Copy the code
VTCompressionOutputCallback argument parsing
/* * outputCallbackRefCon: reference value of the callback function. *sourceFrameRefCon: Frame reference value fromsourceCopy to VTCompressionSessionEncodeFrame FrameRefCon parameters. * status: Returns noErr if compression succeeds; If compression is unsuccessful, an error code is issued. * infoFlags: Contains information about encoding operations. If the encoding is running asynchronously, set kVTEncodeInfo_Asynchronous. If frames are dropped, set kVTEncodeInfo_FrameDropped. * sampleBuffer: contains the compressed frame if the compression succeeds and the frame is not deleted; Otherwise, empty. */ typedef void (*VTCompressionOutputCallback)( void *outputCallbackRefCon, void *sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer
);
Copy the code
Decoding part
Initialize the
- (void)decodeVideoInit { const uint8_t * const parameterSetPointers[2] = {_sps, _pps}; const size_t parameterSetSizes[2] = {_spsSize, _ppsSize}; int naluHeaderLen = 4; / / decoding parameters according to the SPS PPS configuration OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets (kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_decodeDesc);if(status ! = noErr) { NSLog(@"Video hard DecodeSession create H264ParameterSets(sps, pps) failed status= %d", (int)status);
return; } / / configuration video output parameters NSDictionary * destinationPixBufferAttrs = @ {kCVPixelBufferPixelFormatTypeKey (id) : [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange], // On iOS nv12(UVUV configuration) instead of NV21 (VUVU configuration) (ID) [NSNumber numberWithInteger:_width], (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:_height], (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]}; VTDecompressionOutputCallbackRecord callbackRecord; callbackRecord.decompressionOutputCallback = videoDecompressionOutputCallback; callbackRecord.decompressionOutputRefCon = (__bridge void * _Nullable)(self); / / create the decoder status = VTDecompressionSessionCreate (kCFAllocatorDefault _decodeDesc, NULL, (__bridge CFDictionaryRef _Nullable)(destinationPixBufferAttrs), &callbackRecord, &_decoderSession);if(status ! = noErr) { NSLog(@"Video hard DecodeSession create failed status= %d", (int)status);
return; } / / set the decoding real-time VTSessionSetProperty (_decoderSession kVTDecompressionPropertyKey_RealTime, kCFBooleanTrue); }Copy the code
/ * * decompressionOutputRefCon: callback references *sourceFrameRefCon: reference to frames * status: a status flag (containing undefined code) * infoFlags: flag indicating synchronous/asynchronous decoding, or whether the decoder intends to drop frames * imageBuffer: Buffer of the actual image * presentationTimeStamp: the timestamp that appears * presentationDuration: The duration of a * / typedef void VTDecompressionOutputCallback (*) (void * CM_NULLABLE decompressionOutputRefCon, void * CM_NULLABLEsourceFrameRefCon,
OSStatus status,
VTDecodeInfoFlags infoFlags,
CM_NULLABLE CVImageBufferRef imageBuffer,
CMTime presentationTimeStamp,
CMTime presentationDuration );
Copy the code
VTDecompressionSessionCreate argument parsing
/ * * allocator: memory session * videoFormatDescription: describe the source video frame * videoDecoderSpecification: Specify must use a specific video decoder * destinationImageBufferAttributes: describe the source pixel buffer requirements * outputCallback: use the decompression frame to invoke the callback * decompressionSessionOut: Points to a variable to receive the new decompression session * / VT_EXPORT OSStatus VTDecompressionSessionCreate (CM_NULLABLE CFAllocatorRef allocator, CM_NONNULL CMVideoFormatDescriptionRef videoFormatDescription, CM_NULLABLE CFDictionaryRef videoDecoderSpecification, CM_NULLABLE CFDictionaryRef destinationImageBufferAttributes, const VTDecompressionOutputCallbackRecord * CM_NULLABLE outputCallback, CM_RETURNS_RETAINED_PARAMETER CM_NULLABLE VTDecompressionSessionRef * CM_NONNULL decompressionSessionOut) API_AVAILABLE (macosx (10.8), the ios (8.0), tvos (10.2));Copy the code
/ * * decompressionOutputRefCon: callback references *sourceFrameRefCon: reference to frames * status: a status flag (containing undefined code) * infoFlags: flag indicating synchronous/asynchronous decoding, or whether the decoder intends to drop frames * imageBuffer: Buffer of the actual image * presentationTimeStamp: the timestamp that appears * presentationDuration: The duration of a * / typedef void VTDecompressionOutputCallback (*) (void * CM_NULLABLE decompressionOutputRefCon, void * CM_NULLABLEsourceFrameRefCon,
OSStatus status,
VTDecodeInfoFlags infoFlags,
CM_NULLABLE CVImageBufferRef imageBuffer,
CMTime presentationTimeStamp,
CMTime presentationDuration );
Copy the code
- (void) setupAudioConverter: CMSampleBufferRef sampleBuffer {/ / by real-time decoding data to obtain AudioStreamBasicDescription input information inputDescription = *CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer)); / / define the PCM output information AudioStreamBasicDescription outputDescription = {, mSampleRate = 44100, mFormatID = kAudioFormatMPEG4AAC, .mFormatFlags = kMPEG4Object_AAC_LC, .mBytesPerPacket = 0, .mFramesPerPacket = 1024, .mBytesPerFrame = 0, .mChannelsPerFrame = 1, .mBitsPerChannel = 0, .mReserved = 0 }; OSStatus status = AudioConverterNew(&inputDescription, &outputDescription, &_AudioConverter);if(status ! = noErr) { NSLog(@"error");
return; } // Set output quality UInt32 Temp = kAudioConverterQuality_High; AudioConverterSetProperty(_audioConverter, kAudioConverterCodecQuality, sizeof(temp), &temp); // Set bitRate UInt32 bitRate = 96000; AudioConverterSetProperty(_audioConverter, kAudioConverterEncodeBitRate, sizeof(bitRate), &bitRate); }Copy the code
The input information is first obtained through real-time audio data, then the PCM output information is defined, and then the decoder is created through these two parameters. Set some decoder properties after successful creation.
Audio decoding is different from video decoding. The first step is to get the PCM data and store it in a global variable, and then create an AudioBufferList to receive the data. The decoding works in the decoding callback. The pointer, and create a callback AudioBufferList incoming AudioConverterFillComplexBuffer, will continue to decode data just the way they are.
The encoding callback does the work of populating the PCM data so that it can be encoded
Decoding part
Initialize the
- (void) setupEncoder {/ / output parameters PCM AudioStreamBasicDescription outputAudioDes = {0}; outputAudioDes.mSampleRate = (Float64)self.sampleRate; / / sampling rate outputAudioDes mChannelsPerFrame = (UInt32) self. ChannelCount; // Outputauds35. mFormatID = kAudioFormatLinearPCM; / / output format outputAudioDes. MFormatFlags = (kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked); / / code 12 outputAudioDes. MFramesPerPacket = 1; // Number of frames per packet; outputAudioDes.mBitsPerChannel = 16; // The number of bits sampled for each channel in the data frame. outputAudioDes.mBytesPerFrame = outputAudioDes.mBitsPerChannel / 8 *outputAudioDes.mChannelsPerFrame; / / each frame size (sampling) the digits / 8 * the number of channels outputAudioDes. MBytesPerPacket = outputAudioDes. MBytesPerFrame * outputAudioDes mFramesPerPacket; // Each packet size (frame size * frame number) outputauds35s.mreserved = 0; / / the way 0 (8 byte alignment) / / input parameters aac AudioStreamBasicDescription inputAduioDes = {0}; inputAduioDes.mSampleRate = (Float64)self.sampleRate; inputAduioDes.mFormatID = kAudioFormatMPEG4AAC; inputAduioDes.mFormatFlags = kMPEG4Object_AAC_LC; inputAduioDes.mFramesPerPacket = 1024; inputAduioDes.mChannelsPerFrame = (UInt32)self.channelCount; // Input information UInt32inDesSize = sizeof(inputAduioDes);
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &inDesSize, &inputAduioDes);
OSStatus status = AudioConverterNew(&inputAduioDes, &outputAudioDes, &_audioDecodeConverter);
if(status ! = noErr) { NSLog(@"Error! : hard decoding AAC creation failed, status= %d", (int)status);
return; }}Copy the code
The initialization of a decoder is similar to that of an encoder, but with more elaboration
The decoding function
- (void)decodeAudioAACData:(NSData *)aacData {
if(! _audioDecodeConverter) { [self setupEncoder]; } dispatch_async(self.decodeQueue, ^{CCAudioUserData userData = {0}; userData.channelCount = (UInt32)self.channelCount; userData.data = (char *)[aacData bytes]; userData.size = (UInt32)aacData.length; userData.packetDesc.mDataByteSize = (UInt32)aacData.length; userData.packetDesc.mStartOffset = 0; userData.packetDesc.mVariableFramesInPacket = 0; // Output size and Number of packets UInt32 pcmBufferSize = (UInt32)(2048 * self.channelcount); UInt32 pcmDataPacketSize = 1024; PCM uint8_t *pcmBuffer = malloc(pcmBufferSize); memset(pcmBuffer, 0, pcmBufferSize); AudioBufferList outAudioBufferList = {0}; outAudioBufferList.mNumberBuffers = 1; outAudioBufferList.mBuffers[0].mNumberChannels = (uint32_t)self.channelCount; outAudioBufferList.mBuffers[0].mDataByteSize = (UInt32)pcmBufferSize; outAudioBufferList.mBuffers[0].mData = pcmBuffer; / / output describe AudioStreamPacketDescription outputPacketDesc = {0}; // Configure the fill function, To obtain the output data OSStatus status = AudioConverterFillComplexBuffer (self - > _audioDecodeConverter, AudioDecoderConverterComplexInputDataProc, &userData, &pcmDataPacketSize, &outAudioBufferList, &outputPacketDesc);if(status ! = noErr) { NSLog(@"Error: AAC Decoder error, status=%d",(int)status);
return; } // If data is retrievedif(outAudioBufferList.mBuffers[0].mDataByteSize > 0) { NSData *rawData = [NSData dataWithBytes:outAudioBufferList.mBuffers[0].mData length:outAudioBufferList.mBuffers[0].mDataByteSize]; Dispatch_async (self) decodeCallbackQueue, ^ {/ / here can handle decoding of PCM data NSLog (@"% @",rawData);
});
}
free(pcmBuffer);
});
}
Copy the code
Audio decoding function and coding function are AudioConverterFillComplexBuffer, parameter is the same, the only difference is that the output data is PCM data after decoding
