Replaykit introduction

In previous versions of iOS, iOS developers could only get the encoded data, not the original PCM and YUV. After iOS 10, developers could get the original data, but they could only record the content within the App. If they switched to the background, the recording would stop until iOS 11. Apple has upgraded screensharing and given access to both raw data and recordings of the entire system. Here’s what we’ll focus on after iOS 11.

System screen sharing

- (void)initMode_1 { self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)]; self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder"; Self. SystemBroadcastPickerView. BackgroundColor = [UIColor colorWithRed: 53.0/255.0 green: 129.0/255.0 blue: 242.0/255.0 Alpha: 1.0]; self.systemBroadcastPickerView.showsMicrophoneButton = NO; [self.view addSubview:self.systemBroadcastPickerView]; }Copy the code

After creating an Extension in iOS 11, call the above code to start screen sharing, and the system will generate a SampleHandler class for us. In this method, Apple will report different types of data based on RPSampleBufferType.

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType
Copy the code

So how do you send the screen-sharing data through RongRTCLib?

1. Socket-based force play

1.1. The Replaykit framework starts and creates sockets

//// viewController. m// Socket_Replykit//// Created by Sun on 2020/5/19.// Copyright © 2020 Rongcloud. All Rights reserved.//#import "ViewController.h"#import <ReplayKit/ReplayKit.h>#import "RongRTCServerSocket.h"@interface ViewController ()<RongRTCServerSocketProtocol>@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView; /** server socket */@property(nonatomic , strong)RongRTCServerSocket *serverSocket; @end@implementation ViewController- (void)viewDidLoad { [super viewDidLoad]; self.view.backgroundColor = [UIColor whiteColor]; // Do any additional setup after loading the view. [self.serverSocket createServerSocket]; self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)]; self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP"; Self. SystemBroadcastPickerView. BackgroundColor = [UIColor colorWithRed: 53.0/255.0 green: 129.0/255.0 blue: 242.0/255.0 Alpha: 1.0]; self.systemBroadcastPickerView.showsMicrophoneButton = NO; [self.view addSubview:self.systemBroadcastPickerView]; }- (RongRTCServerSocket *)serverSocket { if (! _serverSocket) { RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init]; socket.delegate = self; _serverSocket = socket; } return _serverSocket; }- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer {// this is where you get the final data, so you can use the audio and video SDK RTCLib}@endCopy the code

We use the main App as a Server and the screen sharing Extension as a Client to send data to our main App through the Socket.

In Extension, after we get the screen video data reported by ReplayKit framework:

//// sampleHandler. m// SocketReply//// Created by Sun on 2020/5/19.// Copyright © 2020 Rongcloud. All Rights reserved.//#import "SampleHandler.h"#import "RongRTCClientSocket.h"@interface SampleHandler()/** Client Socket */@property (nonatomic, strong) RongRTCClientSocket *clientSocket; @end@implementation SampleHandler- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo { // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional. self.clientSocket = [[RongRTCClientSocket alloc] init]; [self.clientSocket createCliectSocket]; }- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType { switch (sampleBufferType) { case RPSampleBufferTypeVideo: // Handle video sample buffer [self sendData:sampleBuffer]; break; case RPSampleBufferTypeAudioApp: // Handle audio sample buffer for app audio break; case RPSampleBufferTypeAudioMic: // Handle audio sample buffer for mic audio break; default: break; }}- (void)sendData:(CMSampleBufferRef)sampleBuffer { [self.clientSocket encodeBuffer:sampleBuffer]; }@endCopy the code

So here we create a Client Socket, and then we get the sampleBuffer of the screen sharing video, and we send it to our main App through the Socket, and that’s the process of screen sharing.

1.2 Using the Local Socket

//// rongrtcsocket. m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 Rongcloud. All Rights reserved.//#import "RongRTCSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import "RongRTCThread.h"@interface RongRTCSocket()/** receive thread */@property (nonatomic, strong) RongRTCThread *receiveThread; @end@implementation RongRTCSocket- (int)createSocket { int socket = socket(AF_INET, SOCK_STREAM, 0); self.socket = socket; if (self.socket == -1) { close(self.socket); NSLog(@"socket error : %d", self.socket); } self.receiveThread = [[RongRTCThread alloc] init]; [self.receiveThread run]; return socket; }- (void)setSendBuffer { int optVal = 1024 * 1024 * 2; int optLen = sizeof(int); int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen); NSLog(@"set send buffer:%d", res); }- (void)setReceiveBuffer { int optVal = 1024 * 1024 * 2; int optLen = sizeof(int); int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen ); NSLog(@"set send buffer:%d",res); }- (void)setSendingTimeout {struct timeval timeout = {10,0}; int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int)); NSLog(@"set send timeout:%d", res); }- (void)setReceiveTimeout { struct timeval timeout = {10, 0}; int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int)); NSLog(@"set send timeout:%d", res); }- (BOOL)connect { NSString *serverHost = [self ip]; struct hostent *server = gethostbyname([serverHost UTF8String]); if (server == NULL) { close(self.socket); NSLog(@"get host error"); return NO; } struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0]; struct sockaddr_in addr; addr.sin_family = AF_INET; addr.sin_addr = *remoteAddr; addr.sin_port = htons(CONNECTPORT); int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr)); if (res == -1) { close(self.socket); NSLog(@"connect error"); return NO; } NSLog(@"socket connect to server success"); return YES; }- (BOOL)bind { struct sockaddr_in client; client.sin_family = AF_INET; NSString *ipStr = [self ip]; if (ipStr.length <= 0) { return NO; } const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding]; client.sin_addr.s_addr = inet_addr(ip); client.sin_port = htons(CONNECTPORT); int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client)); if (bd == -1) { close(self.socket); NSLog(@"bind error: %d", bd); return NO; } return YES; }- (BOOL)listen { int ls = listen(self.socket, 128); if (ls == -1) { close(self.socket); NSLog(@"listen error: %d", ls); return NO; } return YES; }- (void)receive { dispatch_async(dispatch_get_global_queue(0, 0), ^{ [self receiveData]; }); }- (NSString *)ip { NSString *ip = nil; struct ifaddrs *addrs = NULL; struct ifaddrs *tmpAddrs = NULL; BOOL res = getifaddrs(&addrs); if (res == 0) { tmpAddrs = addrs; while (tmpAddrs ! = NULL) { if (tmpAddrs->ifa_addr->sa_family == AF_INET) { // Check if interface is en0 which is the wifi connection on the iPhone NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]); if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) { // Get NSString from C String ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]; } } tmpAddrs = tmpAddrs->ifa_next; } } // Free memory freeifaddrs(addrs); NSLog(@"%@",ip); return ip; }- (void)close { int res = close(self.socket); NSLog(@"shut down: %d", res); }- (void)receiveData {}- (void)dealloc { [self.receiveThread stop]; }@endCopy the code

Firstly, create a Socket parent class, and then use Server Socket and Client Socket respectively inheritance class to achieve the link, binding and other operations. You can see that some of the data can be set, some of the data can’t be set, this is not the core, the core is how to send and receive data.

1.3 Sending Screen Share Data

//// rongrtcclientSocket. m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 Rongcloud.all Rights reserved.//#import "RongRTCClientSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import "RongRTCThread.h"#import "RongRTCSocketHeader.h"#import "RongRTCVideoEncoder.h"@interface RongRTCClientSocket() <RongRTCCodecProtocol> { pthread_mutex_t lock; }/** video encoder */@property (nonatomic, strong) RongRTCVideoEncoder *encoder; /** encode queue */@property (nonatomic, strong) dispatch_queue_t encodeQueue; @end@implementation RongRTCClientSocket- (BOOL)createClientSocket { if ([self createSocket] == -1) { return NO; } BOOL isC = [self connect]; [self setSendBuffer]; [self setSendingTimeout]; if (isC) { _encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL); [self createVideoEncoder]; return YES; } else { return NO; }}- (void)createVideoEncoder { self.encoder = [[RongRTCVideoEncoder alloc] init]; self.encoder.delegate = self; RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init]; settings.width = 720; settings.height = 1280; settings.startBitrate = 300; settings.maxFramerate = 30; settings.minBitrate = 1000; [self.encoder configWithSettings:settings onQueue:_encodeQueue]; }- (void)clientSend:(NSData *)data { //data length NSUInteger dataLength = data.length; // data header struct DataHeader dataH; memset((void *)&dataH, 0, sizeof(dataH)); // pre PreHeader preH; memset((void *)&preH, 0, sizeof(preH)); preH.pre[0] = '&'; preH.dataLength = dataLength; dataH.preH = preH; // buffer int headerlength = sizeof(dataH); int totalLength = dataLength + headerlength; // srcbuffer Byte *src = (Byte *)[data bytes]; // send buffer char *buffer = (char *)malloc(totalLength * sizeof(char)); memcpy(buffer, &dataH, headerlength); memcpy(buffer + headerlength, src, dataLength); // tosend [self sendBytes:buffer length:totalLength]; free(buffer); }- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer { [self.encoder encode:sampleBuffer]; }- (void)sendBytes:(char *)bytes length:(int)length { LOCK(self->lock); int hasSendLength = 0; while (hasSendLength < length) { // connect socket success if (self.socket > 0) { // send int sendRes = send(self.socket, bytes, length - hasSendLength, 0); if (sendRes == -1 || sendRes == 0) { UNLOCK(self->lock); NSLog(@"send buffer error"); [self close]; break; } hasSendLength += sendRes; bytes += sendRes; } else { NSLog(@"client socket connect error"); UNLOCK(self->lock); } } UNLOCK(self->lock); }- (void)spsData:(NSData *)sps ppsData:(NSData *)pps { [self clientSend:sps]; [self clientSend:pps]; }- (void)naluData:(NSData *)naluData { [self clientSend:naluData]; }- (void)deallo c{ NSLog(@"dealoc cliect socket"); }@endCopy the code

The idea here is that once you get the data from the screen share, compress it first, and then report it to the current class through a callback. It is sent to the main App via the clientSend method. A header is defined in the data sent to the main App. The header is added with a prefix and a length of bytes sent each time. The receiver can parse the packet after receiving it.

- (void)clientSend:(NSData *)data { //data length NSUInteger dataLength = data.length; // data header struct DataHeader dataH; memset((void *)&dataH, 0, sizeof(dataH)); // pre PreHeader preH; memset((void *)&preH, 0, sizeof(preH)); preH.pre[0] = '&'; preH.dataLength = dataLength; dataH.preH = preH; // buffer int headerlength = sizeof(dataH); int totalLength = dataLength + headerlength; // srcbuffer Byte *src = (Byte *)[data bytes]; // send buffer char *buffer = (char *)malloc(totalLength * sizeof(char)); memcpy(buffer, &dataH, headerlength); memcpy(buffer + headerlength, src, dataLength); // to send [self sendBytes:buffer length:totalLength]; free(buffer); }Copy the code

1.4 Receiving screen share data

//// rongrtcServerSocket. m// SealRTC//// Created by Sun on 2020/5/7.// Copyright © 2020 Rongcloud. All Rights reserved.//#import "RongRTCServerSocket.h"#import <arpa/inet.h>#import <netdb.h>#import <sys/types.h>#import <sys/socket.h>#import <ifaddrs.h>#import <UIKit/UIKit.h>#import "RongRTCThread.h"#import "RongRTCSocketHeader.h"#import "RongRTCVideoDecoder.h"@interface RongRTCServerSocket() <RongRTCCodecProtocol>{ pthread_mutex_t lock; int _frameTime; CMTime _lastPresentationTime; Float64 _currentMediaTime; Float64 _currentVideoTime; dispatch_queue_t _frameQueue; }@property (nonatomic, assign) int acceptSocket; /** data length */@property (nonatomic, assign) NSUInteger dataLength; /** timeData */@property (nonatomic, strong) NSData *timeData; /** decoder queue */@property (nonatomic, strong) dispatch_queue_t decoderQueue; /** decoder */@property (nonatomic, strong) RongRTCVideoDecoder *decoder; @end@implementation RongRTCServerSocket- (BOOL)createServerSocket { if ([self createSocket] == -1) { return NO; } [self setReceiveBuffer]; [self setReceiveTimeout]; BOOL isB = [self bind]; BOOL isL = [self listen]; if (isB && isL) { _decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL); _frameTime = 0; [self createDecoder]; [self receive]; return YES; } else { return NO; }}- (void)createDecoder { self.decoder = [[RongRTCVideoDecoder alloc] init]; self.decoder.delegate = self; RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init]; settings.width = 720; settings.height = 1280; settings.startBitrate = 300; settings.maxFramerate = 30; settings.minBitrate = 1000; [self.decoder configWithSettings:settings onQueue:_decoderQueue]; }- (void)receiveData { struct sockaddr_in rest; socklen_t rest_size = sizeof(struct sockaddr_in); self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size); while (self.acceptSocket ! = -1) { DataHeader dataH; memset(&dataH, 0, sizeof(dataH)); if (! [self receiveData:(char *)&dataH length:sizeof(dataH)]) { continue; } PreHeader preH = dataH.preH; char pre = preH.pre[0]; if (pre == '&') { // rongcloud socket NSUInteger dataLenght = preH.dataLength; char *buff = (char *)malloc(sizeof(char) * dataLenght); if ([self receiveData:(char *)buff length:dataLenght]) { NSData *data = [NSData dataWithBytes:buff length:dataLenght]; [self.decoder decode:data]; free(buff); } } else { NSLog(@"pre is not &"); return; } }}- (BOOL)receiveData:(char *)data length:(NSUInteger)length { LOCK(lock); int receiveLength = 0; while (receiveLength < length) { ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0); if (res == -1 || res == 0) { UNLOCK(lock); NSLog(@"receive data error"); break; } receiveLength += res; data += res; } UNLOCK(lock); return YES; }- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer { _frameTime += 1000; CMTime pts = CMTimeMake(_frameTime, 1000); CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts]; // Check to see if there is a problem with the decoded data. If the image appears, you are right. UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer]; [self.delegate didProcessSampleBuffer:sampleBuffer]; CFRelease(sampleBuffer); }- (void)close { int res = close(self.acceptSocket); self.acceptSocket = -1; NSLog(@"shut down server: %d", res); [super close]; }- (void)dealloc { NSLog(@"dealoc server socket"); }@endCopy the code

The main App continues to receive packets through the Socket, decodes the packets, and calls the decoded data back to the App layer through the didGetDecodeBuffer proxy method. The App layer can send the video data to the peer end through RongRTCLib’s sending custom stream method.

1.5 VideotoolBox hardcoding

////  RongRTCVideoEncoder.m//  SealRTC////  Created by Sun on 2020/5/13.//  Copyright © 2020 RongCloud. All rights reserved.//​#import "RongRTCVideoEncoder.h"​#import "helpers.h"​@interface RongRTCVideoEncoder() {    VTCompressionSessionRef _compressionSession;    int _frameTime;}​/** settings */@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;​/** callback queue */@property (nonatomic , strong ) dispatch_queue_t callbackQueue;​- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;​@end​void compressionOutputCallback(void *encoder,                               void *params,                               OSStatus status,                               VTEncodeInfoFlags infoFlags,                               CMSampleBufferRef sampleBuffer) {    RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder;    if (status != noErr) {        return;    }        if (infoFlags & kVTEncodeInfo_FrameDropped) {        return;    }        BOOL isKeyFrame = NO;    CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);        if (attachments != nullptr && CFArrayGetCount(attachments)) {        CFDictionaryRef attachment = static_cast<CFDictionaryRef>(CFArrayGetValueAtIndex(attachments, 0)) ;        isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync);    }        CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer);    CMBlockBufferRef contiguous_buffer = nullptr;        if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) {        status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer);        if (status != noErr) {            return;        }    } else {        contiguous_buffer = block_buffer;        CFRetain(contiguous_buffer);        block_buffer = nullptr;    }        size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer);    if (isKeyFrame) {        [videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer];    }        if (contiguous_buffer) {        CFRelease(contiguous_buffer);    }        [videoEncoder sendNaluData:sampleBuffer];}​​@implementation RongRTCVideoEncoder​@synthesize settings = _settings;@synthesize callbackQueue = _callbackQueue;​- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue {    self.settings = settings;    if (queue) {        _callbackQueue = queue;    } else {        _callbackQueue = dispatch_get_main_queue();    }        if ([self resetCompressionSession:settings]) {        _frameTime = 0;        return YES;    } else {        return NO;    }}​- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings {    [self destroyCompressionSession];    OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession);    if (status != noErr) {        return NO;    }        [self configureCompressionSession:settings];    return YES;}​- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings {    if (_compressionSession) {        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true);        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel);        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false);                SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10);        uint32_t targetBps = settings.startBitrate * 1000;        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps);        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate);        int bitRate = settings.width * settings.height * 3 * 4 * 4;        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate);        int bitRateLimit = settings.width * settings.height * 3 * 4;        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit);    }}​- (void)encode:(CMSampleBufferRef)sampleBuffer {    CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);    CMTime pts = CMTimeMake(self->_frameTime++, 1000);    VTEncodeInfoFlags flags;    OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession,                                                   imageBuffer,                                                   pts,                                                   kCMTimeInvalid,                                                   NULL, NULL, &flags);        if (res != noErr) {        NSLog(@"encode frame error:%d", (int)res);        VTCompressionSessionInvalidate(self->_compressionSession);        CFRelease(self->_compressionSession);        self->_compressionSession = NULL;        return;    }}​- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {    CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);    const uint8_t *sps ;    const uint8_t *pps;    size_t spsSize ,ppsSize , spsCount,ppsCount;    OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL);    OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL);    if (spsStatus == noErr && ppsStatus == noErr) {        const char bytes[] = "\x00\x00\x00\x01";        size_t length = (sizeof bytes) - 1;                NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize];        NSMutableData *ppsData  = [NSMutableData dataWithCapacity:4 + ppsSize];        [spsData appendBytes:bytes length:length];        [spsData appendBytes:sps length:spsSize];                [ppsData appendBytes:bytes length:length];        [ppsData appendBytes:pps length:ppsSize];        if (self && self.callbackQueue) {            dispatch_async(self.callbackQueue, ^{                if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) {                    [self.delegate spsData:spsData ppsData:ppsData];                }            });        }    } else {        NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus));    }}​- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer {    size_t totalLength = 0;    size_t lengthAtOffset=0;    char *dataPointer;    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);    OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer);        if (status1 != noErr) {        NSLog(@"video encoder error, status = %d", (int)status1);        return;    }        static const int h264HeaderLength = 4;    size_t bufferOffset = 0;        while (bufferOffset < totalLength - h264HeaderLength) {        uint32_t naluLength = 0;        memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength);        naluLength = CFSwapInt32BigToHost(naluLength);​        const char bytes[] = "\x00\x00\x00\x01";        NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength];        [naluData appendBytes:bytes length:4];        [naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength];                dispatch_async(self.callbackQueue, ^{            if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) {                [self.delegate naluData:naluData];            }        });                bufferOffset += naluLength + h264HeaderLength;    }}​- (void)destroyCompressionSession {    if (_compressionSession) {        VTCompressionSessionInvalidate(_compressionSession);        CFRelease(_compressionSession);        _compressionSession = nullptr;    }}​- (void)dealloc {    if (_compressionSession) {        VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid);        VTCompressionSessionInvalidate(_compressionSession);        CFRelease(_compressionSession);        _compressionSession = NULL;    }}@end
Copy the code

1.6 VideotoolBox decoding

////  RongRTCVideoDecoder.m//  SealRTC////  Created by Sun on 2020/5/14.//  Copyright © 2020 RongCloud. All rights reserved.//​#import "RongRTCVideoDecoder.h"#import <UIKit/UIKit.h>#import "helpers.h"​@interface RongRTCVideoDecoder() {    uint8_t *_sps;    NSUInteger _spsSize;    uint8_t *_pps;    NSUInteger _ppsSize;    CMVideoFormatDescriptionRef _videoFormatDescription;    VTDecompressionSessionRef _decompressionSession;}​/** settings */@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;​/** callback queue */@property (nonatomic, strong) dispatch_queue_t callbackQueue;​@end​void DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,                           void * CM_NULLABLE sourceFrameRefCon,                           OSStatus status,                           VTDecodeInfoFlags infoFlags,                           CM_NULLABLE CVImageBufferRef imageBuffer,                           CMTime presentationTimeStamp,                           CMTime presentationDuration ) {    if (status != noErr) {        NSLog(@" decoder callback error :%@", @(status));        return;    }        CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;    *outputPixelBuffer = CVPixelBufferRetain(imageBuffer);    RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon);    dispatch_async(decoder.callbackQueue, ^{        [decoder.delegate didGetDecodeBuffer:imageBuffer];        CVPixelBufferRelease(imageBuffer);    });}​@implementation RongRTCVideoDecoder​@synthesize settings = _settings;@synthesize callbackQueue = _callbackQueue;​- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue {    self.settings = settings;    if (queue) {        _callbackQueue = queue;    } else {        _callbackQueue = dispatch_get_main_queue();    }    return YES;}​- (BOOL)createVT {    if (_decompressionSession) {        return YES;    }        const uint8_t * const parameterSetPointers[2] = {_sps, _pps};    const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};    int naluHeaderLen = 4;    OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription );    if (status != noErr) {        NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status));        return false;    }        NSDictionary *destinationImageBufferAttributes =                                        @{                                            (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],                                            (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width],                                            (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height],                                            (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]                                        };                                            VTDecompressionOutputCallbackRecord CallBack;    CallBack.decompressionOutputCallback = DecoderOutputCallback;    CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self);    status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession);​    if (status != noErr) {        NSLog(@"VTDecompressionSessionCreate error:%@", @(status));        return false;    }        status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);    return YES;}​- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {    CVPixelBufferRef outputPixelBuffer = NULL;    CMBlockBufferRef blockBuffer = NULL;    CMBlockBufferFlags flag0 = 0;        OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);        if (status != kCMBlockBufferNoErr) {        NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status);        CFRelease(blockBuffer);        return outputPixelBuffer;    }        CMSampleBufferRef sampleBuffer = NULL;    const size_t sampleSizeArray[] = {frameSize};        status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);        if (status != noErr || !sampleBuffer) {        NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status);        CFRelease(blockBuffer);        return outputPixelBuffer;    }        VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;    VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;        status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);        if (status == kVTInvalidSessionErr) {        NSLog(@"decode frame error with session err status =%d", (int)status);        [self resetVT];    } else  {        if (status != noErr) {            NSLog(@"decode frame error with  status =%d", (int)status);        }    }​    CFRelease(sampleBuffer);    CFRelease(blockBuffer);        return outputPixelBuffer;}​- (void)resetVT {    [self destorySession];    [self createVT];}​- (void)decode:(NSData *)data {    uint8_t *frame = (uint8_t*)[data bytes];    uint32_t length = data.length;    uint32_t nalSize = (uint32_t)(length - 4);    uint32_t *pNalSize = (uint32_t *)frame;    *pNalSize = CFSwapInt32HostToBig(nalSize);        int type = (frame[4] & 0x1F);    CVPixelBufferRef pixelBuffer = NULL;    switch (type) {        case 0x05:            if ([self createVT]) {                pixelBuffer= [self decode:frame withSize:length];            }            break;        case 0x07:            self->_spsSize = length - 4;            self->_sps = (uint8_t *)malloc(self->_spsSize);            memcpy(self->_sps, &frame[4], self->_spsSize);            break;        case 0x08:            self->_ppsSize = length - 4;            self->_pps = (uint8_t *)malloc(self->_ppsSize);            memcpy(self->_pps, &frame[4], self->_ppsSize);            break;        default:            if ([self createVT]) {                pixelBuffer = [self decode:frame withSize:length];            }            break;    }}​- (void)dealloc {    [self destorySession];}​- (void)destorySession {    if (_decompressionSession) {        VTDecompressionSessionInvalidate(_decompressionSession);        CFRelease(_decompressionSession);        _decompressionSession = NULL;    }}​@end
Copy the code

1.7 tools

//// RongRTCBufferUtil. M // SealRTC//// Created by Sun on 2020/5/8.// Copyright © 2020 Rongcloud. All Rights //#import "rongrtcBufferutil. h"// Do not release the following methods, but release them outside. @implementation RongRTCBufferUtil+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer {CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer); CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer]; CIContext *temporaryContext = [CIContext contextWithOptions:nil]; CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))]; UIImage *image = [UIImage imageWithCGImage:videoImage]; CGImageRelease(videoImage); return image; }+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth { if (! image) return nil; float imageWidth = image.size.width; float imageHeight = image.size.height; float width = newImageWidth; float height = image.size.height/(image.size.width/width); float widthScale = imageWidth /width; float heightScale = imageHeight /height; UIGraphicsBeginImageContext(CGSizeMake(width, height)); if (widthScale > heightScale) { [image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)]; } else { [image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)]; } UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext(); UIGraphicsEndImageContext(); return newImage; }+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img { CGSize size = img.size; CGImageRef image = [img CGImage]; NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey, [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil]; CVPixelBufferRef pxbuffer = NULL; CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer); NSParameterAssert(status == kCVReturnSuccess && pxbuffer ! = NULL); CVPixelBufferLockBaseAddress(pxbuffer, 0); void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer); NSParameterAssert(pxdata ! = NULL); CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB(); CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst); NSParameterAssert(context); CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image); CGColorSpaceRelease(rgbColorSpace); CGContextRelease(context); CVPixelBufferUnlockBaseAddress(pxbuffer, 0); return pxbuffer; }+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time { CMSampleBufferRef sampleBuffer = NULL; / / for video information CMVideoFormatDescriptionRef videoInfo = NULL; OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo); CMTime currentTime = time; // CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid}; CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid}; result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer); CFRelease(videoInfo); return sampleBuffer; }+ (size_t)getCMTimeSize { size_t size = sizeof(CMTime); return size; }@endCopy the code

The implementation in this utility class is handled by the CPU, When you go CMSampleBufferRef to UIImage, UIImage to CVPixelBufferRef, CVPixelBufferRef to CMSampleBufferRef and crop the image, It is important to release the used objects in time, otherwise there will be a large memory leak.

2. Send video

2.1 Preparation

To use RongRTCLib of Rongyun, you need an AppKey, which can be obtained from the official website (www.rongcloud.cn/). After obtaining the token through AppKey, you can connect to the RTC room. This is the preparation stage for sending screen sharing.

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo { // User has requested to start Setup info from the UI extension can be supplied but optional. // Please fill in your AppKey self. AppKey = @""; // Please fill in the user Token self. Token = @""; Self. roomId = @"123456"; self.roomId = @"123456"; [[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey]; [[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose]; / / connections IM [[RCIMClient sharedRCIMClient] connectWithToken: self. The token dbOpened: ^ (RCDBErrorCode code) {NSLog (@ "dbOpened: %zd", code); } success:^(NSString *userId) { NSLog(@"connectWithToken success userId: [[RCRTCEngine sharedInstance] joinRoom: self.Roomid Completion :^(RCRTCRoom * _Nullable Room, RCRTCCode code) { self.room = room; self.room.delegate = self; [self publishScreenStream]; }]; } error:^(RCConnectErrorCode errorCode) { NSLog(@"ERROR status: %zd", errorCode); }]; }Copy the code

The above is the whole process of connecting IM and joining RTC room, which also includes calling self publishScreenStream; This method can be performed only after the room is successfully added.

- (void)publishScreenStream { RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init]; param.videoSizePreset = RongRTCVideoSizePreset1280x720; self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"]; [self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) {if (isSuccess) {NSLog(@" success in publishing custom stream ");}}]; }Copy the code

You can customize a RongRTCAVOutputStream and use it to send screen share data.

2.2 Screen share Data starts to be sent

Above, we have connected the cloud IM and joined the RTC room, and defined a custom stream to send screen share, how to publish this stream?

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {    switch (sampleBufferType) {        case RPSampleBufferTypeVideo:            // Handle video sample buffer            [self.videoOutputStream write:sampleBuffer error:nil];            break;        case RPSampleBufferTypeAudioApp:            // Handle audio sample buffer for app audio            break;        case RPSampleBufferTypeAudioMic:            // Handle audio sample buffer for mic audio            break;        default:            break;    }}
Copy the code

When we receive the data reported by Apple, we call the write: Error: method in RongRTCAVOutputStream to send the sampleBuffer to the remote end. At this point, the screen share data is sent.

[self.videoOutputStream write:sampleBuffer error:nil];
Copy the code

The core code of the cloud is to send a sampleBuffer via the above connection IM, join the room, publish the custom stream, and then send the sampleBuffer through the custom stream’s write:error: method.

Either ReplayKit is used to get the on-screen video, or Socket is used to transmit it between processes for the final write:error: service.

conclusion

1. The memory of Extension is limited to 50 MB at most. Therefore, extra attention should be paid to memory release when processing data in Extension.

2. If VideotoolBox keeps failing to decode in the background, just restart VideotoolBox, as shown in the code above.

3. If you don’t need to transfer the data of Extension to the main App, you just need to directly publish the flow through RongRTCLib in Extension. The disadvantage is that the user who publishes the custom flow in Extension is not the same as the user in the main App. This is the same problem that was solved above by passing data to the main App through the Socket;

4. If the main App needs to get the screen share data processing, use the Socket to first send the stream to the main App, and then send the stream from the main App via RongRTCLib.

Finally, the Demo is attached.