Introduction: Due to the need to use screen sharing in the recent project, so the iOS screen sharing research, here also share the road to step on the pit.

Introduction to Screen Sharing

Screen sharing is the sharing of content on a screen for others to view, which is divided into two key technologies:

  • Screen content collection: related to the system permissions, as well as the need for the system to provide the SCREEN content collection Api; (Screen recording frame provided by AppleReplayKit)
  • Streaming media service: the collected video and audio streams are pushed to the streaming media server and broadcast to users; (Usually in the form of RTC push flow processing)

Application scenarios of screen sharing: mobile game live broadcasting, customer service guidance, business meetings, teaching whiteboard, etc.

The overall process of screen sharing is as follows:

  • Trigger recording screen
  • The preparatory work
  • Began to record the screen
  • Processing data streams (audio, video, etc.)
  • Share ExtensionApp data to MainApp (or operate on it directly)
  • End to record screen

Screen sharing collection:

  • Add a Target
  • Create a Broadcast Upload Extension

  • Add AppGroups

You can apply for AppGroupID on the Official website of Apple developers, and associate the relevant profile file and certificate. You can also add it in Xcode, which will be automatically generated by Xcode. AppGroups need to be added in both the main App and the extension.

  • After adding

There will be two files in the final project, and AppGroupID should be the same;

This is where you can start working on screen sharing. Since it involves the communication between processes (between the main App and the extended App), the notification method is used to deal with the start and end events. Currently, NSUserDefault and Socket are used to communicate with the screen sharing data acquisition process. However, apple extended App has a 50M memory limit first. If the frame number is not too high, NSUserDefault can be used to transmit sampleBuffer. If the screen sharing requirements are high (high frame rate, high resolution), you can directly connect to the stream in the extension App, or use sockets. Using socket can be independent of memory to a certain extent, but need to deal with the memory explosion caused by frame accumulation, can avoid the extension program is forced to KILL by the system, but the socket has a certain instability, need to deal with disconnection and network exceptions and other problems;

Recommended process communication methods:

  • Event Notification:CFNotification;
  • Simple value passing:NSUserDefault;
  • Complex data transfer:Socket;

Start Screen Sharing

This method has uncertainty, 🤷 do not know whether one day in the future can be used, can use it first! Apple does not provide a clear method, only one control can call up the screen sharing popup, so here is a clever way to encapsulate the method, call App; After the screen sharing is switched on, there is still a step of user confirmation until we actually start the screen sharing. Therefore, we need to know the event callback of the screen sharing extender before processing some logic, so as to truly form a closed loop.

/ / system popup window - (RPSystemBroadcastPickerView *) systemPicker {if (! _systemPicker) { RPSystemBroadcastPickerView* picker = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 0, 44, 44)]; picker.showsMicrophoneButton = NO; picker.autoresizingMask = UIViewAutoresizingFlexibleTopMargin | UIViewAutoresizingFlexibleRightMargin; _systemPicker = picker; } return _systemPicker; } - (void) launchBroadcaster API_AVAILABLE (ios (12.0)) {NSArray * contents = [NSFileManager defaultManager contentsOfDirectoryAtPath:plugInPath error:nil]; for (NSString *content in contents) { NSURL *url = [NSURL fileURLWithPath:plugInPath]; NSBundle *bundle = [NSBundle bundleWithPath:[url URLByAppendingPathComponent:content].path]; NSDictionary *extension = [bundle.infoDictionary objectForKey:@"NSExtension"]; if (extension == nil) { continue; } NSString *identifier = [extension objectForKey:@"NSExtensionPointIdentifier"]; if ([identifier isEqualToString:@"com.apple.broadcast-services-upload"]) { self.systemPicker.preferredExtension = bundle.bundleIdentifier; break; } } for (UIView *view in self.systemPicker.subviews) { UIButton *button = (UIButton *)view; [button sendActionsForControlEvents:UIControlEventAllTouchEvents]; }}Copy the code

Stopping screen Sharing

The stop screen sharing method is provided by the ReplayKit extension. It cannot be called directly from MainApp. You can call the ExtensionApp method by notification.

// MainApp sends a notification to ExtensionApp, // MainApp - (void)stopBroadcaster {CFNotificationName notificationName = CFNotificationName(TScreenShareHostRequestStopNotification); CFNotificationCenterPostNotification(CFNotificationCenterGetDarwinNotifyCenter(), notificationName, nil, nil, true); } / / ExtensionApp / / listen to inform CFNotificationCenterAddObserver (CFNotificationCenterGetDarwinNotifyCenter (), (__bridge const void *)(self), onHostRequestFinishBroadcast, (__bridge CFStringRef)ScreenShareHostRequestStopNotification, NULL, CFNotificationSuspensionBehaviorDeliverImmediately);  static void onHostRequestFinishBroadcast(CFNotificationCenterRef center, void *observer, CFStringRef name, const void *object, CFDictionaryRef userInfo) { ScreenShareSampleHandler *self = (__bridge ScreenShareSampleHandler *)(observer); NSError *error = [NSError errorWithDomain:NSStringFromClass(self.class) code:0 userInfo:@{ NSLocalizedFailureReasonErrorKey: NSLocalizedString (@ "you have to stop the screen sharing.," nil)}]; [self finishBroadcastWithError:error]; }Copy the code

ExtensionApp event callback

- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo { // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional. CFNotificationName notificationName = CFNotificationName(ScreenShareBroadcastStartedNotification); CFNotificationCenterPostNotification(CFNotificationCenterGetDarwinNotifyCenter(), notificationName, nil, nil, true); } - (void)broadcastPaused { // User has requested to pause the broadcast. Samples will stop being delivered. CFNotificationName notificationName = CFNotificationName(ScreenShareBroadcastPausedNotification); CFNotificationCenterPostNotification(CFNotificationCenterGetDarwinNotifyCenter(), notificationName, nil, nil, true); } - (void)broadcastResumed { // User has requested to resume the broadcast. Samples delivery will resume. CFNotificationName notificationName = CFNotificationName(ScreenShareBroadcastResumedNotification); CFNotificationCenterPostNotification(CFNotificationCenterGetDarwinNotifyCenter(), notificationName, nil, nil, true); } - (void)broadcastFinished { // User has requested to finish the broadcast. CFNotificationName notificationName = CFNotificationName(ScreenShareBroadcastFinishedNotification); CFNotificationCenterPostNotification(CFNotificationCenterGetDarwinNotifyCenter(), notificationName, nil, nil, true); } - (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType { // Use NSUserDefault}Copy the code

MainApp listens for notifications of screen sharing events

/ / screen sharing begin CFNotificationCenterAddObserver (CFNotificationCenterGetDarwinNotifyCenter (), (__bridge const void *) (self), onBroadcastStarted, (__bridge CFStringRef)TScreenShareBroadcastStartedNotification, NULL, CFNotificationSuspensionBehaviorDeliverImmediately); / / screen sharing complete CFNotificationCenterAddObserver (CFNotificationCenterGetDarwinNotifyCenter (), (__bridge const void *) (self), onBroadcastFinished, (__bridge CFStringRef)TScreenShareBroadcastFinishedNotification, NULL, CFNotificationSuspensionBehaviorDeliverImmediately); / / screen sharing suspended CFNotificationCenterAddObserver (CFNotificationCenterGetDarwinNotifyCenter (), (__bridge const void *) (self), onBroadcastPaused, (__bridge CFStringRef)TScreenShareBroadcastPausedNotification, NULL, CFNotificationSuspensionBehaviorDeliverImmediately); / / screen sharing suspended CFNotificationCenterAddObserver (CFNotificationCenterGetDarwinNotifyCenter (), (__bridge const void *) (self), onBroadcastResumed, (__bridge CFStringRef)TScreenShareBroadcastResumedNotification, NULL, CFNotificationSuspensionBehaviorDeliverImmediately); Static void onBroadcastStarted(CFNotificationCenterRef Center, void * Observer, CFStringRef name, const void *object, CFDictionaryRef userInfo) { } static void onBroadcastFinished(CFNotificationCenterRef center, void *observer, CFStringRef name, const void *object, CFDictionaryRef userInfo) { } static void onBroadcastPaused(CFNotificationCenterRef center, void *observer, CFStringRef name, const void *object, CFDictionaryRef userInfo) { } static void onBroadcastResumed(CFNotificationCenterRef center, void *observer, CFStringRef name, const void *object, CFDictionaryRef userInfo) { }Copy the code

Streaming services:

Most of these use third-party services, and the approach of each SDK is pretty much the same; RTC screen sharing methods include start screen sharing, stop screen sharing, and push stream. There are basically dumb tutorials on integration, so I won’t go into them here;

Conclusion:

IOS is too convoluted for screen sharing, and ReplayKit as a whole isn’t very developer-friendly. Compatibility between iOS versions is also a headache. There are still many details to realize screen sharing, and processing after screen sharing is particularly critical, such as processing of video frames and audio frames; There’s also the 50 MB memory limit for the screen sharing extension, which is something developers need to be careful about;