Live on video calls or scenario, if the number of communications, equipment performance and downlink bandwidth consumed will continue to rise, will bring the great degree communication card, drops, black screen problem, how to call API is optimized in this paper, the scene narrative, content with the iOS interface as an example, the other end can find replacement in the center of the document.
The lazy method
In the process of many audio and video communications, equipment of network traffic will be indirect fluctuations (sudden traffic, the network signal problem), influence call quality, quality index of the developer needs to talk to the current do statistical analysis, through the analysis results, to call the API to solve the problem, but frequent switch or improper API call can lead to abnormal phenomenon such as. AnyRTC starts from the reality and analyzes the audio and video quality of subscribers according to multiple internal indicators, and develops the automatic audio and video stream backoff function. Users simply call methods, listen for status callbacks, and give friendly reminders in the video window.
The premise
Call the enableDualStreamMode method to enable dual-stream mode before using the audio and video stream rollback function. For each of the following modes, dual-flow mode should be enabled in advance (small window pulls small flow and large window pulls flow to save traffic).
Audio and video stream rollback method
- (int)setRemoteSubscribeFallbackOption:(ARStreamFallbackOptions)option;
Copy the codeAfter joining the channel, call this method. ARStreamFallbackOptions has three options:
ARStreamFallbackOptionDisabled: Does not use the audio and video stream rollback functionARStreamFallbackOptionVideoStreamLow: In the case of poor downlink network conditions, the SDK will receive small video streams (low resolution, low bit rate video streams).ARStreamFallbackOptionAudioOnly: When the downlink network is weak, try to receive only small video streams (low resolution and low bit rate video streams) first. If the network cannot display videos, switch back to receiving only audio streams
Audio and video flow back to callback
-(void)rtcEngine:(ARtcEngineKit * _Nonnull)engine didRemoteSubscribeFallbackToAudioOnly:(BOOL)isFallbackOrRecover byUid:(NSString *_Nonnull)uid;
Copy the codeAfter the rollback function of the remote audio and video stream is invoked, you only need to listen to the corresponding callback and use the callback information to prompt the call window.
The callback information is distinguished from the audio and video switch status of the peer party. The audio and video switch status is whether the peer party enables or disables the audio and video streaming. The callback information is whether the local accepts the user’s video.
The meeting model
The configuration before the meeting is displayed
Before entering a meeting, users can enter the meeting based on their preferences. This configuration item effectively protects users’ privacy. If you deselect cameras and microphones at the same time, the traffic in a meeting is reduced.
A method is called
| methods | instructions | Call Precautions |
|---|---|---|
| – (int)muteLocalAudioStream:(BOOL)mute; | Enables local audio transmission | Does not send its own audio data in the channel |
| – (int)muteLocalVideoStream:(BOOL)mute; | Enable local video sending | Does not send its own video data in the channel |
| – (int) enableDualStreamMode (BOOL) enabled; | Switch video dual-stream mode | Dual-stream mode is recommended for multi-party calls |
| – (int)setRemoteVideoStream:(NSString *_Nonnull)uid type:(ARVideoStreamType)streamType; | Sets the type of video stream to subscribe to | Set the video stream type, subscribe to stream on large screen, and subscribe to small screen on small screen |
| – (int)setRemoteDefaultVideoStreamType:(ARVideoStreamType)streamType; | Sets the type of video stream to subscribe to by default | Before entering the meeting, set the subscription of all stream types, in the page switching with the subscription stream type to do dynamic switching |
The callback method
| methods | instructions | Call Precautions |
|---|---|---|
| – (void)rtcEngine:(ARtcEngineKit *_Nonnull)engine remoteVideoStateChangedOfUid:(NSString *_Nonnull)uid state:(ARVideoRemoteState)state reason:(ARVideoRemoteStateReason)reason elapsed:(NSInteger)elapsed; | Description The remote video status changed | Monitor the video status of the other party and make corresponding prompts on the page |
| – (void)rtcEngine:(ARtcEngineKit * _Nonnull)engine remoteAudioStateChangedOfUid:(NSString *_Nonnull)uid state:(ARAudioRemoteState)state reason:(ARAudioRemoteStateReason)reason elapsed:(NSInteger)elapsed; | A remote audio stream status change callback | Monitor the audio status of the other party and make corresponding prompts on the page |
The method can be called before or after entering the channel. After this method is called, the local collection will continue, but the audio and video data will not be transmitted. When sending is enabled, the speed is relatively fast, so it is recommended to use this method for audio and video switch operation.
As the video collection is continuing, the picture can be previewed, a mask can be made on the top of the picture to cover the video picture, or a status map can be made to identify the local video transmission state.
Voice stimulation mode + lock screen
To save bandwidth, voice excitation mode is the best mode, showing only one line of video, that is, the user who is currently speaking loudest, and everyone else only receiving audio. In the actual development process, the developer can also manually take over the screen. When the user is motivated by voice, the video image of the user will be displayed in a large screen, and the button “lock” will be added to the page. At this time, only the user image can be viewed, which meets the requirement of multi-person communication with small bandwidth.
A method is called
| methods | instructions | Call Precautions |
|---|---|---|
| – (int)enableAudioVolumeIndication:(NSInteger)interval smooth:(NSInteger)smooth report_vad:(BOOL)report_vad; | Enable speaker volume prompt | After enabled, the volume of listening anchor returns to |
| – (int)muteRemoteVideoStream:(NSString *_Nonnull)uid mute:(BOOL)mute; | Stops/resumes receiving the specified video stream | When the received video stream is not the current excited user, the receiving is suspended first, and when the detection is excited user, the method is called to receive |
The callback method
In addition to listening for the speaker’s voice callback, the local call also monitors the release status of the video of the other party. When receiving the video of the user, the local call needs to determine whether the user has released the video. If the user has not released the video, the receiving method is not set.
| methods | instructions | Call Precautions |
|---|---|---|
| – (void)rtcEngine:(ARtcEngineKit * _Nonnull)engine reportAudioVolumeIndicationOfSpeakers:(NSArray<ARtcAudioVolumeInfo *> * _Nonnull)speakers totalVolume:(NSInteger)totalVolume; | Callbacks that indicate who is talking in the channel, the volume of the speaker, and whether the local user is talking | Please enable the speaker volume prompt method first |
| – (void)rtcEngine:(ARtcEngineKit * _Nonnull)engine didVideoSubscribeStateChange:(NSString *_Nonnull)channel withUid:(NSString * _Nonnull)uid oldState:(ARStreamSubscribeState)oldState newState:(ARStreamSubscribeState)newState elapseSinceLastState:(NSInteger)elapseSinceLastState; | Video subscription status change callback | When subscribing to the user’s video stream, the page can be switched according to the current subscription status |
The difficulties in
- When using voice stimulation, ensure that you have received the user’s screen before switching to avoid a black screen after switching.
- Voice stimulation should not change views frequently, and there should be a certain buffer time to avoid visual fatigue caused by frequent screen switching.
Paging display – Gallery mode
When there are too many calls, it is also an optimization method to display video data in pages. Traditionally, four screens are displayed on each page. When there are too many calls, video data is displayed in pages.
The whole process
Setting before joining a room :(before joinChannel)
1. EnableDualStreamMode Enable dual-stream
2, setRemoteDefaultVideoStreamType receiving mode is set to the default, streams
3. MuteLocalAudioStream and muteLocalVideoStream are optional, depending on the Settings before the meeting
4, call private method: user quality report
NSDictionary *quality = [[NSDictionary alloc] initWithObjectsAndKeys:@"UserQuality", @"Cmd",[NSNumber numberWithBool:YES], @"Enable",nil];
[_rtcKit setParameters:[ARCommon returnJSONStringWithDictionary:quality]];
Copy the codeOperation after adding room
1. Listen for didJoinedOfUid, callback to insert the user into the local array, and set the first 3 users to do nothing, and subsequent users to call muteRemoteVideoStream method to not receive subsequent users’ video stream, so that one screen plus oneself and three other users is exactly one screen of data. When the number of people increases, 4 pages per page will be displayed according to logic.
2, listen to didOfflineOfUid, there is a callback to the local array to delete the user, while updating the display screen, if the user in the display screen, need to remove the screen, the data behind the forward fill, fill the user call muteRemoteVideoStream to pull the user’s video stream; If the user is not in the displayed screen, do not operate in the back of the array that displays the current window, just delete the user in the array. If the user is online in front of the array of the window, it is necessary to move the current screen, and the user in front of the display page moves forward. And muteRemoteVideoStream is not subscribed to the user’s video stream, the user in the array after the current page is in the online area, and muteRemoteVideoStream is subscribed to the user’s video stream.
3. Page-turning: When page-turning, users who turn to the area displayed on the page call muteRemoteVideoStream to subscribe to the video stream, and users who turn away call muteRemoteVideoStream to cancel the subscription to the video stream.
4, audio and video status: the client to monitor remoteVideoStateChangedOfUid and remoteAudioStateChangedOfUid methods do prompt state of audio and video of the current page.
5, listen to networkQuality callback, most users on each screen do good or bad network prompt.
The difficulties in
- People joining and leaving in paging scenarios: People joining and leaving trigger a change in the number of pages and a change in the layout of users on the current page
- Staff subscription status and audio and video status: staff status maintenance, users will dynamically switch on and off the local audio and video stream, the page to do the corresponding prompt
conclusion
The gameplay of video conferencing is basically the scenario described above, and the communication problems caused by traffic and equipment performance problems are broken through some of the methods described above. With the development of technology and the improvement of equipment performance, super resolution, AI codec and other technologies will also be integrated into multi-person video conferencing. Of course, with the popularization of 5G and the application of 6G, traffic problems will also be breached by infrastructure.
