How to combine CallKit and Agora SDK to implement video VoIP call application

Gong Yuhua is the chief iOS R&D engineer of Agora. IO, responsible for the product design and technical architecture of iOS mobile applications.

CallKit is a system framework designed for VoIP calls in iOS10. It provides system-level support for VoIP calls on iOS.

Before iOS10, there were many limitations to the VoIP experience. For example, there is no special notification method for incoming calls. After receiving incoming calls in the background, the App can only use the general system notification method to prompt users. If users turn off notifications, they miss calls. VoIP calls themselves can be easily interrupted. For example, when a user opens another application using an audio device during a call, or receives a call from a carrier, the VoIP call will be interrupted.

To improve the user experience of VoIP calls, the CallKit framework improves VoIP calls to the same level as carrier calls at the system level. When the App receives an incoming call, it can register the VoIP call with the system through the CallKit, so that the system can use the same interface as the operator’s phone to prompt the user. During a call, the app’s audio and video rights become the same as those of the carrier’s phone without being interrupted by other apps. When receiving a VoIP call from carrier, users can determine whether to suspend or hang up the VoIP call.

In addition, VoIP calls that use the CallKit framework will also appear in the system’s call records, just like carrier calls. Users can initiate new VoIP calls in the address book and phone records.

Therefore, an application with VoIP call scenarios should integrate CallKit as soon as possible to greatly improve user experience and ease of use.

Let’s take a look at CallKit and integrate it into a video calling application using Agora SDK.

CallKit basic classes

The two most important CallKit classes are CXProvider and CXCallController. These two classes are at the heart of the CallKit framework.

CXProvider

The CXProvider controls call flow, registers calls with the system, and updates the connection status of calls. Important apis include the following:

1 Open class CXProvider: NSObject {2 /// Initialization method 3 Public init(Configuration: CXProviderConfiguration) 4 /// Set the callback object 5 open funcsetDelegate(_ delegate: CXProviderDelegate? , queue: DispatchQueue?) 6 /// Register a call with the system. 7 Open Func reportNewIncomingCall(with UUID: UUID, update: CXCallUpdate, completion: @escaping (Error?) -> swift.void) 9 Open func reportCall(with UUID: UUID, updated Update: CXCallUpdate) 11 Open func reportOutgoingCall(with UUID: UUID, startedConnectingAt dateStartedConnecting: Date?) 12 /// Open func reportOutgoingCall(with UUID: UUID, connectedAt dateConnected: Date?) 14 /// Tell the system the call ends 15 Open func reportCall(with UUID: UUID, endedAt dateEnded: Date? , reason endedReason: CXCallEndedReason) 16}Copy the code

As you can see, the CXProvider uses the UUID to identify a call and the CXCallUpdate class to set the properties of the call. Developers can create uuids for each call using a properly formatted string; You can also use the system-created UUID directly.

The user’s operations on the call are notified to the application through the callback method in the CXProviderDelegate.

CXCallController

The CXCallController performs call operations.

1open class CXCallController : NSObject {2 public convenience init() 4 /// You can use the callObserver to obtain the Uuid and call status of all ongoing calls. 5 Open var callObserver: CXCallObserver {get} 6 /// Perform operations on a call. 7 Open Func Request (_ Transaction: CXTransaction, completion: @escaping (Error?) -> Swift.Void) 8}Copy the code

CXTransaction is the encapsulation of an operation, including the CXAction and the call UUID. Actions such as initiating, answering, hanging, and mute a call have subclasses of CXAction.

Integrate with Agora SDK

Let’s take a look at how to integrate CallKit with a video calling application using Agora SDK.

Video Call

First quickly implement a video call function.

Create AgoraRtcEngineKit instance with AppId:

1private lazy var rtcEngine: AgoraRtcEngineKit = AgoraRtcEngineKit.sharedEngine(withAppId: <#Your AppId#>, delegate: self)
Copy the code

Set ChannelProfile and local preview:

1override func viewDidLoad() {
2    super.viewDidLoad()
3    rtcEngine.setChannelProfile(.communication)
4    let canvas = AgoraRtcVideoCanvas()
5    canvas.uid = 0
6    canvas.view = localVideoView
7    canvas.renderMode = .hidden
8    rtcEngine.setupLocalVideo(canvas)
9}
Copy the code

Set the remote view in the AgoraRtcEngineDelegate remote user join channel event:

 1extension ViewController: AgoraRtcEngineDelegate {
 2    func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) {
 3        let canvas = AgoraRtcVideoCanvas()
 4        canvas.uid = uid
 5        canvas.view = remoteVideoView
 6        canvas.renderMode = .hidden
 7        engine.setupRemoteVideo(canvas)
 8        remoteUid = uid
 9        remoteVideoView.isHidden = false11 10}}Copy the code

You can start, mute, and end a call using the following methods:

 1extension ViewController {
 2    func startSession(_ session: String) {
 3        rtcEngine.startPreview()
 4        rtcEngine.joinChannel(byToken: nil, channelId: session, info: nil, uid: 0, joinSuccess: nil)
 5    }
 6    func muteAudio(_ mute: Bool) {
 7        rtcEngine.muteLocalAudioStream(mute)
 8    }
 9    func stopSession() {
10        remoteVideoView.isHidden = true
11        rtcEngine.leaveChannel(nil)
12        rtcEngine.stopPreview()
13    }
14}
Copy the code

At this point, a simple video call application is completed. Both parties can make a video call by calling startSession(_:) to join the same channel.

Caller id

Let’s start by creating a special class called CallCenter to unify the CXProvider and CXCallController.

 1class CallCenter: NSObject {
 2    fileprivate let controller = CXCallController()
 3    private let provider = CXProvider(configuration: CallCenter.providerConfiguration)
 4    private static var providerConfiguration: CXProviderConfiguration {
 5        let appName = "AgoraRTCWithCallKit"
 6        let providerConfiguration = CXProviderConfiguration(localizedName: appName)
 7        providerConfiguration.supportsVideo = true
 8        providerConfiguration.maximumCallsPerCallGroup = 1
 9        providerConfiguration.maximumCallGroups = 1
10        providerConfiguration.supportedHandleTypes = [.phoneNumber]
11        if let iconMaskImage = UIImage(named: <#Icon file name#>) {
12            providerConfiguration.iconTemplateImageData = UIImagePNGRepresentation(iconMaskImage)
13        }
14        providerConfiguration.ringtoneSound = <#Ringtone file name#>
15        return providerConfiguration
16    }
17}
Copy the code

ProviderConfiguration sets some basic properties that CallKit needs to register calls with the system. For example, localizedName tells the system to display the name of the application to the user. IconTemplateImage Provides the system with an image to display on the call screen on the lock screen. RingtoneSound is a custom ringtone file.

Next, we create a method that registers the call to the system via CallKit when it is received.

 1func showIncomingCall(of session: String) {
 2    let callUpdate = CXCallUpdate()
 3    callUpdate.remoteHandle = CXHandle(type: .phoneNumber, value: session)
 4    callUpdate.localizedCallerName = session
 5    callUpdate.hasVideo = true
 6    callUpdate.supportsDTMF = false
 7    let uuid = pairedUUID(of: session)
 8    provider.reportNewIncomingCall(with: uuid, update: callUpdate, completion: { error in
 9        if let error = error {
10            print("reportNewIncomingCall error: \(error.localizedDescription)") 11} 12}) 13}Copy the code

For simplicity, we use the phone number string of the other party as the call session identifier, and construct a simple session and UUID matching query system. After invoking the reportNewIncomingCall(with: Update: Completion 🙂 method of CXProvider, the system will display an interface similar to that of carrier phones to remind users based on the information in CXCallUpdate. Users can answer or reject the call, or click the sixth button to open the app.

Answer/end a call

After the user clicks “Accept” or “reject” on the system interface, CallKit notifies app through the related callback of CXProviderDelegate.

 1func provider(_ provider: CXProvider, perform action: CXAnswerCallAction) {
 2    guard let session = pairedSession(of:action.callUUID) else {
 3        action.fail()
 4        return5 } 6 delegate? .callCenter(self, answerCall: session) 7 action.fulfill() 8} 9func provider(_ provider: CXProvider, perform action: CXEndCallAction) { 10 guardlet session = pairedSession(of:action.callUUID) else {
11        action.fail()
12        return13 } 14 delegate? .callCenter(self, declineCall: session) 15 action.fulfill() 16}Copy the code

By calling back the CXAction object passed in, we know the user’s action type and the UUID corresponding to the call. The final notification is sent to the app’s ViewController via our own CallCenterDelegate callback.

Initiate/Mute/End a call

Using CXStartCallAction to construct a CXTransaction, we can register an initiated call with the system using the REQUEST (_: Completion 🙂 method of the CXCallController.

 1func startOutgoingCall(of session: String) {
 2    let handle = CXHandle(type: .phoneNumber, value: session)
 3    let uuid = pairedUUID(of: session)
 4    let startCallAction = CXStartCallAction(call: uuid, handle: handle)
 5    startCallAction.isVideo = true
 6    let transaction = CXTransaction(action: startCallAction)
 7    controller.request(transaction) { (error) in
 8        if let error = error {
 9            print("startOutgoingSession failed: \(error.localizedDescription)") 10} 11} 12}Copy the code

Similarly, we can mute/end a call using CXSetMutedCallAction and CXEndCallAction.

 1func muteAudio(of session: String, muted: Bool) {
 2    let muteCallAction = CXSetMutedCallAction(call: pairedUUID(of: session), muted: muted)
 3    let transaction = CXTransaction(action: muteCallAction)
 4    controller.request(transaction) { (error) in
 5        if let error = error {
 6            print("muteSession \(muted) failed: \(error.localizedDescription)")
 7        }
 8    }
 9}
10func endCall(of session: String) {
11    let endCallAction = CXEndCallAction(call: pairedUUID(of: session))
12    let transaction = CXTransaction(action: endCallAction)
13    controller.request(transaction) { error in
14        if let error = error {
15            print("endSession failed: \(error.localizedDescription)") 16} 17} 18}Copy the code

Simulate incoming calls and calls

Real VoIP applications use the signaling system or iOS PushKit to implement calls. For simplicity, we added two buttons to the Demo that directly simulate receiving a new call and making a new call.

1private lazy var callCenter = CallCenter(delegate: self)
2@IBAction func doCallOutPressed(_ sender: UIButton) {
3    callCenter.startOutgoingCall(of: session)
4}
5@IBAction func doCallInPressed(_ sender: UIButton) {
6    callCenter.showIncomingCall(of: session)
7}
Copy the code

A complete CallKit video application is then completed by implementing the CallCenterDelegate callback, which calls the Agora SDK video call function we have already implemented in advance.

 1extension ViewController: CallCenterDelegate {
 2    func callCenter(_ callCenter: CallCenter, startCall session: String) {
 3        startSession(session)
 4    }
 5    func callCenter(_ callCenter: CallCenter, answerCall session: String) {
 6        startSession(session)
 7        callCenter.setCallConnected(of: session)
 8    }
 9    func callCenter(_ callCenter: CallCenter, declineCall session: String) {
10        print("call declined")
11    }
12    func callCenter(_ callCenter: CallCenter, muteCall muted: Bool, session: String) {
13        muteAudio(muted)
14    }
15    func callCenter(_ callCenter: CallCenter, endCall session: String) {
16        stopSession()
17    }
18}
Copy the code

Address book/System call history

A VoIP call that uses the CallKit is displayed in the call history of the user’s system. You can tap the call history to initiate a new VoIP call just like a carrier’s call. The user address book also provides an option for the user to directly initiate a call using an application that supports CallKit.

 1func application(_ application: UIApplication, continue userActivity: NSUserActivity, restorationHandler: @escaping ([Any]?) -> Void) -> Bool {
 2    guard let interaction = userActivity.interaction else{3return false
 4    }
 5    var phoneNumber: String?
 6    if let callIntent = interaction.intent as? INStartVideoCallIntent {
 7        phoneNumber = callIntent.contacts?.first?.personHandle?.value
 8    } else if let callIntent = interaction.intent as? INStartAudioCallIntent {
 9        phoneNumber = callIntent.contacts?.first?.personHandle?.value
10    }
11    letcallVC = window? .rootViewController as? ViewController 12 callVC? .applyContinueUserActivity(toCall:phoneNumber) 13return true
14}
15extension ViewController {
16    func applyContinueUserActivity(toCall phoneNumber: String?) {
17        guard letphoneNumber = phoneNumber, ! phoneNumber.isEmptyelse{18return
19        }
20        phoneNumberTextField.text = phoneNumber
21        callCenter.startOutgoingCall(of: session)
22    }
23}
Copy the code

Some caveats

1. The VoIP mode must be enabled in the background mode of the project to enable CallKit functions. This mode needs to be enabled by adding voip items under the UIBackgroundModes field of the info.plist file. If the background VoIP mode is not enabled, calling reportNewIncomingCall(with: Update: Completion 🙂 does not work.

2. After registering a call with the system using CXStartCallAction, the system starts AudioSession of the application and increases its priority to carrier call level. If an application sets up AudioSession itself during this process, AudioSession may fail to start. Therefore, the application needs to wait for the system to start AudioSession, and then set AudioSession after receiving the PROVIDER (_:didActive:) callback from the CXProviderDelegate. We handle this logic in Demo through the Agora SDK’s disableAudio() and enableAudio() interfaces.

3. After the CallKit is integrated, VoIP calls are affected by the do not Disturb Settings of the user system just as carrier calls are.

4. Press the lock button in handset mode, and the system will hang up. This behavior is also consistent with carrier phone calls. Go to Github for the full code :github.com/AgoraIO/Ago…