Summary of development of wechat iOS voice reminder of receipt to account

This is the first article I published about the company’s internal KM, which was reprinted in Bugly and Tencent lecture hall. Notes some details of the development of the receipt-to-account voice reminder feature in 2017, which I also mentioned at the end of the changes for iOS13. The original link

The background,

In order to solve the pain point that small business owners are not convenient to check and confirm the receipt of accounts in frequent transactions, product MM proposed a new version to support the voice reminder function of receipt of accounts. This article summarizes the pitfalls and tips encountered during development.

Ii. Technical scheme

Background Wake up App

The voice reminder of receiving money requires the receiver to play a TTS synthesized voice to broadcast the amount after receiving the money. When wechat is in the foreground, it can bring down the amount to be broadcast through the template message, and then request TTS data and play it. However, how can the APP request the voice data and play it when it is suspended or killed? IOS provides two ways to wake up suspended or killed apps. For Silent Notification and VoIP Push Notification respectively, the client will get 30 seconds of background running time after being woken up, which is long enough to request synthesized voice data and play it.

1. Silent Notification: Silent Notification is supported in iOS7 or later. However, the number of Silent Notification notifications that can be pushed per hour is limited.
2. VoIP Push Notification VoIP Push Notification is a new Push type supported only in iOS8. Compared with Silent Notification, VoIP Push has advantages of high priority and low latency. And there is no limit to the number of times. Compared with these two technical solutions, VoIP Push Notification is obviously more suitable for the wake up scheme of receiving-to-account voice reminder.

TTS synthetic speech

TTS speech synthesis scheme is divided into off-line synthesis scheme and online synthesis scheme. The off-line synthesis scheme saves network request, and the synthesis speed is faster and network traffic is saved. However, the synthesized speech sounds mechanical and the processing of speed and pause is poor. If you don’t have a strong need for synthesized speech, consider using the AVSpeechSynthesis framework that comes with iOS to reduce the size of your speech library.

Online compositing was more vocal and emotional. Considering the product experience, we adopted the online speech synthesis solution provided by the Search Products department (the public development platform also has an API interface for third party development). The synthesized voice formats support WAV, MP3, Silk, AMR and Speex. After comparison, it is found that when the same text is synthesized, THE COMPRESSION rate of AMR is the highest, but the sound quality decreases obviously. The Silk format has the second highest compression rate and can maintain relatively clear sound quality. The size of a single synthesized voice is around 2KB.

Update in 2019: Wechat’s payment voice broadcast has been replaced by offline synthesis, and has been online. As an aside, Although Long Ge has expressed some doubts about artificial intelligence in public occasions, wechat actually has a strong deep learning research team, and the research on voice is also very in-depth.

Wake up and play the audio file

After requesting the synthesized voice, play the audio file in the background or on a locked screen, AVAudio Session Category of value need to use AVAudioSessionCategoryPlayback or AVAudioSessionCategoryPlayAndRecord, CategoryOptions Select MixWithOthers (mix with other sounds) or DuckOthers (lower the volume of other sounds) as required.

Note that only iOS10 and above supports playback of audio in background/locked screen after an app is woken up. Therefore, devices below iOS10 can only set a fixed ringtone on local Push after receiving VoIP Push, which is also why there is only “wechat Payment receivables received” below iOS10, but no specific amount value behind.

Three, mute switch detection

Unfortunately, the product was mocked by an Internet bigshot not long after its launch.

AVAudioSessionCategoryPlayback
AVAudioSessionCategoryPlayAndRecord

Apple can listen for the mute button in the following ways before iOS5

- (BOOL)isMuted  
{  
    CFStringRef route;  
    UInt32 routeSize = sizeof(CFStringRef);  

    OSStatus status = AudioSessionGetProperty(kAudioSessionProperty_AudioRoute, &routeSize, &route);  
    if (status == kAudioSessionNoError)  
    {  
        if (route == NULL || !CFStringGetLength(route))  
            return YES;  
    }  

    return NO;  
}
Copy the code

Apple has banned listening for mute buttons in this way since iOS5. The reason behind this is that apple wants developers to use AVAudioSession to provide a unified audio playback effect.

In the end, I took to Reddit to find an easy way to save the country: Using AudioServicesPlaySystemSound play a period of 0.2 s blank audio, and events to monitor audio playback is complete, if from the start playing to the callback method of interval time is less than 0.1 s, means the mute switch is open.

void SoundMuteNotificationCompletionProc(SystemSoundID  ssID,void* clientData){
    MMSoundSwitchDetector* detecotr = (__bridge MMSoundSwitchDetector*)clientData;
    [detecotr complete];
}

- (instancetype)init {
    self = [super init];
    if (self) {
        NSURL *pathURL = [[NSBundle mainBundle] URLForResource:@"mute" withExtension:@"caf"];
        if (AudioServicesCreateSystemSoundID((__bridge CFURLRef)pathURL, &_soundId) == kAudioServicesNoError){
            AudioServicesAddSystemSoundCompletion(self.soundId, CFRunLoopGetMain(), kCFRunLoopDefaultMode, SoundMuteNotificationCompletionProc,(__bridge void(*)self));
            UInt32 yes = 1;
            AudioServicesSetProperty(kAudioServicesPropertyIsUISound, sizeof(_soundId),&_soundId,sizeof(yes), &yes);
        } else {
            MMErrorWithModule(LOGMODULE, @"Create Sound Error.");
            _soundId = 0; }}return self;
}

- (void)checkSoundSwitchStatus:(CheckSwitchStatusCompleteBlk)completHandler {
    if (self.soundId == 0) {
        completHandler(YES);
        return;
    }
    self.completeHandler = completHandler;
    self.beginTime = CACurrentMediaTime(a); AudioServicesPlaySystemSound(self.soundId);
}

- (void)complete {
    CFTimeInterval elapsed = CACurrentMediaTime() - self.beginTime;
    BOOL isSwitchOn = elapsed > 0.1;
    if (self.completeHandler) {
        self.completeHandler(isSwitchOn); }}Copy the code

Set the sound threshold

Another problem frequently reported by users was that they could not hear the broadcast sound. After checking the logs, it was found that the system volume set by users was too low when the voice broadcast was triggered. The first solution was to set the AVAudioPlayer volume (or the kAudioQueueParam_Volume in the AudioQueue) directly. However, this did not work. The volume property is subject to the system volume (e.g., the system volume is 0.5, AVAudioPlayer’s volume is 0.6, the final volume is 0.5*0.6 =0.3). To solve the problem of too small volume, or need to adjust the system volume. The final solution draws on the scheme that automatically adjusts the screen brightness when entering the QR code of payment display: if the screen brightness does not reach the threshold, adjust the screen brightness to the threshold, and when leaving the page, set the brightness back to the original brightness. Similarly, if the system volume is lower than the threshold, adjust it to the threshold. After the prompt tone is played, the tone is returned to the original volume.

There are two ways to control system volume:

• Method 1: Set the volume through MPMusicPlayerController

MPMusicPlayerController *mpc = [MPMusicPlayerController applicationMusicPlayer];
//This property is deprecated -- use MPVolumeView for volume control instead.
mpc.volume = 0;  / / 0.0 ~ 1.0
Copy the code

The first method is simple and rude. The system volume prompt box will pop up when setting. If the volume prompt box suddenly pops up when users are using the app, it will cause trouble to users.

• Method 2: Use MPVolumeView to set the volume

The second method is to add an invisible MPVolumeView to the current view, so that the volume prompt box will not be displayed. Note that you need to remove the MPVolumeView after adjusting the volume. Otherwise, the volume prompt box will not be displayed when you manually adjust the volume.

To adjust the volume, fetch a child View named MPVolumeSlider in MPVolumeView and send an event to it that simulates the user’s actions.

- (void)setSystemVolume:(float)volume {
    UISlider* volumeViewSlider = nil;
    for (UIView *view in [self.m_privateVoulmeView subviews]){
        if ([view.class.description isEqualToString:@"MPVolumeSlider"]){
            volumeViewSlider = (UISlider*)view;
            break; }}if(volumeViewSlider ! =nil) {
        [volumeViewSlider setValue:volume animated:NO];
        / / by send
        [volumeViewSlider sendActionsForControlEvents:UIControlEventTouchUpInside]; }}Copy the code

IOS13 no longer uses PushKit for voice broadcast

Apple no longer allows voip push applications on non-voip phones in iOS13, as mentioned in session707, to address battery life, performance and user privacy concerns and reduce the abuse of background tasks. If pushKit is used, you need to access the Callkit interface. As a result, a Voip Push will pull up a full-screen interface for receiving and making calls.

Changes in iOS13 Voip Push:

[[NSString alloc] initWithFormat:@”%@”, self.m_token] is no longer used to pass tokens to the background.

2. Need before the end of the didReceiveIncomingPushWithPayload method, called the call of kit [CXProvider reportNewIncomingCallWithUUID:] method. Otherwise, the following crash will occur:

killing app because it never posted an incoming call to system after receiving a PushKit Voip push CallBack

After receiving voip push for several times, the system forbids receiving voip push.

3. Call call kit [CXProvider reportNewIncomingCallWithUUID] will lead to a full screen on the phone interface, this interface temporarily there is no way to get rid of, So this makes it impossible in iOS13 to pull up the App and perform background tasks without the user being aware of it.

How to implement voice broadcast in iOS13?

Is it impossible to do this without VoipPush? The answer is no, similar functionality can also be achieved using Notification Service Extension. I am currently migrating this part of functionality, and I will write another article about the implementation details and pitfalls in the process when the project is officially launched.