IOS end audio recording while turning and the realization of sonogram

Our company and I shared the technical articles with our Android partners of the project. We should not lag behind in iOS. We sorted out some content about audio processing, hoping it would be helpful to everyone.

Talk is cheap, show you the code WaveDemo

The project I am in needs to upload the audio to the server. The PCM file generated by the iOS native recording is too large, so in order to unify the three ends, we decide to use mp3 format. , so we need to transcode using lame. A is mostly iOS6 and 7, and some of them don’t support bitcode. Let me make a sad face 🤣 I found a library on Github that builds lame source code. It supports bitcode and can be modified to the lowest supported version. Modify the sh file

Tip: Lipo operation can operate libxxx. a file, add and delete platform dependencies.

## Transcoding with lame finished, we started coding. At first I googled and found a way to use lame. The core code is as follows:

@try {

        int read, write;

FILE *pcm = fopen([cafFilePath cStringUsingEncoding:1], "rb"); //source The location of the converted audio file

        fseek(pcm, 4*1024, SEEK_CUR);                                   //skip file header

FILE *mp3 = fopen([mp3FilePath cStringUsingEncoding:1], "wb"); //output Indicates the location of the generated Mp3 file

        const int PCM_SIZE = 8192;

        const int MP3_SIZE = 8192;

        short int pcm_buffer[PCM_SIZE*2];

        unsigned char mp3_buffer[MP3_SIZE];



        lame_t lame = lame_init();

Lame_set_in_samplerate (lame, 22050.0);

        lame_set_VBR(lame, vbr_default);

        lame_init_params(lame);



        do {

            read = fread(pcm_buffer, 2*sizeof(short int), PCM_SIZE, pcm);

            if (read == 0)

                write = lame_encode_flush(lame, mp3_buffer, MP3_SIZE);

            else

                write = lame_encode_buffer_interleaved(lame, pcm_buffer, read, mp3_buffer, MP3_SIZE);



            fwrite(mp3_buffer, write, 1, mp3);



} while (read ! = 0);



        lame_close(lame);

        fclose(mp3);

        fclose(pcm);

    }

    @catch (NSException *exception) {

        NSLog(@"%@",[exception description]);

    }

    @finally {

        return mp3FilePath;

    }



Copy the code

We finished the first step. After recording, we converted the PCM to MP3. Let me check whether the audio can be played and so on. No need to ask, there must be something wrong with transcoding. After searching some information, we learned that the parameter vbr_default of lame_set_VBR is in the form of VBR, and the default AVPlayer uses the form of CBR average bit rate to recognize the playback, resulting in incorrect duration, so we adjust the above line of code:

lame_set_VBR(lame, vbr_off);
Copy the code

Note that the sampling rate of the simulator and the real machine is slightly different, such as the situation of playing noise can be adjusted to

lame_set_in_samplerate(lame, 44100);
Copy the code

## Optimization transcoding is actually turning while recording, here I looked at some articles, mostly based on AVAudioRecorder implementation of the way is rather rough, do not want to use. I also found an AVAudioQueue based API with multiple C languages. Remember the introduction of AVAudioEngine in Apple’s session a while ago, it is more oc to use. In the heart of learning is to toss, AVAudioEngine is used for transcoding. IOS AVAudioEngine uses AVAudioEngine to get the audio buffer from time to time, transcodes it, and append the transcoded data (you can modify it by yourself, Stream upload using AFNetworking). The core code is as follows:

Transcoding prep, create engine, and initialize lame

  private func initLame(a) {
    
    engine = AVAudioEngine(a)guard let engine = engine,
          let input = engine.inputNode else {
        return
    }
    
    let format = input.inputFormat(forBus: 0)
    let sampleRate = Int32(format.sampleRate) / 2
    
    lame = lame_init()
    lame_set_in_samplerate(lame, sampleRate);
    lame_set_VBR_mean_bitrate_kbps(lame, 96);
    lame_set_VBR(lame, vbr_off);
    lame_init_params(lame);
  }
Copy the code

Set AVAudioSession to set the preferred sampling rate and IO frequency to get buffer

 let session = AVAudioSession.sharedInstance()
    do{ try session.setCategory(AVAudioSessionCategoryPlayAndRecord) try session.setPreferredSampleRate(44100) try Session. SetPreferredIOBufferDuration (0.1) try session. SetActive (true)
      initLame()
    } catch {
      print("Setup" seesion)
      return
    }
Copy the code

For volume calculation, Accelerate library is used for efficient calculation. For details, see Level Metering with AVAudioEngine

letLevelLowpassTrig: Float = 0.5 var avgValue: Float32 = 0 vDSP_meamgv(buf, 1, &avgValue, vDSP_Length(frameLength)) this.averagePowerForChannel0 = (levelLowpassTrig * ((avgValue==0) ? - 100:20.0 *log10f(avgValue))) + ((1-levelLowpassTrig) * this.averagePowerForChannel0)
        
        letVolume = min (enclosing averagePowerForChannel0 + Float (55)) / 55.0, 1.0) enclosing minLevel = min (enclosing minLevel, Volume) this.maxlevel = Max (this.maxlevel, volume) voluem: volume) }Copy the code

End of operation

public func stop(a){ engine? .inputNode? .removeTap(onBus:0)
    engine = nil
    do {
      var url = try FileManager.default.url(for: .cachesDirectory, in: .userDomainMask, appropriateFor: nil, create: false)
      let name = String(CACurrentMediaTime()).appending(".mp3")
      url.appendPathComponent(name)
      if! data.isEmpty {try data.write(to: url)
      }
      else {
        print("Empty file")}}catch {
      print("File operation")
    }
    data.removeAll()
  }

Copy the code

A couple of caveats

Lame is an open source MP3 decoder. I didn’t go back to the file, so I cleaned the VBR header and used CBR instead

lame_set_bWriteVbrTag(_lame, 0);
Copy the code

In the input callback, we changed the frameLength of buffer. Since the default input callback frequency is 0.375s, we can change the frequency by changing frameLength. Using this method is not elegant enough, so set up a session To achieve a similar goal.
You cannot obtain the current recording duration during recording. Use NSDate and NSTimer to calculate the duration.
On the simulator, everything is ok, on my small 6, the recorded audio playback is fast forward 🤣,debug found for a long time, on the real machine, if the continuous playback recording, Sometimes the format in the input of AVAudioEngine gets the sampleRate rate of 16,000 rather than 44100 as expected. There are two solutions

Set the preferdSampleRate of AudioSession

AVAudioSession *sessionInstance = [AVAudioSession sharedInstance];
[sessionInstance setPreferredSampleRate:kPreferredSampleRate error:&error]
Copy the code

Or use mixNode to handle the sampling rate, see this post

Due to the reconstruction of the content of the above part, the business was pulled up. My Android friend has completed the drawing of the waveform, you can check the effect picture.

Android self-drawing animation implementation and some optimization thinking — Take the example of the recording waveform animation of The App of Wisdom Class Correction as an example. My Android partner introduced how to draw this graph in detail. With his help, I also realized this effect on iOS.

The basic flow

Calculate the position of the curve point and the symmetric decay function
Draw points based on the calculation, using CAShapeLayer and UIBezierPath
According to the time, change the edge φ value to achieve curve displacement effect
According to the volume volume and attenuation function, change the amplitude to achieve up and down fluctuation.

The optimization methods are basically the same

Reduce drawing density
Reduce the amount of repeated real-time computation
Reuse reduces object creation and destruction

The core code is as follows:

CGFloat reduction[kPointNumber];

CGFloat perVolume;

NSInteger count;

@property (nonatomic, assign) CGFloat targetVolue;

@property (nonatomic, copy) NSArray<NSNumber *> *amplitudes;

@property (nonatomic, copy) NSArray<CAShapeLayer *> *shapeLayers;

@property (nonatomic, copy) NSArray<UIBezierPath *> *paths;

- (void)doSomeInit {

PerVolume = 0.15;

  count = 0;

Self. amplitudes = @[@0.6, @0.35, @0.1, @-0.1];

  self.shapeLayers = [self.amplitudes bk_map:^id(id obj) {

      CAShapeLayer *layer = [self creatLayer];

      [self.layer addSublayer:layer];

      return layer;

    }];

    self.shapeLayers.firstObject.lineWidth = 2;

    self.paths = [self.amplitudes bk_map:^id(id obj) {

      return [UIBezierPath bezierPath];

    }];

  for (int i = 0; i < kPointNumber; i++) {

Reduction [I] = self.height / 2.0 * 4 /(4 + pow((I /(CGFloat)kPointNumber -0.5) * 3, 4));

    }

}

 - (CAShapeLayer *)creatLayer {

  CAShapeLayer *layer = [CAShapeLayer layer];

  layer.fillColor = [UIColor clearColor].CGColor;

  layer.strokeColor = [UIColor defaultColor].CGColor;

Layer. Our lineWidth = 0.2;

  return layer;

}

// To ignore small fluctuations

- (void)setTargetVolue:(CGFloat)targetVolue {

  if (ABS(_targetVolue - targetVolue) > perVolume) {

    _targetVolue = targetVolue;

  }

}

// Adjust the volume smoothly during each CADisplayLink cycle.

- (void)softerChangeVolume {

  CGFloat target = self.targetVolue;

  if (volume < target - perVolume) {

    volume += perVolume;

  } else if (volume > target + perVolume) {

    if (volume < perVolume * 2) {

      volume = perVolume * 2;

    } else {

      volume -= perVolume;

    }

  } else {

    volume = target;

  }

}



- (void)updatePaths:(CADisplayLink *)sender {

// Take the axis [-3,3] and screen 64 pixels

  NSInteger xLen = 64;

  count++;

  [self softerChangeVolume];

  for (int i = 0; i < xLen; i++) {

    CGFloat left = i/(CGFloat)xLen * self.width;

    CGFloat x = (i/(CGFloat)xLen - 0.5) * 3;

TmpY = volume * reduction[I] * sin(M_PI* x-count *0.2);

    for (int j = 0; j < self.amplitudes.count ; j++) {

      CGPoint point = CGPointMake(left, tmpY * [self.amplitudes[j] doubleValue]  + self.height/2);

      UIBezierPath *path = self.paths[j];

      if (i == 0) {

        [path moveToPoint:point];

      } else {

        [path addLineToPoint:point];

      }

    }

  }

  for (int i = 0; i < self.paths.count; i++) {

    self.shapeLayers[i].path = self.paths[i].CGPath;

  }

  [self.paths bk_each:^(UIBezierPath *obj) {

    [obj removeAllPoints];

  }];

}

Copy the code

Of course, there are some small shortcomings, when changing the path, I found that the CPU usage hit 15%, I do not know whether there is a way to continue to optimize, we brainstorm 😜

Update: The waveform is implemented in a different way, which is full of compliments to Android students 🙆🏻♂️

After the Demo was finished, I used Swift to write and found that the effect was different from oc, so I made some adjustments… Link below: WaveDemo

If helped you, can point a wave of attention, go a wave of fish ball is not right, to the article point like, the author points a concern on the line 😘

IOS build-lame-for-ios Converts PCM files to MP3 using lame on iOS. IOS converts PCM files to MP3 using lame. IOS AVAudioEngine The realization of Android self-drawing animation and some optimization thinking — a case study of App recording waveform animation of Wisdom class correction

IOS end audio recording while turning and the realization of sonogram

Related Posts

2017 iOS Recent Phone Number regular Expression (OC/Swift)

[iOS features]- NSString string handling (interception, inclusion, capitalization, etc.)

IOS Timer Task