Double speed is a very important feature of the player, and the principle of double speed sounds simple enough: both audio and video frames have a PTS to identify the current timestamp. Double speed scenarios require audio streaming double speed and video streaming double speed.

  • Set the frame rate when decoding the video. Let MediaCodec decode the video frame according to the set frame rate.
  • Audio speed is more troublesome, because the sound speed at the same time, also have to consider the sound bit rate and tone changes. When going too fast or too slow, the audio is actually distorted, this is also something to consider.

Video is a frame by frame picture, audio is a stream of bits, then adjust the bit rate and amplitude of the bit stream, using AudioTrack to render the audio.

Playbackparameters.java is defined in ExoPlayer to store parameters related to this speed:

  /** The factor by which playback will be sped up. */
  public final float speed;

  /** The factor by which the audio pitch will be scaled. */
  public final float pitch;

  public final float volume;
Copy the code

We set times the speed of time by set PlaybackParameters into SimpleExoPlayer instances, then through ExoPlayerImplInternal incoming setPlaybackParametersInternal function

  private void setPlaybackParametersInternal(PlaybackParameters playbackParameters) {
    mediaClock.setPlaybackParameters(playbackParameters);
    sendPlaybackParametersChangedInternal(
        mediaClock.getPlaybackParameters(), /* acknowledgeCommand= */ true);
  }
Copy the code

DefaultMediaClock = setPlaybackParameters;

@Override public void setPlaybackParameters(PlaybackParameters playbackParameters) { if (rendererClock ! = null) { rendererClock.setPlaybackParameters(playbackParameters); playbackParameters = rendererClock.getPlaybackParameters(); } standaloneClock.setPlaybackParameters(playbackParameters); }Copy the code

This rendererClock is defined render instances, have also talked about before, audio rendering processing class is MediaCodecAudioRenderer. Java, render of the video processing class MediaCodecVideoRenderer. Java, the parent class is M Set PlaybackParameters instance ediaCodecRenderer. Java here to MediaCodecVideoRenderer, because only MediaCodecAudioRenderer. Java implementation MediaClock:

public class MediaCodecAudioRenderer extends MediaCodecRenderer implements MediaClock
Copy the code

The reason for this is that audio is very strict with time calibration, and we found that audio is very strict with time calculation when we analyzed audio and video synchronization. Basically, it’s based on audio PTS.

Perform to MediaCodecAudioRenderer setPlaybackParameters, then set to DefaultAudioSink. SetPlaybackParameters

  @Override
  public void setPlaybackParameters(PlaybackParameters playbackParameters) {
    audioSink.setPlaybackParameters(playbackParameters);
  }
Copy the code

Audio times speed

DefaultAudioSink is the module that controls the sound. To realize the double speed of audio, we need to process the sound first, because the audio cannot be played according to the normal audio after the double speed, and the bit rate and pitch of the sound will change. Is the selected AudioProcessor SonicAudioProcessor here, look at the SonicAudioProcessor. IsActive function, speed changing, SonicAudioProcessor will be activated.

@Override public boolean isActive() { return pendingOutputAudioFormat.sampleRate ! = Format.NO_VALUE && (Math.abs(speed - 1f) >= CLOSE_THRESHOLD || Math.abs(pitch - 1f) >= CLOSE_THRESHOLD || Math.abs(volume - 1f) >= CLOSE_THRESHOLD || pendingOutputAudioFormat.sampleRate ! = pendingInputAudioFormat.sampleRate); }Copy the code

When SonicAudioProcessor initializes, the current Sonic Settings are updated to include speed/pitch/volume.

@Override public void flush() { if (isActive()) { inputAudioFormat = pendingInputAudioFormat; outputAudioFormat = pendingOutputAudioFormat; if (pendingSonicRecreation) { sonic = new Sonic( inputAudioFormat.sampleRate, inputAudioFormat.channelCount, speed, pitch, volume, outputAudioFormat.sampleRate); } else if (sonic ! = null) { sonic.flush(); } } outputBuffer = EMPTY_BUFFER; inputBytes = 0; outputBytes = 0; inputEnded = false; }Copy the code

In SonicAudioProcessor. QueueInput in incoming audio stream:

@Override public void queueInput(ByteBuffer inputBuffer) { Sonic sonic = Assertions.checkNotNull(this.sonic); if (inputBuffer.hasRemaining()) { ShortBuffer shortBuffer = inputBuffer.asShortBuffer(); int inputSize = inputBuffer.remaining(); inputBytes += inputSize; sonic.queueInput(shortBuffer); inputBuffer.position(inputBuffer.position() + inputSize); } int outputSize = sonic.getOutputSize(); if (outputSize > 0) { if (buffer.capacity() < outputSize) { buffer = ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder()); shortBuffer = buffer.asShortBuffer(); } else { buffer.clear(); shortBuffer.clear(); } sonic.getOutput(shortBuffer); outputBytes += outputSize; buffer.limit(outputSize); outputBuffer = buffer; }}Copy the code
  • InputBuffer is the incoming raw audio stream
  • The raw audio stream is processed by Sonic, and the Sonic. QueueInput processes the raw bitstream
  • The processed bit stream is returned via sonic. GetOutput (shortBuffer), and the processed bit stream is directly returned to AudioTrack for processing.

Sonic. Java reference: github.com/waywardgeek… Sonic is a simple algorithm for speeding up or slowing down speech. However, unlike the previous change algorithm, it optimizes speech speed for more than 2x acceleration. The Sonic Library is a very simple ANSI C library designed to be easily integrated into streaming voice applications such as TTS backends.

The main motivation behind Sonic is to enable the blind and visually impaired to increase their productivity with an open source voice engine like Espeak. Sonic can also be used by sighted people. For example, soundwave can improve the experience of listening to audiobooks on Android phones.

ExoPlayer introduced Sonic to handle audio multiple speeds and pitch and volume operations.

public void queueInput(ShortBuffer buffer) { int framesToWrite = buffer.remaining() / channelCount; int bytesToWrite = framesToWrite * channelCount * 2; inputBuffer = ensureSpaceForAdditionalFrames(inputBuffer, inputFrameCount, framesToWrite); buffer.get(inputBuffer, inputFrameCount * channelCount, bytesToWrite / 2); inputFrameCount += framesToWrite; processStreamInput(); } private void processStreamInput() { // Resample as many pitch periods as we have buffered on the input. int originalOutputFrameCount = outputFrameCount; float s = speed / pitch; float r = rate * pitch; If (s > 1.00001 | | s < 0.99999) {changeSpeed (s); } else { copyToOutput(inputBuffer, 0, inputFrameCount); inputFrameCount = 0; } if (r ! AdjustRate (r, originalOutputFrameCount); } if(volume ! = 1.0 f) {/ / Adjust the output volume. ScaleSamples (outputBuffer originalOutputFrameCount, outputFrameCount - originalOutputFrameCount, volume); }}Copy the code

ChangeSpeed: float s = speed/pitch Speed is closely related to pitch, because the speed changes, and the duration of the pitch changes.

private void changeSpeed(float speed) { if (inputFrameCount < maxRequiredFrameCount) { return; } int frameCount = inputFrameCount; int positionFrames = 0; do { if (remainingInputToCopyFrameCount > 0) { positionFrames += copyInputToOutput(positionFrames); } else { int period = findPitchPeriod(inputBuffer, positionFrames); If (speed > 1.0) {positionFrames += period + skipPitchPeriod(inputBuffer, positionFrames, speed, period); } else { positionFrames += insertPitchPeriod(inputBuffer, positionFrames, speed, period); } } } while (positionFrames + maxRequiredFrameCount <= frameCount); removeProcessedInputFrames(positionFrames); }Copy the code
  • Process all data between the current point in time position and the input frame count
  • Copy the previously processed ByteBuffer data to the output buffer
  • If the current speed > 1.0, is fast playback, that must be a part of the audio data, how to skip, or pay attention to strategy.

Generally speaking, if speed > 2.0, the audio is no longer heard, so you don’t need to make sure the audio is heard after that, just skip the audio frame. If speed between 1.0 and 2.0, audio playback basically has no effect, then copy the audio data to remainingInputToCopyFrameCount assignment, copy the original audio data in the past, while jumping frames, but part of the original data after processing or output to the oup Ut Buffer queue

  private int copyInputToOutput(int positionFrames) {
    int frameCount = Math.min(maxRequiredFrameCount, remainingInputToCopyFrameCount);
    copyToOutput(inputBuffer, positionFrames, frameCount);
    remainingInputToCopyFrameCount -= frameCount;
    return frameCount;
  }
Copy the code
  • If speed < 1.0, it is slow down, and the original data needs to be filled with some data to spread the whole timestamp

If the speed is less than 0.5F, the audio cannot be heard at all and no data will be rendered. If the speed is between 0.5 and 1.0, some data will be copied to the output buffer

  • The filled Ouput buffer is sent to AudioTrack to render and play

Video times speed

MediaCodecRenderer. UpdateCodecOperatingRate incoming codecOperatingRate used for control video times the speed of video MediaCodec decoding is used to control the rate of times of,

private void updateCodecOperatingRate() throws ExoPlaybackException { if (Util.SDK_INT < 23) { return; } float newCodecOperatingRate = getCodecOperatingRateV23(rendererOperatingRate, codecFormat, getStreamFormats()); if (codecOperatingRate == newCodecOperatingRate) { // No change. } else if (newCodecOperatingRate == CODEC_OPERATING_RATE_UNSET) { // The only way to clear the operating rate is to instantiate a new codec instance. See //  [Internal ref: b/71987865]. drainAndReinitializeCodec(); } else if (codecOperatingRate ! = CODEC_OPERATING_RATE_UNSET || newCodecOperatingRate > assumedMinimumCodecOperatingRate) { // We need to set the operating rate, either because we've set it previously or because it's // above the assumed minimum rate. Bundle codecParameters = new Bundle(); codecParameters.putFloat(MediaFormat.KEY_OPERATING_RATE, newCodecOperatingRate); codec.setParameters(codecParameters); codecOperatingRate = newCodecOperatingRate; }}Copy the code
  • Set MediaCodec frame rate to control video codec decoding to achieve video double speed effect
  • There’s no complicated logic to video doubling, right

The key to seize the key point of audio speed: jump frame processing