A normal URL is set to the player. The player generally goes through the following process:

  • The request url
  • decapsulation
  • Separate audio, video and subtitle tracks
  • Start a thread to decode the audio, and start a thread to decode the video, and then sync the audio and video
  • Audio and video are played using AudioTrack or OpenSL ES, and videos are rendered using TextureView or SurfaceView

In the last article, we analyzed how the ExoPlayer requests urls, and when we get a certain amount of data, we need to unpack the source data. The premise of unpacking is to know the video packaging format? The process of exploring video package format is Extractor. This article mainly analyzes the Extractor module of ExoPlayer.

1. Package formats supported by ExoPlayer

Basically see DefaultExtractorsFactory. CreateExtractors methods in Java classes:

public synchronized Extractor[] createExtractors() { Extractor[] extractors = new Extractor[14]; extractors[0] = new MatroskaExtractor(matroskaFlags); extractors[1] = new FragmentedMp4Extractor(fragmentedMp4Flags); extractors[2] = new Mp4Extractor(mp4Flags); extractors[3] = new Mp3Extractor( mp3Flags | (constantBitrateSeekingEnabled ? Mp3Extractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING : 0)); extractors[4] = new AdtsExtractor( adtsFlags | (constantBitrateSeekingEnabled ? AdtsExtractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING : 0)); extractors[5] = new Ac3Extractor(); extractors[6] = new TsExtractor(tsMode, tsFlags); extractors[7] = new FlvExtractor(); extractors[8] = new OggExtractor(); extractors[9] = new PsExtractor(); extractors[10] = new WavExtractor(); extractors[11] = new AmrExtractor( amrFlags | (constantBitrateSeekingEnabled ? AmrExtractor.FLAG_ENABLE_CONSTANT_BITRATE_SEEKING : 0)); extractors[12] = new Ac4Extractor(); if (FLAC_EXTENSION_EXTRACTOR_CONSTRUCTOR ! = null) { try { extractors[13] = FLAC_EXTENSION_EXTRACTOR_CONSTRUCTOR.newInstance(); } catch (Exception e) { // Should never happen. throw new IllegalStateException("Unexpected error creating FLAC extractor", e); } } else { extractors[13] = new FlacExtractor(); } return extractors; }Copy the code
  • MatroskaExtractor: Extracts data from the Matroska and WebM Container formats
  • FragmentedMp4Extractor: Extracts data from the FMP4 Container format
  • Mp4Extractor: Extracts Data from the MP4 container format
  • Mp3Extractor: Extracts data from the MP3 container format
  • AdtsExtractor: Extracts Data from AAC bit Streams with ADTS Framing
  • Ac3Extractor: Extracts Data from (E-)AC-3 BitStreams
  • TsExtractor: Extracts data from the MPEG-2 TS Container format
  • FlvExtractor: Extracts data from the FLV container format
  • OggExtractor: Extracts data from the Ogg Container format
  • PsExtractor: Extracts data from the MPEG-2 PS Container format
  • WavExtractor: Extracts data from WAV Byte Streams
  • AmrExtractor: Extracts Data from the AMR Containers format (either AMR or AMR-WB)
  • Ac4Extractor: Extracts data from AC-4 Bitstreams
  • FlacExtractor: Extracts data from FLAC Container format

2. The Extractor process

Extractor interface has the following methods:

public interface Extractor {
  int RESULT_CONTINUE = 0;
  int RESULT_SEEK = 1;
  int RESULT_END_OF_INPUT = C.RESULT_END_OF_INPUT;

  boolean sniff(ExtractorInput input) throws IOException, InterruptedException;

  void init(ExtractorOutput output);

  int read(ExtractorInput input, PositionHolder seekPosition)
      throws IOException, InterruptedException;

  void seek(long position, long timeUs);

  void release();
}
Copy the code
  • Sniffing function: A function to sniff out video encapsulation formats. This function mainly reads binary data in video files and compares it with encapsulated protocols.
  • Init function: Now that the package format is known, start to initialize the specific package Extractor, because it is ready to read the track data in the video.
  • Read function: Reads concrete video data.

Detection of video format of encapsulation place in ProgressiveMediaPeriod. ExtractorHolder inner class in Java.

public Extractor selectExtractor(ExtractorInput input, ExtractorOutput output, Uri uri) throws IOException, InterruptedException { if (extractor ! = null) { return extractor; } if (extractors.length == 1) { this.extractor = extractors[0]; } else { for (Extractor extractor : extractors) { try { if (extractor.sniff(input)) { this.extractor = extractor; break; } } catch (EOFException e) { // Do nothing. } finally { input.resetPeekPosition(); } } if (extractor == null) { throw new UnrecognizedInputFormatException( "None of the available extractors (" + Util.getCommaDelimitedSimpleClassNames(extractors) + ") could read the stream.", uri); } } extractor.init(output); return extractor; }Copy the code

After iterating through the Extractors array and finding rules that fit a particular format, it is assumed to be this encapsulation format.

Take the Mp4Extractor as an example to analyze how ExoPlayer recognizes specific package formats internally.

  • Extractor. Sniff Was called to Mp4Extractor
  • Then call to sniffers. SniffUnfragmented, sniffers. SniffInternal

In sniffer. sniffInternal there is code to detect whether MP4 video is available. Here’s a problem: MP4 videos are sequenced, mainly mooV and MDAT. Moov represents a collection of video-related properties, and MDAT is specific audio and video data. The MOOV data must be read before mdAT data can be properly parsed. Otherwise, MP4 video cannot be played properly. Normally, MOOV is before MDAT. Adding moov just after MDAT makes it easy for the player to see the parsing process. In the sniffer.sniffinternal function:This is the dual-IO detection mechanism. It is very helpful to solve the MOOV position of MP4 video.

To obtain the correct Extractor, can begin to read video data, we’re now going back to ProgressiveMediaPeriod. The load method:

          while (result == Extractor.RESULT_CONTINUE && !loadCanceled) {
            loadCondition.block();
            result = extractor.read(input, positionHolder);
            if (input.getPosition() > position + continueLoadingCheckIntervalBytes) {
              position = input.getPosition();
              loadCondition.close();
              handler.post(onContinueLoadingRequestedRunnable);
            }
          }
Copy the code
    onContinueLoadingRequestedRunnable =
        () -> {
          if (!released) {
            Assertions.checkNotNull(callback)
                .onContinueLoadingRequested(ProgressiveMediaPeriod.this);
          }
        };
Copy the code

Loading data is done in child threads.

The extractor.read operation corresponds to mp4Extractor.read

public int read(ExtractorInput input, PositionHolder seekPosition) throws IOException, InterruptedException { while (true) { switch (parserState) { case STATE_READING_ATOM_HEADER: if (! readAtomHeader(input)) { return RESULT_END_OF_INPUT; } break; case STATE_READING_ATOM_PAYLOAD: if (readAtomPayload(input, seekPosition)) { return RESULT_SEEK; } break; case STATE_READING_SAMPLE: return readSample(input, seekPosition); default: throw new IllegalStateException(); }}}Copy the code
  • STATE_READING_ATOM_HEADER Starts reading atom header information
  • STATE_READING_ATOM_PAYLOAD If atom is not fully read, the player sets parserState to STATE_READING_ATOM_PAYLOAD
  • STATE_READING_SAMPLE atom Reads the original data of the media file

ReadAtomHeader: Reads the header of an MP4 file

ReadAtomPayload: The atom header is not read

ReadSample: Start to read track information parsed by MOOv