Android FFmpeg+OpenSL ES audio decoding playback

Sound Ming

First of all, this series of articles are based on their own understanding and practice, there may be wrong places, welcome to correct. Secondly, this is an introductory series, covering only enough knowledge, and there are many blog posts on the Internet for in-depth knowledge. Finally, in the process of writing the article, I will refer to the articles shared by others and list them at the end of the article, thanking these authors for their sharing.

Code word is not easy, reproduced please indicate the source!

Tutorial code: [Making portal】

You can read about it in this article

This article introduces how to use FFmpeg for audio decoding, focusing on how to use OpenSL ES in the NDK layer audio rendering playback.

First, audio decoding

In the last article, detailed introduction of FFmepg playback process, as well as the abstract decoding process framework, integration of video and audio decoding process common ground, formed BaseDecoder class. By inheriting BaseDecoder to achieve the video decoder subclass VideoDeocder, and integrated into the Player, to achieve the video playback rendering.

This article has defined the use of decoding base class BaseDecoder to achieve audio decoding subclass AudioDecoder.

Implement audio decoding subclasses

Let’s first look at what we need to do to achieve audio decoding.

Define the decoding process

We define the required member variables and process methods using the header a_decoder.h.

I. Member variable definition

//a_decoder.h

class AudioDecoder: public BaseDecoder {
private:

 const char *TAG = "AudioDecoder";   // Audio converter  SwrContext *m_swr = NULL;   // Audio renderer  AudioRender *m_render = NULL;   // Output buffer  uint8_t *m_out_buffer[1] = {NULL};   // The number of samples each channel contains after resampling  // Acc defaults to 1024 and may change after resampling  int m_dest_nb_sample = 1024;   // The size of a frame after resampling  size_t m_dest_data_size = 0;   / /... } Copy the code

Among them, SwrContext is an audio conversion tool provided by FFmpeg, located in swresample, which can be used to convert sampling rate, number of decoded channels, sampling bits and so on. This is used to convert audio data into two – channel stereo sound, unified sampling bits.

⚠️ AudioRender is a custom audio renderer that will be covered later.

Other variables, which need to be used in audio conversion, are the output buffer, buffer size, and number of samples.

Ii. Define member methods

// a_decoder.h

class AudioDecoder: public BaseDecoder {
private:

 // omit the member variable......   // Initialize the conversion tool  void InitSwr(a);   // Initialize the output buffer  void InitOutBuffer(a);   // Initialize the renderer  void InitRender(a);   // Release the buffer  void ReleaseOutBuffer(a);   // Sampling format: 16 bits  AVSampleFormat GetSampleFmt(a) {  return AV_SAMPLE_FMT_S16;  }   // Target sample rate  int GetSampleRate(int spr) {  return AUDIO_DEST_SAMPLE_RATE; //44100Hz  }  public:  AudioDecoder(JNIEnv *env, const jstring path, bool forSynthesizer);  ~AudioDecoder();   void SetRender(AudioRender *render);  protected:  void Prepare(JNIEnv *env) override;  void Render(AVFrame *frame) override;  void Release(a) override;   bool NeedLoopDecode(a) override {  return true;  }   AVMediaType GetMediaType(a) override {  return AVMEDIA_TYPE_AUDIO;  }   const char *const LogSpec(a) override {  return "AUDIO";  } }; Copy the code

The above code is also not complicated, it is some initialization-related methods, and the implementation of abstract methods defined in BaseDecoder.

Focus on these two methods:

/ * ** Sampling format: 16 bits* /
AVSampleFormat GetSampleFmt(a) {
    return AV_SAMPLE_FMT_S16;
}  / * ** Target sampling rate* / int GetSampleRate(int spr) {  return AUDIO_DEST_SAMPLE_RATE; //44100Hz } Copy the code

The first thing to know is that these two methods are intended to be compatible with future coding.

We know that the sampling rate and sampling bits of the audio are unique to the audio data, and that each audio may be different, so when playing or recoding, the data is usually converted to a fixed size so that it can be played or recoded normally.

The configuration of playback and encoding is also slightly different. Here, the sampling bit is 16 bits and the sampling rate is 44100.

Next, look at the concrete implementation.

Implement decoding process

// a_decoder.cpp

AudioDecoder::AudioDecoder(JNIEnv *env, const jstring path, bool forSynthesizer) : BaseDecoder(env, path, forSynthesizer) {
    
}
 void AudioDecoder::~AudioDecoder() {  if(m_render ! =NULL) {  delete m_render;  } }  void AudioDecoder::SetRender(AudioRender *render) {  m_render = render; }  void AudioDecoder::Prepare(JNIEnv *env) {  InitSwr();  InitOutBuffer();  InitRender(); }  // omit other.... Copy the code

I. the initialization

Focus on the Prepare method, which is called after the base BaseDecoder class has initialized the decoder.

In the Prepare method, we call:

InitSwr(), initializes the converter    
InitOutBuffer(), initializes the output buffer    
InitRender(), initializes the rendererCopy the code

The following explains how to configure initialization parameters.

SwrContext configuration:

// a_decoder.cpp

void AudioDecoder::InitSwr() {
    
    // codec_cxt() is the decoding context, obtained from the subclass BaseDecoder
 AVCodecContext *codeCtx = codec_cxt();   // Initialize the format conversion tool  m_swr = swr_alloc();   // Set the input/output channel type  av_opt_set_int(m_swr, "in_channel_layout", codeCtx->channel_layout, 0);   AUDIO_DEST_CHANNEL_LAYOUT = AV_CH_LAYOUT_STEREO  av_opt_set_int(m_swr, "out_channel_layout", AUDIO_DEST_CHANNEL_LAYOUT, 0);   // Set the input/output sampling rate  av_opt_set_int(m_swr, "in_sample_rate", codeCtx->sample_rate, 0);  av_opt_set_int(m_swr, "out_sample_rate", GetSampleRate(codeCtx->sample_rate), 0);   // Configure the input/output data format  av_opt_set_sample_fmt(m_swr, "in_sample_fmt", codeCtx->sample_fmt, 0);  av_opt_set_sample_fmt(m_swr, "out_sample_fmt", GetSampleFmt(), 0);   swr_init(m_swr); } Copy the code

Initialization is simple, first call FFmpeg’s swr_alloc method, allocate memory, get a conversion tool m_SWR, then call the corresponding method, set the input and output audio data parameters.

Input and output parameters can also be set using a unified method, swr_alloc_set_opts, as described in the interface comment.

Output buffering configuration:

// a_decoder.cpp

void AudioDecoder::InitOutBuffer() {
    // Resampling the number of samples for the next channel
    m_dest_nb_sample = (int)av_rescale_rnd(ACC_NB_SAMPLES, GetSampleRate(codec_cxt()->sample_rate),
 codec_cxt()->sample_rate, AV_ROUND_UP);  // Resampling the size of the next frame  m_dest_data_size = (size_t)av_samples_get_buffer_size(  NULL, AUDIO_DEST_CHANNEL_COUNTS,  m_dest_nb_sample, GetSampleFmt(), 1);   m_out_buffer[0] = (uint8_t *) malloc(m_dest_data_size); }  void AudioDecoder::InitRender() {  m_render->InitRender(); } Copy the code

Before converting the audio data, we need a data buffer to store the converted data, so we need to know how big the converted audio data is and allocate the buffer accordingly.

There are three factors that affect the size of data buffer, namely, the number of samples, the number of channels, and the number of samples.

Sampling number calculation

We know that a frame of AAC contains 1024 samples. If a frame of audio data is resampled, the number of samples will change.

If the sampling rate increases, the number of samples will increase. When the sampling rate decreases, the number of samples decreases. And it’s proportional.

The calculation method is as follows: [Number of target samples = number of original samples * (target sampling rate/original sampling rate)]

FFmpeg provides AV_rescale_Rnd for calculating this scaling relationship, optimizing the computational benefit problem.

FFmpeg provides the av_samples_get_buffer_size method to help us calculate the size of the cache. We only need to provide the calculated number of target samples, channels and sampling bits.

Once the cache size is obtained, memory is allocated via malloc.

Ii. The rendering

// a_decoder.cpp

void AudioDecoder::Render(AVFrame *frame) {
    // return the number of samples per channel
    int ret = swr_convert(m_swr, m_out_buffer, m_dest_data_size/2. (const uint8_t **) frame->data, frame->nb_samples);  if (ret > 0) {  m_render->Render(m_out_buffer[0], (size_t) m_dest_data_size);  } } Copy the code

After subclass BaseDecoder decodes the data, call back to Render. Before rendering, the swr_convert method is called to convert the audio data.

The interface prototype is as follows:

/ * ** out: output buffer* out_count: indicates the number of output data samples in a channel* IN: The original audio data is to be converted* in_count: number of original audio samples in a single channel* / int swr_convert(struct SwrContext *s, uint8_t **out, int out_count,  const uint8_t **in , int in_count); Copy the code

Finally call m_render render play.

Iii. Release resources

// a_decoder.cpp

void AudioDecoder::Release() {
    if(m_swr ! =NULL) {
        swr_free(&m_swr);
 }  if(m_render ! =NULL) {  m_render->ReleaseRender();  }  ReleaseOutBuffer(); }  void AudioDecoder::ReleaseOutBuffer() {  if (m_out_buffer[0] != NULL) {  free(m_out_buffer[0]);  m_out_buffer[0] = NULL;  } } Copy the code

After decoding, exit playback, need to release the converter, output buffer.

2. Access OpenSL ES

Audio playback on Android, usually using AudioTrack, but in the NDK layer, there is no direct class, need to call the Java layer through the NDK, callback implementation playback. It’s relatively cumbersome and inefficient.

In the NDK layer, there is another way to play audio: OpenSL ES.

What is OpenSL ES

OpenSL ES (Open Sound Library for Embedded Systems) is a non-licensing, cross-platform, carefully optimized hardware audio acceleration API for Embedded Systems. It provides a standardized, high-performance, low-response approach to audio functionality for native application developers on embedded mobile multimedia devices, and enables direct cross-platform deployment of software/hardware audio performance, reducing execution difficulty.

What does OpenSL ES offer

OpenSL ES mainly provides the functions of recording and playing, and this paper focuses on the function of playing.

The playback sources can be PCM, SDcard, RES/Assets, and network resources.

We use FFmpeg decoding, so the playback source is PCM.

OpenSL ES state machine

OpenSL ES is a library developed based on C language, but its interface is written using object-oriented programming ideas. Its interface cannot be called directly, but must be called through objects after object creation and initialization.

The Object and Interface

OpenSL ES provides a series of objects with basic operations such as Realize, Resume, GetState, Destroy, GetInterface, and so on.

An Object has one or more Interface methods, but an Intefcace belongs to only one Obejct.

To call the Interface method in Object, you must first get the Interface from Object’s GetInterface, and then call it from the Interface obtained.

Such as:

// Create engine
SLObjectItf m_engine_obj = NULL;
SLresult result = slCreateEngine(&m_engine_obj, 0.NULL.0.NULL.NULL);

// Initialize the engine
result = (*m_engine_obj)->Realize(m_engine_obj, SL_BOOLEAN_FALSE);  // Get the engine interface SLEngineItf m_engine = NULL; result = (*m_engine_obj)->GetInterface(m_engine_obj, SL_IID_ENGINE, &m_engine); Copy the code

As you can see, objects need to be created and initialized before they can be used. This is the state machine mechanism in OpenSL ES.

After an Object is created, it enters the Unrealized state. After calling Realize(), it allocates memory resources and Realized the Realized state. Only then can Interface methods of an Object be obtained and used.

During subsequent execution, if an error occurs, the Object will enter the Suspended state. Calling Resume() can restore the Realized state.

OpenSL ES plays the initial configuration

Take a look at the official broadcast flow chart

This is a very clear illustration of how OpenSL ES works.

The two cores required for OpenSL ES playback are the Audio Player and Output Mix, both created by the OpenSL ES Engine (Creates).

Therefore, the entire initialization process can be summarized as follows:

Engine creates an Output Mix/ mixer and binds the mixer to the Audio Player as Output when creating the Audio Player.

The DataSource and DataSink

When you create an Audio Player, you need to set the data source and output target so that the Player knows how to get the playback data and where to output it for playback.

This requires the DataSource and DataSink of OpenSL ES.

typedef struct SLDataSource_ {
 void *pLocator;
 void *pFormat;
} SLDataSource;

typedef struct SLDataSink_ {  void *pLocator;  void *pFormat; } SLDataSink; Copy the code

The SLDataSource pLocator has the following types:

SLDataLocator_Address
SLDataLocator_BufferQueue
SLDataLocator_IODevice
SLDataLocator_MIDIBufferQueue
SLDataLocator_URI
Copy the code

The PCM is played using SLDataLocator_BufferQueue.

SLDataSink pLocator is generally SL_DATALOCATOR_OUTPUTMIX.

The other parameter pFormat is the format of the data.

Implement the rendering process

Before accessing OpenSL ES, define the audio rendering interface mentioned above for the convenience of specification and expansion.

// audio_render.h

class AudioRender {
public:
    virtual void InitRender(a) = 0;
 virtual void Render(uint8_t *pcm, int size) = 0;  virtual void ReleaseRender(a) = 0;  virtual ~AudioRender() {} }; Copy the code

In cmakelist.txt, open OpenSL ES support.

# CMakeList.txt

# omit other...
Cmake specifies the library to link to when compiling the target librarytarget_link_libraries(   native-lib   avutil  swresample  avcodec  avfilter  swscale  avformat  avdevice   -landroid  # Enable OpenSL ES support OpenSLES   # Links the target library to the log library  # included in the NDK.  ${log-lib} ) Copy the code

Initialize the

I. Define member variables

Define the engine, mixer, player, buffer queue interface, volume control interface, etc.

// opensl_render.h

class OpenSLRender: public AudioRender {

private:
  // Engine interface  SLObjectItf m_engine_obj = NULL;  SLEngineItf m_engine = NULL;   / / mixer  SLObjectItf m_output_mix_obj = NULL;  SLEnvironmentalReverbItf m_output_mix_evn_reverb = NULL;  SLEnvironmentalReverbSettings m_reverb_settings = SL_I3DL2_ENVIRONMENT_PRESET_DEFAULT;   / / PCM player  SLObjectItf m_pcm_player_obj = NULL;  SLPlayItf m_pcm_player = NULL;  SLVolumeItf m_pcm_player_volume = NULL;   // Buffer queue interface  SLAndroidSimpleBufferQueueItf m_pcm_buffer;   // omit other...... } Copy the code

Ii. Define relevant member methods

// opensl_render.h

class OpenSLRender: public AudioRender {

private:
  // omit member variables...   // Create engine  bool CreateEngine(a);   // Create a mixer  bool CreateOutputMixer(a);   // Create a player  bool CreatePlayer(a);   // Start rendering  void StartRender(a);   // Audio data is pressed into the buffer queue  void BlockEnqueue(a);   // Check for errors  bool CheckError(SLresult result, std: :string hint);   // Data populates the notification interface. We'll see what this method does later  void static sReadPcmBufferCbFun(SLAndroidSimpleBufferQueueItf bufferQueueItf, void *context);  public:  OpenSLRender();  ~OpenSLRender();   void InitRender(a) override;  void Render(uint8_t *pcm, int size) override;  void ReleaseRender(a) override; Copy the code

Iii. Implement the initialization process

// opensl_render.cpp

OpenSLRender::OpenSLRender() {
}

OpenSLRender::~OpenSLRender() { }  void OpenSLRender::InitRender() {  if(! CreateEngine())return;  if(! CreateOutputMixer())return;  if(! CreatePlayer())return; }  // omit other...... Copy the code

Create the engine

// opensl_render.cpp

bool OpenSLRender::CreateEngine() {
    SLresult result = slCreateEngine(&m_engine_obj, 0.NULL.0.NULL.NULL);
    if (CheckError(result, "Engine")) return false;
  result = (*m_engine_obj)->Realize(m_engine_obj, SL_BOOLEAN_FALSE);  if (CheckError(result, "Engine Realize")) return false;   result = (*m_engine_obj)->GetInterface(m_engine_obj, SL_IID_ENGINE, &m_engine);  return! CheckError(result,"Engine Interface"); } Copy the code

Create a mixer

// opensl_render.cpp

bool OpenSLRender::CreateOutputMixer() {
    SLresult result = (*m_engine)->CreateOutputMix(m_engine, &m_output_mix_obj, 1.NULL.NULL);
    if (CheckError(result, "Output Mix")) return false;
  result = (*m_output_mix_obj)->Realize(m_output_mix_obj, SL_BOOLEAN_FALSE);  if (CheckError(result, "Output Mix Realize")) return false;   return true; } Copy the code

According to the previous state machine mechanism, create engine object M_engine_obj first, and then Realize initialization, and then through GetInterface method, get engine interface m_engine.

Creating a player

// opensl_render.cpp

bool OpenSLRender::CreatePlayer() {

    // [1. Configure DataSource] ----------------------
  // Configure the PCM format  SLDataLocator_AndroidSimpleBufferQueue android_queue = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, SL_QUEUE_BUFFER_COUNT};  SLDataFormat_PCM pcm = {  SL_DATAFORMAT_PCM,// Play PCM data  (SLuint32)2.//2 audio channels (stereo)  SL_SAMPLINGRATE_44_1,/ / 44100 hz frequency  SL_PCMSAMPLEFORMAT_FIXED_16,// The number of bits is 16  SL_PCMSAMPLEFORMAT_FIXED_16,// The number is the same as the number of digits  SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT,// Stereo (front left front right)  SL_BYTEORDER_LITTLEENDIAN// End flag  };  SLDataSource slDataSource = {&android_queue, &pcm};   [2. Configure the output DataSink] ----------------------   SLDataLocator_OutputMix outputMix = {SL_DATALOCATOR_OUTPUTMIX, m_output_mix_obj};  SLDataSink slDataSink = {&outputMix, NULL};   const SLInterfaceID ids[3] = {SL_IID_BUFFERQUEUE, SL_IID_EFFECTSEND, SL_IID_VOLUME};  const SLboolean req[3] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};   // [3. Create player] ----------------------   SLresult result = (*m_engine)->CreateAudioPlayer(m_engine, &m_pcm_player_obj, &slDataSource, &slDataSink, 3, ids, req);  if (CheckError(result, "Player")) return false;   // Initialize the player  result = (*m_pcm_player_obj)->Realize(m_pcm_player_obj, SL_BOOLEAN_FALSE);  if (CheckError(result, "Player Realize")) return false;   // [4. Get player interface] ----------------------   // Get the Player interface  result = (*m_pcm_player_obj)->GetInterface(m_pcm_player_obj, SL_IID_PLAY, &m_pcm_player);  if (CheckError(result, "Player Interface")) return false;   // Get the volume interface  result = (*m_pcm_player_obj)->GetInterface(m_pcm_player_obj, SL_IID_VOLUME, &m_pcm_player_volume);  if (CheckError(result, "Player Volume Interface")) return false;   [5. Get buffer queue interface] ----------------------   // Register the callback buffer to get the buffer queue interface  result = (*m_pcm_player_obj)->GetInterface(m_pcm_player_obj, SL_IID_BUFFERQUEUE, &m_pcm_buffer);  if (CheckError(result, "Player Queue Buffer")) return false;   // Register buffer interface callback  result = (*m_pcm_buffer)->RegisterCallback(m_pcm_buffer, sReadPcmBufferCbFun, this);  if (CheckError(result, "Register Callback Interface")) return false;   LOGI(TAG, "OpenSL ES init success")   return true; } Copy the code

The initialization of the player is a bit more difficult, but it is done step by step according to the initialization process described earlier.

Configure the data source, output, and after initialization, get the playback interface, volume adjustment interface, etc.

⚠️ Note the last step, [5] in the code.

When the data source is a buffer queue, you need to obtain a buffer interface to fill the buffer with data.

So when do you populate the data? This is where the last registered callback interface comes in.

We need to register a callback function in the player. When the data in the player is finished, it will call back this method to tell us that the data is finished and we need to fill in the new data.

SReadPcmBufferCbFun is a static method. It can be inferred that OpenSL ES playback audio is an independent thread that continuously reads the data in the buffer, renders the data, and notifies us to fill in the new data through the callback interface after rendering the data.

Implement broadcast

Starting OpenSL ES rendering is as simple as calling the player’s play interface and pressing a frame into the buffer to start the rendering process.

If playing a PCM file for SDcard, just read a frame in the callback method sReadPcmBufferCbFun and fill it in.

However, it’s not that simple in our case. Remember we started a decoding thread in BaseDeocder? OpenSL ES rendering is also a separate thread, so it becomes a data synchronization issue for the two threads here.

Of course, FFmpeg can also be made into a simple decoding module that can be decoded and played in the OpenSL ES render thread, which is much easier to handle.

In order to unify the decoding process, two independent threads will be used.

I. Enable play waiting

As mentioned above, playback and decoding are two, so the data needs to be synchronized. Therefore, after initialization as OpenSL, it cannot start playing immediately, but wait for the first frame of decoded data to start playing.

In this case, the thread waits for data.

In the InitRender method above, we initialize OpenSL first, and at the end of the method, we put the playback into a wait state.

// opensl_render.cpp

OpenSLRender::OpenSLRender() {
}

OpenSLRender::~OpenSLRender() { }  void OpenSLRender::InitRender() {  if(! CreateEngine())return;  if(! CreateOutputMixer())return;  if(! ConfigPlayer())return;   // Start the thread and wait for playback  std: :thread t(sRenderPcm, this);  t.detach(); }  void OpenSLRender::sRenderPcm(OpenSLRender *that) {  that->StartRender(); }  void OpenSLRender::StartRender() {  while (m_data_queue.empty()) {  WaitForCache();  }  (*m_pcm_player)->SetPlayState(m_pcm_player, SL_PLAYSTATE_PLAYING);  sReadPcmBufferCbFun(m_pcm_buffer, this); }  // The thread enters the wait void OpenSLRender::WaitForCache() {  pthread_mutex_lock(&m_cache_mutex);  pthread_cond_wait(&m_cache_cond, &m_cache_mutex);  pthread_mutex_unlock(&m_cache_mutex); }  // Tell the thread to resume execution void OpenSLRender::SendCacheReadySignal() {  pthread_mutex_lock(&m_cache_mutex);  pthread_cond_signal(&m_cache_cond);  pthread_mutex_unlock(&m_cache_mutex); } Copy the code

The final StartRender() method is the one that is actually executed by the thread, which first checks whether the data buffer queue has data, and waits until it does.

M_data_queue is the user-defined data buffer queue, as follows:

// opensl_render.h

class OpenSLRender: public AudioRender {

private:
 / * ** Encapsulation of PCM data, mainly used to achieve data memory release* /  class PcmData {  public:  PcmData(uint8_t *pcm, int size) {  this->pcm = pcm;  this->size = size;  }  ~PcmData() {  if(pcm ! =NULL) {  // Release used memory  free(pcm);  pcm = NULL;  used = false;  }  }  uint8_t *pcm = NULL;  int size = 0;  bool used = false;  };   // Data buffer list  std: :queue<PcmData *> m_data_queue;   // omit other... } Copy the code

Ii. Data synchronization and playback

Next, take a look at how to synchronize and play data mindfully.

When initializing OpenSL, the playback callback interface sReadPcmBufferCbFun is registered at the end. Let’s look at its implementation first.

// opensl_render.cpp

void OpenSLRender::sReadPcmBufferCbFun(SLAndroidSimpleBufferQueueItf bufferQueueItf, void *context) {
    OpenSLRender *player = (OpenSLRender *)context;
    player->BlockEnqueue();
}  void OpenSLRender::BlockEnqueue() {  if (m_pcm_player == NULL) return;   // Remove the used data first  while(! m_data_queue.empty()) { PcmData *pcm = m_data_queue.front();  if (pcm->used) {  m_data_queue.pop();  delete pcm;  } else {  break;  }  }   // Wait for data buffering  while(m_data_queue.empty() && m_pcm_player ! =NULL) {// if m_pcm_player is NULL, stop render  WaitForCache();  }   PcmData *pcmData = m_data_queue.front();  if (NULL! = pcmData && m_pcm_player) { SLresult result = (*m_pcm_buffer)->Enqueue(m_pcm_buffer, pcmData->pcm, (SLuint32) pcmData->size);  if (result == SL_RESULT_SUCCESS) {  // Just make the mark that is already in use and remove it before the next frame is pressed  // Ensure that the data can be used normally; otherwise, broken tones may occur  pcmData->used = true;  }  } } Copy the code

When StartRender() waits for buffered data to arrive, it starts playing using the following method

(*m_pcm_player)->SetPlayState(m_pcm_player, SL_PLAYSTATE_PLAYING);
sReadPcmBufferCbFun(m_pcm_buffer, this);
Copy the code

At this point, the BlockEnqueue() method is finally called after a series of calls.

In this method,

First, delete the data already used in m_datA_queue to reclaim resources.

Then, it determines whether there is any buffer data that has not been played. If there is no buffer data, wait.

Finally, the data is pushed into the OpenSL queue using the (* m_PCM_buffer)->Enqueue() method.

⚠️ Note: In the following playing process, OpenSL will automatically call sReadPcmBufferCbFun to re-enter the above playing process as long as it finishes playing data.

Press in the data and start playing

Void Render(Uint8_t * PCM, int size) as defined by AudioRender

// opensl_render.cpp

void OpenSLRender::Render(uint8_t *pcm, int size) {
    if (m_pcm_player) {
        if(pcm ! =NULL && size > 0) {
 // Cache only two frames of data to avoid occupying too much memory, resulting in memory application failure and noise during playback  while (m_data_queue.size() >= 2) {  SendCacheReadySignal();  usleep(20000);  }   // Make a copy of the data and queue it  uint8_t *data = (uint8_t *) malloc(size);  memcpy(data, pcm, size);   PcmData *pcmData = new PcmData(pcm, size);  m_data_queue.push(pcmData);   // Tell the player thread to wait and resume playing  SendCacheReadySignal();  }  } else {  free(pcm);  } } Copy the code

In fact, it is very simple, the decoded data into the queue, and send a signal that the data buffer is ready, notifying the player thread can start playing.

In this way, the whole process is completed, to sum up:

Initialize theOpenSL, start the “Start playing waiting thread” and enter the playing waiting;
Press the data into the buffer queue, notify the playback thread to resume execution, enter the playback;
When playing is enabled, theOpenSLSet to play state and press a frame of data;
OpenSLAfter playing a frame of data, automatic callback notification continues to press data;
Decoding threads continually push data into the buffered queue;
In the following process, “OpenSL ES player thread” and “FFMpeg decoder thread” will execute at the same time, repeating “2 ~ 5”, and in the case of insufficient data buffer, “player thread” will wait for “decoder thread” to press into the data, and then continue to execute, until the completion of playback, both sides exit the thread.

Three, integration play

In the previous section, OpenSL ES player functions have been completed and the interface defined in AudioRander has been implemented, as long as the AudioDecoder is correctly called.

How to call them was also covered in the first section, but now you just need to integrate them into the Player to play the audio.

In player, added audio decoder and renderer:

//player.h

class Player {
private:
    VideoDecoder *m_v_decoder;
 VideoRender *m_v_render;   // Add audio decoder and renderer  AudioDecoder *m_a_decoder;  AudioRender *m_a_render;  public:  Player(JNIEnv *jniEnv, jstring path, jobject surface);  ~Player();   void play(a);  void pause(a); }; Copy the code

Instantiate audio decoders and renderers:

// player.cpp

Player::Player(JNIEnv *jniEnv, jstring path, jobject surface) {
    m_v_decoder = new VideoDecoder(jniEnv, path);
    m_v_render = new NativeRender(jniEnv, surface);
 m_v_decoder->SetRender(m_v_render);   // Instantiate the audio decoder and renderer  m_a_decoder = new AudioDecoder(jniEnv, path, false);  m_a_render = new OpenSLRender();  m_a_decoder->SetRender(m_a_render); }  Player::~Player() {  // The delete member pointer is not required here  // Threads in BaseDecoder already use smart Pointers, which are automatically released }  void Player::play() {  if(m_v_decoder ! =NULL) {  m_v_decoder->GoOn();  m_a_decoder->GoOn();  } }  void Player::pause() {  if(m_v_decoder ! =NULL) {  m_v_decoder->Pause();  m_a_decoder->Pause();  } } Copy the code

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Android FFmpeg+OpenSL ES audio decoding playback

Sound Ming

directory

First, Android audio and video hard decoding:

2. Use OpenGL to render video frames

Android FFmpeg audio and video decoding

You can read about it in this article

First, audio decoding

Implement audio decoding subclasses

2. Access OpenSL ES

What is OpenSL ES

What does OpenSL ES offer

OpenSL ES state machine

OpenSL ES plays the initial configuration

Implement the rendering process

Three, integration play

Android FFmpeg+OpenSL ES audio decoding playback

Sound Ming

directory

First, Android audio and video hard decoding:

2. Use OpenGL to render video frames

Android FFmpeg audio and video decoding

You can read about it in this article

First, audio decoding

Implement audio decoding subclasses

2. Access OpenSL ES

What is OpenSL ES

What does OpenSL ES offer

OpenSL ES state machine

OpenSL ES plays the initial configuration

Implement the rendering process

Three, integration play

Related Posts

Is it 35-year-old angst being trafficked?

Graduation project So Easy: Qomolangma FM audio radio APP

Android development & using the viewFlipper to achieve the rotation map, super simple