Recently the direction of audio and video development is very hot, many friends do not know how to get started, encountered some technical problems, here to do a technology to share.

The original address: www.jianshu.com/p/a926591e9…

What is OpenSL ES?

OpenSL ES is an open hardware audio acceleration library for embedded systems. It can also be considered as a set of audio standards for embedded platforms. The Open Sound Library for Embedded Systems provides a high-performance, low-latency approach to audio functionality, and enables cross-platform deployment of hardware and software audio performance, making it easier to develop upper-layer processing audio applications.

In Android development, Google has officially supported and extended the OpenSL ES standard since Android 2.3 (API 9). This article introduces OpenSL ES for Android NDK development.

Some basic concepts of OpenSL ES

Object-oriented interface based on C language

OpenSL ES is implemented based on C language, but the interface it provides is implemented in an object-oriented way. Most of the API of OpenSL ES is called through objects. For example, in the following code snippet, the main logic is to instantiate the engine object and get the engine object interface:

SLresult result;

// realize the engine
result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
assert(SL_RESULT_SUCCESS == result);
(void)result;

result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);
assert(SL_RESULT_SUCCESS == result);
(void)result;
Copy the code

Object and interface concepts

The two basic concepts in OpenSL ES can be compared to objects and interfaces in Java. In OpenSL ES, each Object can have a series of interfaces, and each Object provides a series of basic operations, such as Realize, GetState, Destroy, and so on. An important point is that you can use the functionality provided by Object only if you get the Interface of the Object using GetInterface.

The life cycle of an object

OpenSL ES Object generally has three states: SL_OBJECT_STATE_UNREALIZED (unavailable), SL_OBJECT_STATE_REALIZED (available), SL_OBJECT_STATE_SUSPENDED (suspended).

If an Object is in the SL_OBJECT_STATE_UNREALIZED (unavailable) state, the system does not allocate resources to it. After calling the Realize method, the system enters the SL_OBJECT_STATE_REALIZED (available) state. At this point, all functions and resources of the object can be accessed normally. OpenSL ES Object enters the SL_OBJECT_STATE_SUSPENDED state when the system audio hardware is occupied by another process. A subsequent call to the Resume method returns the object to its SL_OBJECT_STATE_REALIZED (available) state. When Object is used, the Destroy method is called to release the resource, and the Object returns to the SL_OBJECT_STATE_UNREALIZED state.

OpenSL ES Common Object and Interface

Audio engine object and interface

Audio Engine Object and Interface, namely Engine Object and SLEngineItf Interface. The Main function of the Engine Object is to manage the life cycle of the Audio Engine and provide the management interface for the Engine Object. The engine object can be used as follows:

SLresult result; Result = slCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL); assert(SL_RESULT_SUCCESS == result); (void)result; Result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE); assert(SL_RESULT_SUCCESS == result); (void)result; Result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine); assert(SL_RESULT_SUCCESS == result); (void)result; Result = (*engineObject)->Destroy(engineObject, SL_BOOLEAN_FALSE); assert(SL_RESULT_SUCCESS == result); (void)result;Copy the code

SLRecordItf and SLPlayItf

SLRecordItf and SLPlayItf abstract multimedia functions recorder and Player respectively, Create player and Recorder object instances using the CreateAudioPlayer and CreateAudioRecorder methods of SLEngineItf.

// CreateAudioRecorder object result = (*engineEngine)->CreateAudioRecorder(engineEngine, &recorderObject, &recSource, &dataSink, NUM_RECORDER_EXPLICIT_INTERFACES, iids, required); SLresult result = (*engineEngine)->CreateAudioPlayer(engineEngine, &AudioPlayerObject, &dataSource, &dataSink, 1, interfaceIDs, requiredInterfaces );Copy the code

SLDataSource and SLDataSink

OpenSL ES SLDataSource and SLDataSink constructs the Audio Player and Recorder objects, where the SLDataSource represents the source of audio data. SLDataSink Indicates audio data output information.

SLDataLocator_AndroidSimpleBufferQueue dataSou SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEU 1}; sldatalocator_androidSimpleBufferQueu 1}; // SLDataFormat_PCM dataSourceFormat = {SL_DATAFORMAT_PCM, // Wav_get_bits (wav), // Bit width WAV_get_bits (wav), SL_SPEAKER_FRONT_CENTER, SL_BYTEORDER_LITTLEENDIAN}; // SLDataSource dataSource = {&datasourcelocator, &datasourceformat}; SLDataLocator_OutputMix dataSinkLocator = {SL_DATALOCATOR_OUTPUTMIX, SL_DATALOCATOR_OUTPUTMIX, // outputMixObject // outputMixObject}; SLDataSink dataSink = {&dataSinkLocator, // locator 0,};Copy the code

OpenSL ES Recorder and Player function build

Audio Recorder

Audio Player

PS: The Data Source of the Audio Player can also be locally stored or cached Audio Data. The above picture is from Jhuster’s blog.

Code implementation

The following code mainly realizes the collection, saving and playing of audio data.

//
// Created by haohao on 2018/1/12.
//

#include <jni.h>
#include <string>
#include <assert.h>
#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>
#include <android/log.h>

#define AUDIO_SRC_PATH "/sdcard/audio.pcm"

#define LOGI(FORMAT, ...) __android_log_print(ANDROID_LOG_INFO,"haohao",FORMAT,##__VA_ARGS__);
#define LOGE(FORMAT, ...) __android_log_print(ANDROID_LOG_ERROR,"haohao",FORMAT,##__VA_ARGS__);

#define NUM_RECORDER_EXPLICIT_INTERFACES 2
#define NUM_BUFFER_QUEUE 1
#define SAMPLE_RATE 44100
#define PERIOD_TIME 20  // 20ms
#define FRAME_SIZE SAMPLE_RATE * PERIOD_TIME / 1000
#define CHANNELS 2
#define BUFFER_SIZE   (FRAME_SIZE * CHANNELS)

// engine interfaces
static SLObjectItf engineObject = NULL;
static SLEngineItf engineEngine = NULL;

// audio recorder interfaces
static SLObjectItf recorderObject = NULL;
static SLRecordItf recorderRecord = NULL;
static SLAndroidSimpleBufferQueueItf recorderBuffQueueItf = NULL;
static SLAndroidConfigurationItf configItf = NULL;

// pcm audio player interfaces
static SLObjectItf playerObject = NULL;
static SLPlayItf playerPlay = NULL;
static SLObjectItf outputMixObjext = NULL; // 混音器
static SLAndroidSimpleBufferQueueItf playerBufferQueueItf = NULL;

void createEngine(){
    SLEngineOption EngineOption[] = {
            {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE}
    };
    SLresult result;
    result = slCreateEngine(&engineObject, 1, EngineOption, 0, NULL, NULL);
    assert(SL_RESULT_SUCCESS == result);

    /* Realizing the SL Engine in synchronous mode. */
    result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);

    // get the engine interface, which is needed in order to create other objects
    result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);
    assert(SL_RESULT_SUCCESS == result);
}

class AudioContext {
public:
    FILE *pfile;
    uint8_t *buffer;
    size_t bufferSize;

    AudioContext(FILE *pfile, uint8_t *buffer, size_t bufferSize){
        this->pfile = pfile;
        this->buffer = buffer;
        this->bufferSize = bufferSize;
    }
};

static AudioContext *recorderContext = NULL;

// 录制音频时的回调
void AudioRecorderCallback(SLAndroidSimpleBufferQueueItf bufferQueueItf, void *context){
    AudioContext *recorderContext = (AudioContext*)context;
    assert(recorderContext != NULL);
    if (recorderContext->buffer != NULL) {
        fwrite(recorderContext->buffer, recorderContext->bufferSize, 1, recorderContext->pfile);
        LOGI("save a frame audio data.");
        SLresult result;
        SLuint32 state;
        result = (*recorderRecord)->GetRecordState(recorderRecord, &state);
        assert(SL_RESULT_SUCCESS == result);
        (void) result;

        if (state == SL_RECORDSTATE_RECORDING) {
            result = (*bufferQueueItf)->Enqueue(bufferQueueItf, recorderContext->buffer, recorderContext->bufferSize);
            assert(SL_RESULT_SUCCESS == result);
            (void) result;
        }
    }

}

// 播放音频时的回调
void AudioPlayerCallback(SLAndroidSimpleBufferQueueItf bufferQueueItf, void *context){
    AudioContext *playerContext = (AudioContext*)context;
    if (!feof(playerContext->pfile)) {
        fread(playerContext->buffer, playerContext->bufferSize, 1, playerContext->pfile);
        LOGI("read a frame audio data.");
        (*bufferQueueItf)->Enqueue(bufferQueueItf, playerContext->buffer, playerContext->bufferSize);
    } else {
        fclose(playerContext->pfile);
        delete playerContext->buffer;
    }
}

// 创建音频播放器
void createAudioPlayer(SLEngineItf engineEngine, SLObjectItf outputMixObject, SLObjectItf &audioPlayerObject){
    SLDataLocator_AndroidSimpleBufferQueue dataSourceLocator = {
            SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,
            1
    };

    // PCM 数据源格式
    SLDataFormat_PCM dataSourceFormat = {
            SL_DATAFORMAT_PCM,
            2,
            SL_SAMPLINGRATE_44_1,
            SL_PCMSAMPLEFORMAT_FIXED_16,
            16,
            SL_SPEAKER_FRONT_LEFT| SL_SPEAKER_FRONT_RIGHT,
            SL_BYTEORDER_LITTLEENDIAN
    };

    SLDataSource dataSource = {
            &dataSourceLocator,
            &dataSourceFormat
    };

    SLDataLocator_OutputMix dataSinkLocator = {
            SL_DATALOCATOR_OUTPUTMIX, // 定位器类型
            outputMixObject // 输出混合
    };

    SLDataSink dataSink = {
            &dataSinkLocator, // 定位器
            0,
    };

    // 需要的接口
    SLInterfaceID interfaceIDs[] = {
            SL_IID_BUFFERQUEUE
    };
    SLboolean requiredInterfaces[] = {
            SL_BOOLEAN_TRUE
    };

    // 创建音频播放对象
    SLresult result = (*engineEngine)->CreateAudioPlayer(
            engineEngine,
            &audioPlayerObject,
            &dataSource,
            &dataSink,
            1,
            interfaceIDs,
            requiredInterfaces
    );
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

}

extern "C" {

// 开始播放音频
JNIEXPORT void JNICALL
Java_com_haohao_opensl_1es_AudioRecorder_startPlay(JNIEnv *env, jobject instance) {
    // 创建引擎
    if (engineEngine == NULL) {
        createEngine();
    }

    // 创建混音器
    SLresult result;
    result = (*engineEngine)->CreateOutputMix(engineEngine, &outputMixObjext, 0, 0, 0);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    result = (*outputMixObjext)->Realize(outputMixObjext, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    FILE *p_file = fopen(AUDIO_SRC_PATH, "r");

    // 创建播放器
    createAudioPlayer(engineEngine, outputMixObjext, playerObject);

    result = (*playerObject)->Realize(playerObject, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    result = (*playerObject)->GetInterface(playerObject, SL_IID_BUFFERQUEUE,
                                                &playerBufferQueueItf);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    uint8_t *buffer = new uint8_t[BUFFER_SIZE];
    AudioContext *playerContext = new AudioContext(p_file, buffer, BUFFER_SIZE);
    result = (*playerBufferQueueItf)->RegisterCallback(playerBufferQueueItf, AudioPlayerCallback,
                                                    playerContext);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    result = (*playerObject)->GetInterface(playerObject, SL_IID_PLAY, &playerPlay);
    assert(SL_RESULT_SUCCESS == result);
    (void) result;

    result = (*playerPlay)->SetPlayState(playerPlay, SL_PLAYSTATE_PLAYING);
    assert(SL_RESULT_SUCCESS == result);

    AudioPlayerCallback(playerBufferQueueItf, playerContext);

}

// 停止播放音频
JNIEXPORT void JNICALL
Java_com_haohao_opensl_1es_AudioRecorder_stopPlay(JNIEnv *env, jobject instance) {
    if (playerPlay != NULL) {
        SLresult result;
        result = (*playerPlay)->SetPlayState(playerPlay, SL_PLAYSTATE_STOPPED);
        assert(SL_RESULT_SUCCESS == result);
    }
}

// 开始采集音频数据,并保存到本地
JNIEXPORT void JNICALL
Java_com_haohao_opensl_1es_AudioRecorder_startRecord(JNIEnv *env, jobject instance) {

    if (engineEngine == NULL) {
        createEngine();
    }

    if (recorderObject != NULL) {
        LOGI("Audio recorder already has been created.");
        return ;
    }

    FILE *p_file = fopen(AUDIO_SRC_PATH, "w");

    if (p_file == NULL) {
        LOGI("Fail to open file.");
        return ;
    }

    SLresult result;

    /* setup the data source*/
    SLDataLocator_IODevice ioDevice = {
            SL_DATALOCATOR_IODEVICE,
            SL_IODEVICE_AUDIOINPUT,
            SL_DEFAULTDEVICEID_AUDIOINPUT,
            NULL
    };

    SLDataSource recSource = {&ioDevice, NULL};

    SLDataLocator_AndroidSimpleBufferQueue recBufferQueue = {
            SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,
            NUM_BUFFER_QUEUE
    };

    SLDataFormat_PCM pcm = {
            SL_DATAFORMAT_PCM, // pcm 格式的数据
            2,  // 2 个声道(立体声)
            SL_SAMPLINGRATE_44_1, // 44100hz 的采样频率
            SL_PCMSAMPLEFORMAT_FIXED_16,
            SL_PCMSAMPLEFORMAT_FIXED_16,
            SL_SPEAKER_FRONT_LEFT| SL_SPEAKER_FRONT_RIGHT,
            SL_BYTEORDER_LITTLEENDIAN
    };

    SLDataSink dataSink = { &recBufferQueue, &pcm };
    SLInterfaceID iids[NUM_RECORDER_EXPLICIT_INTERFACES] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE, SL_IID_ANDROIDCONFIGURATION};
    SLboolean required[NUM_RECORDER_EXPLICIT_INTERFACES] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};

    /* Create the audio recorder */
    result = (*engineEngine)->CreateAudioRecorder(engineEngine, &recorderObject , &recSource, &dataSink,
                                                  NUM_RECORDER_EXPLICIT_INTERFACES, iids, required);
    assert(SL_RESULT_SUCCESS == result);


    /* get the android configuration interface*/
    result = (*recorderObject)->GetInterface(recorderObject, SL_IID_ANDROIDCONFIGURATION, &configItf);
    assert(SL_RESULT_SUCCESS == result);

    /* Realize the recorder in synchronous mode. */
    result = (*recorderObject)->Realize(recorderObject, SL_BOOLEAN_FALSE);
    assert(SL_RESULT_SUCCESS == result);

    /* Get the buffer queue interface which was explicitly requested */
    result = (*recorderObject)->GetInterface(recorderObject, SL_IID_ANDROIDSIMPLEBUFFERQUEUE, (void*) &recorderBuffQueueItf);
    assert(SL_RESULT_SUCCESS == result);


    /* get the record interface */
    result = (*recorderObject)->GetInterface(recorderObject, SL_IID_RECORD, &recorderRecord);
    assert(SL_RESULT_SUCCESS == result);

    uint8_t *buffer = new uint8_t[BUFFER_SIZE];
    recorderContext = new AudioContext(p_file, buffer, BUFFER_SIZE);
    result = (*recorderBuffQueueItf)->RegisterCallback(recorderBuffQueueItf, AudioRecorderCallback, recorderContext);
    assert(SL_RESULT_SUCCESS == result);

    /* Enqueue buffers to map the region of memory allocated to store the recorded data */
    result = (*recorderBuffQueueItf)->Enqueue(recorderBuffQueueItf, recorderContext->buffer, BUFFER_SIZE);
    assert(SL_RESULT_SUCCESS == result);

    /* Start recording */
    // 开始录制音频
    result = (*recorderRecord)->SetRecordState(recorderRecord, SL_RECORDSTATE_RECORDING);
    assert(SL_RESULT_SUCCESS == result);
    LOGI("Starting recording");

}

// 停止音频采集
JNIEXPORT void JNICALL
Java_com_haohao_opensl_1es_AudioRecorder_stopRecord(JNIEnv *env, jobject instance) {
    if (recorderRecord != NULL) {
        SLresult result = (*recorderRecord)->SetRecordState(recorderRecord, SL_RECORDSTATE_STOPPED);
        assert(SL_RESULT_SUCCESS == result);

        if (recorderContext != NULL) {
            fclose(recorderContext->pfile);
            delete recorderContext->buffer;
        }
    }
}

// 释放资源
JNIEXPORT void JNICALL
Java_com_haohao_opensl_1es_AudioRecorder_release(JNIEnv *env, jobject instance) {
    if (recorderObject != NULL) {
        (*recorderObject)->Destroy(recorderObject);
        recorderObject = NULL;
        recorderRecord = NULL;
        recorderBuffQueueItf = NULL;
        configItf = NULL;
        recorderContext = NULL;
    }

    if (playerObject != NULL) {
        (*playerObject)->Destroy(playerObject);
        playerObject = NULL;
        playerPlay = NULL;
        playerBufferQueueItf = NULL;
        outputMixObjext = NULL;
    }

    // destroy engine object, and invalidate all associated interfaces
    if (engineObject != NULL) {
        (*engineObject)->Destroy(engineObject);
        engineObject = NULL;
        engineEngine = NULL;
    }
}
};
Copy the code

CMake script cmakelists.txt.

Cmake_minimum_required (VERSION 3.4.1) add_library(# Sets the name of the library. Audio-recorder # Sets the library as a shared library. SHARED # Provides a relative path to your source file(s). src/main/cpp/audio-recorder.cpp) target_link_libraries(audio-recorder android log OpenSLES)Copy the code

Learning and Communication

Fortunately, bytedance’s internal arrangement of the “582 pages of Android NDK seven modules learning treasure, from the principle to the actual combat, everything!

Adhering to the principle of good things to share, of course, today to share a wave, I believe that this “582 pages of Android NDK seven modules learning treasure Canon” can also make your audio and video learning road twice the result with half the effort!

Treasure Book Catalog:

  • NDK module development
  • JNI module
  • Native development tools
  • Linux programming
  • Underlying image processing
  • Audio and Video development
  • Machine learning

Need friends can slide to the end of the text quickly get.

Or directly【 Click on me 】Enter the background of the public account and noteNDKGet it for free!

(PS: The information is only for open source technology sharing, we do not take to do other ha)

1. NDK module development

Main Contents:

  • C++ and C# data types summary

  • C and C++ memory structure and management

  • C and C++ preprocessing commands and typedef naming existing types

  • C and C++ structure, common body

  • C and C++ Pointers

  • C/C++ multithreading

  • C/C++ functions and initializers list

JNI module

Main Contents:

  • Static and dynamic registration developed by JNI

Static registration, dynamic registration, JNINativeMethod, data type mapping, jNI function default parameters

  • JNI developed method signature and Java communication

Android NDK develops JNI type signature and method signature, and JNI implements communication between Java and C/C ++

  • Local, global, and weak global references for JNI development

Iii. Native development tools

Main Contents:

  • Compilers, packaging tools, and analyzers

Top 10 Most Popular React Native App Development editors and React-Native Packaging processes

  • Static and dynamic libraries

  • CPU architecture and precautions

ABI management, CPU processing, NEON support

  • Build scripts and build tools

Environment setup, NDK project, Cmake, Makefile

  • Cross compilation and migration

FFmpeg compilation, FFmpeg+LIBX264+FACC cross-compilation, FFmpeg+LIBX264+FACC cross-compilation, FFmpeg+LIBX264+FACC cross-compilation, FFmpeg+LIBX264+FACC cross-compilation, FFmpeg cross-compilation, X264 FAAC cross-compilation, solve all the porting problems

  • AS builds the NDK project

Configure NDK environment, establish APP project, generate. H header file, create C file, implement native method, jni.h file

Fourth, Linux programming

  • Linux environment construction, system management, permission system and tool use (VIM, etc.)

Linux environment construction, Linux system management operation (25 commands)

  • Shell scripting

Shell script, write simple Shell script, process control statement, plan task service program

Fifth, the bottom picture processing

  • PNG/JPEG/WEBP image processing and compression

Four image formats, recommended several image processing sites, Squoosh online lossless image compression tool, JPG/webP/PNG/ interconversion

  • Wechat picture compression

Calculate the original width height, calculate the approximate width height, obtain the target image by sampling for the first time, and approach the target size by cycle

  • GIF synthesis principle and implementation

GIF image analysis, GIF image synthesis (sequence image synthesis GIF image)

Vi. Audio and video development

  • Multimedia system

Camera and mobile phone screen acquisition, image raw data format YUV420(NV21 and YV12, etc.), audio acquisition and playback system, codec MediaCodec, MediaMuxer multiplexing and MediaExtractor

  • FFmpeg

Ffmpeg module introduction, audio and video decoding, audio and video synchronization, I frame,B frame,P frame decoding principle, X264 video coding and FAAC audio coding, OpenGL rendering and NativeWindow rendering

  • Streaming media protocol

RTMP, P2P WebRtc for audio and video calls

  • OpenGL ES filter development beauty effect

Gaussian blur, high contrast retention, strong light processing, fusion

  • Analysis and implementation of Douyin video effect

Process list, video shooting, video editing, and video export

  • Audio and video transmission principle

Variable speed entrance analysis, audio variable speed, video variable speed

Machine learning

  • Opencv

  • Image preprocessing

Gray scale and binarization, corrosion and expansion, face detection, ID card recognition

The last

Due to the space limitation, the detailed information of the document is too comprehensive, too many details, so only part of the knowledge point screenshots out of the rough introduction, each small node has more detailed content!

In addition to the above I also sorted out the following series of learning progress information:

Android Development Seven modules core Knowledge Notes

2246 pages of the latest Android high Frequency interview questions

Note content is free to share, there are friends who need the complete version of notes【 Click on me 】Enter the background of the public account and noteNDKGet it for free!

(PS: The information is only for open source technology sharing, we do not take to do other ha)

The last

A journey of a thousand miles begins with a single step. You and I!