Audio and video learning (11) Android end RTMP push stream

preface

Let’s review the previous two articles on how to build the RTMP live server and how to develop an Android player with pull streaming capability. So now you have the player and the live server and you don’t have the push stream. In this article, we will implement the RTMP push stream on Android. To achieve the RTMP push stream on Android, we must go through the following stages, as shown in the figure below:

This article mainly completes the function part of the yellow color in the figure above, and then begins to get into the topic, code writing.

Effect of the project

Push the flow monitoring

Soft coding

Hard coded

At the end of the article, the hard and soft codec will be introduced.

Audio collection

Android SDK provides two sets of audio collection API, respectively MediaRecorder, AudioRecord. The former is an upper API, which can directly encode and compress the audio data input by the mobile phone microphone (such as AMR/MP3), and store it as a file. The latter is closer to the bottom, with more freedom and flexibility, allowing developers to get raw PCM audio data streams in memory. If you want to do a simple recorder, output audio files are recommended to use MediaRecorder; AudioRecord or OpenSL ES can only be used in scenarios where audio needs to be further algorithmic or encoded using a third-party code library or network transmission. In fact, MediaRecorder base is also called AudioRecord and Android Framework layer AudioFlinger interaction. The scenario in this article is more likely to be implemented in the second way, which is to use AudioRecord to collect audio.

If you want to use the AudioRecord API, you need to configure the following in the configuration file of the application Androidmanifest.xml:

 <uses-permission android:name="android.permission.RECORD_AUDIO"></uses-permission>
Copy the code

Of course, if you want to store the collected PCM original data in SDcard, you need to add additional write permission:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
Copy the code

Now take a look at the workflow of AudioRecord.

1. Initialize the AudioRecord

First, let’s look at the configuration parameters of AudioRecord. AudioRecord uses the constructor to configure parameters. Its function prototype is as follows:

public AudioRecord(int audioSource, int sampleRateInHz, int channelConfig, int audioFormat,int bufferSizeInBytes)
Copy the code

The meanings of the functions represented by the above parameters and the values that should be passed in various scenarios are described below:

AudioSource: This parameter refers to the input source of the audio capture. The optional values are defined as constants in the audioSource class (an internal class in MediaRecorder).

• The DEFAULT (the DEFAULT)
• VOICE_RECOGNITION (for speech recognition, equivalent to default)
• MIC (input from mobile phone microphone)
• VOICE_COMMUNICATION (For VOIP application scenarios)

SampleRateInHz: Used to specify the sampling frequency at which audio should be collected. The most compatible sampling frequency currently used is 44100 (44.1KHZ).

ChannelConfig: This parameter is used to specify that the recorder collects several channels of sound. Optional values are defined as constants in the AudioFormat class. Common values include:

• CHANNEL_IN_MONO (currently recommended on mobile devices)
• CHANNEL_IN_STEREO stereo

AudioFormat: The sampling format, defined as constants in the audioFormat class. Common values include:

• ENCODING_PCM_16BIT (16bit compatible with most Android phones)
• ENCODING_PCM_8BIT (8bit)

BufferSizeInBytes: Set the size of the internal audio buffer (the smaller the buffer value is, the lower the latency is). The specific size may vary from phone to phone, so use the following API to determine the buffer size:

AudioRecord.getMinBufferSize(int sampleRateInHz, int channelConfig, int audioFormat);
Copy the code

Once configured, check whether the current state of the AudioRecord is available for recording. You can obtain the current state by using AudioRecord##getState:

• STATE_UNINITIALIZED is not initialized or initialization failed
• STATE_INITIALIZED Initialized successfully.

2. Enable collection

After creating an AudioRecord, you can open the audio data collection, you can control the microphone collection by calling the following function:

mAudioRecord.startRecording();
Copy the code

3. Extract data

After the previous step, a child thread needs to be started to continuously read PCM data from the AudioRecord buffer, calling the following function to read data:

int read(@NonNull byte[] audioData, int offsetInBytes, int sizeInBytes);
Copy the code

4. Stop collection

If you want to stop the collection, you only need to call the Stop method of AudioRecord to achieve, and finally can control the child thread to stop reading data through a variable, and then call stop to stop and finally release the AudioRecord instance.

    public void stopEncode(a) {
      	// Stop the variable marker
        mStopFlag = true;
        if(mAudioEncoder ! =null) {
          	// Stop collecting
            mAudioEncoder.stop();
          	// Free memory
            mAudioEncoder = null; }}Copy the code

Video acquisition

Video picture collection is mainly achieved by using the camera API provided by each platform. After setting appropriate parameters for the camera, the video frame captured by the camera in real time is rendered on the screen for the user to preview, and then the video frame is passed to the coding channel for encoding.

1. Configure permissions

<uses-permission android:name="android.permission.CAMERA"></uses-permission>
Copy the code

2. Turn on the camera

2.1 Checking Cameras

public static void checkCameraService(Context context)
            throws CameraDisabledException {
    // Check if device policy has disabled the camera.
    DevicePolicyManager dpm = (DevicePolicyManager) context.getSystemService(
            Context.DEVICE_POLICY_SERVICE);
    if (dpm.getCameraDisabled(null)) {
        throw newCameraDisabledException(); }}Copy the code

2.2 Checking the Number of Cameras

After checking the camera service, you also need to check the number of cameras on the phone. If the number is 0, there is no camera on the phone, in which case, subsequent operations cannot be carried out.

public static List<CameraData> getAllCamerasData(boolean isBackFirst) {
    ArrayList<CameraData> cameraDatas = new ArrayList<>();
    Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
    int numberOfCameras = Camera.getNumberOfCameras();
    for (int i = 0; i < numberOfCameras; i++) {
        Camera.getCameraInfo(i, cameraInfo);
        if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
            CameraData cameraData = new CameraData(i, CameraData.FACING_FRONT);
            if(isBackFirst) {
                cameraDatas.add(cameraData);
            } else {
                cameraDatas.add(0, cameraData); }}else if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_BACK) {
            CameraData cameraData = new CameraData(i, CameraData.FACING_BACK);
            if(isBackFirst) {
                cameraDatas.add(0, cameraData);
            } else{ cameraDatas.add(cameraData); }}}return cameraDatas;
}
Copy the code

In the above method, you need to pass in a Boolean variable that indicates whether to enable the rear camera first. If the variable is true, place the rear camera first in the list. Then, when opening the camera, directly obtain the parameters related to the first camera in the list and open it. This design makes it very simple to switch the camera. When switching the camera, first close the current camera, then change the sequence in the camera list, and then open the camera again, that is, open the camera pointed to by the first camera parameter in the camera list each time.

2.3 Enabling the Camera

Before opening the camera, obtain the first camera parameter from the camera list, and then open the camera according to CameraId in the parameter. After opening the camera successfully, change the relevant state. The relevant codes are as follows:

public synchronized Camera openCamera(a)
            throws CameraHardwareException, CameraNotSupportException {
    CameraData cameraData = mCameraDatas.get(0);
    if(mCameraDevice ! =null && mCameraData == cameraData) {
        return mCameraDevice;
    }
    if(mCameraDevice ! =null) {
        releaseCamera();
    }
    try {
        Log.d(TAG, "open camera " + cameraData.cameraID);
        mCameraDevice = Camera.open(cameraData.cameraID);
    } catch (RuntimeException e) {
        Log.e(TAG, "fail to connect Camera");
        throw new CameraHardwareException(e);
    }
    if(mCameraDevice == null) {
        throw new CameraNotSupportException();
    }
    mCameraData = cameraData;
    mState = State.OPENED;
    return mCameraDevice;
}
Copy the code

Camera. Open (cameraData. CameraID) throws an exception, indicating that the Camera is not available. But on some phones camera.open (cameraData. CameraID) does not throw an exception, but returns NULL.

3. Set camera parameters

After setting parameters for the camera, you need to record them for use elsewhere. For example, if there is a flash in the current camera, you can determine whether the BUTTON to turn on the flash is displayed on the UI. Use CameraData to record these parameters in a live project. The CameraData class looks like this:

public class CameraData {
    public static final int FACING_FRONT = 1;
    public static final int FACING_BACK = 2;

    public int cameraID;            / / camera id
    public int cameraFacing;        // Distinguish between front and rear cameras
    public int cameraWidth;         // Camera capture width
    public int cameraHeight;        // Camera height
    public boolean hasLight;        // Camera has flash
    public int orientation;         // Camera rotation Angle
    public boolean supportTouchFocus;   // Whether camera supports manual focus
    public boolean touchFocusMode;      // Whether the camera is in autofocus mode

    public CameraData(int id, int facing, int width, int height){
        cameraID = id;
        cameraFacing = facing;
        cameraWidth = width;
        cameraHeight = height;
    }

    public CameraData(int id, int facing) { cameraID = id; cameraFacing = facing; }}Copy the code

When setting parameters for the camera, note that exceptions will be thrown if the parameters are invalid. Therefore, you need to set each parameter separately to avoid exceptions thrown if one parameter is invalid and all parameters are not set.

4. Enable the preview function

There are two ways to set the preview interface: 1. Display it through SurfaceView; 2. Display through GLSurfaceView When displayed for the SurfaceView, the SurfaceHolder of the SurfaceView needs to be passed to the Camera. When using GLSurfaceView, Renderer is used to render the texture. First, OpenGL generates the SurfaceTexture from the texture ID of the generated texture, and then passes the SurfaceTexture to the Camera. The texture can then be rendered in Render using the GLSurfaceView.

4.1 Setting the Preview Callback

public static void setPreviewFormat(Camera camera, Camera.Parameters parameters) {
    // Set the image format for the preview callback
    try {
        parameters.setPreviewFormat(ImageFormat.NV21);
        camera.setParameters(parameters);
    } catch(Exception e) { e.printStackTrace(); }}Copy the code

After setting the image format for the preview Callback, you need to set the Callback for the preview Callback.

Camera.PreviewCallback myCallback = new Camera.PreviewCallback() {
    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        // Get the corresponding image data
        //Do something}};public static void setPreviewCallback(Camera camera, Camera.PreviewCallback callback) {
    camera.setPreviewCallback(callback);
}
Copy the code

Android’s recommended PreViewFormat is NV21, and in PreviewCallback the Preview N21 image is returned. If it is soft editing, because H264 supports I420 image format, so you need to convert N21 format to I420 format, and then handed over to X264 encoding library. Android hardencoder supports I420(COLOR_FormatYUV420Planar) and NV12(COLOR_FormatYUV420SemiPlanar) It is therefore possible to convert an N21 image into an I420 or NV12 image and hand it to a hard encoder.

4.2 Setting the Preview Image size

In camera related processing, it is important to deal with the inconsistency between the screen display size and the camera preview size. In Android, there are a number of previewsizes for the camera, and we need to select the right PreviewSize. The code for selecting the appropriate camera PreviewSize is shown below:

public static Camera.Size getOptimalPreviewSize(Camera camera, int width, int height) {
    Camera.Size optimalSize = null;
    double minHeightDiff = Double.MAX_VALUE;
    double minWidthDiff = Double.MAX_VALUE;
    List<Camera.Size> sizes = camera.getParameters().getSupportedPreviewSizes();
    if (sizes == null) return null;
    // find the smallest width gap
    for(Camera.Size size:sizes){
        if(Math.abs(size.width - width) < minWidthDiff) { minWidthDiff = Math.abs(size.width - width); }}// Find the smallest height gap within the smallest width gap
    for(Camera.Size size:sizes){
        if(Math.abs(size.width - width) == minWidthDiff) {
            if(Math.abs(size.height - height) < minHeightDiff) { optimalSize = size; minHeightDiff = Math.abs(size.height - height); }}}return optimalSize;
}

public static void setPreviewSize(Camera camera, Camera.Size size, Camera.Parameters parameters) {
    try {    
        parameters.setPreviewSize(size.width, size.height);           
        camera.setParameters(parameters);
    } 
    catch(Exception e) { e.printStackTrace(); }}Copy the code

After setting the most appropriate PreviewSize, the size information is stored in CameraData. When you select the SurfaceView display mode, you can place the SurfaceView in a LinearLayout and then change the size of the SurfaceView according to the ratio of the camera PreviewSize, so that the ratio is consistent and the image is normal. When selecting GLSurfaceView display, you can crop the texture to keep the size ratio of the texture consistent with that of GLSurfaceView to ensure that the image display is normal.

4.3 Image Rotation

On Android, images from the camera need to be rotated before being presented to the screen, and if the app supports screen rotation, the camera Angle needs to be adjusted in real time based on the rotation. There are also two ways to rotate the camera image in Android, one is through the setDisplayOrientation(Result) method of the camera, and the other is through the OpenGL matrix for rotation. Here is the code for rotation via the setDisplayOrientation(result) method:

public static int getDisplayRotation(Activity activity) {
    int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
    switch (rotation) {
        case Surface.ROTATION_0: return 0;
        case Surface.ROTATION_90: return 90;
        case Surface.ROTATION_180: return 180;
        case Surface.ROTATION_270: return 270;
    }
    return 0;
}

public static void setCameraDisplayOrientation(Activity activity, int cameraId, Camera camera) {
    // See android.hardware.Camera.setCameraDisplayOrientation for
    // documentation.
    Camera.CameraInfo info = new Camera.CameraInfo();
    Camera.getCameraInfo(cameraId, info);
    int degrees = getDisplayRotation(activity);
    int result;
    if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        result = (info.orientation + degrees) % 360;
        result = (360 - result) % 360; // compensate the mirror
    } else { // back-facing
        result = (info.orientation - degrees + 360) % 360;
    }
    camera.setDisplayOrientation(result);
}
Copy the code

4.4 Setting the Preview Frame Rate

Through the Camera. The Parameters of getSupportedPreviewFpsRange () can get the Camera support frame rate range, choose the appropriate Settings to the Camera. The relevant code is as follows:

public static void setCameraFps(Camera camera, int fps) {
    Camera.Parameters params = camera.getParameters();
    int[] range = adaptPreviewFps(fps, params.getSupportedPreviewFpsRange());
    params.setPreviewFpsRange(range[0], range[1]);
    camera.setParameters(params);
}

private static int[] adaptPreviewFps(int expectedFps, List<int[]> fpsRanges) {
    expectedFps *= 1000;
    int[] closestRange = fpsRanges.get(0);
    int measure = Math.abs(closestRange[0] - expectedFps) + Math.abs(closestRange[1] - expectedFps);
    for (int[] range : fpsRanges) {
        if (range[0] <= expectedFps && range[1] >= expectedFps) {
            int curMeasure = Math.abs(range[0] - expectedFps) + Math.abs(range[1] - expectedFps);
            if(curMeasure < measure) { closestRange = range; measure = curMeasure; }}}return closestRange;
}
Copy the code

4.5 Setting the Camera focus

Generally, there are two ways to focus a camera: manual focus and touch focus. The following code sets the autofocus and touch focus modes respectively:

public static void setAutoFocusMode(Camera camera) {
    try {
        Camera.Parameters parameters = camera.getParameters();
        List<String> focusModes = parameters.getSupportedFocusModes();
        if (focusModes.size() > 0 && focusModes.contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_PICTURE)) {
            parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_PICTURE);
            camera.setParameters(parameters);
        } else if (focusModes.size() > 0) {
            parameters.setFocusMode(focusModes.get(0)); camera.setParameters(parameters); }}catch(Exception e) { e.printStackTrace(); }}public static void setTouchFocusMode(Camera camera) {
    try {
        Camera.Parameters parameters = camera.getParameters();
        List<String> focusModes = parameters.getSupportedFocusModes();
        if (focusModes.size() > 0&& focusModes.contains(Camera.Parameters.FOCUS_MODE_AUTO)) { parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);  camera.setParameters(parameters); }else if (focusModes.size() > 0) {
            parameters.setFocusMode(focusModes.get(0)); camera.setParameters(parameters); }}catch(Exception e) { e.printStackTrace(); }}Copy the code

For autofocus this is done, but for touch focus you need to set the corresponding focus area. To set up the focus area accurately, there are three steps: 1. 2. Convert the coordinate position to the coordinate system on the camera preview interface by clicking the coordinate position; Third, generate the focus area according to the coordinates and set it to the camera. The entire camera preview interface defines the following coordinate system, and the focusing area also needs to correspond to this coordinate system.

If the camera preview interface is displayed through the SurfaceView, it is relatively simple. To ensure no deformation, the SurfaceView will be stretched so that the size ratio of the SurfaceView and the preview image is consistent. Therefore, the whole SurfaceView is equivalent to the preview interface. All you need to do is get the coordinates of the current click point on the whole SurfaceView, and then convert it to the corresponding focus area. If the camera preview interface is shown by GLSurfaceView is a bit more complicated, because of the texture to cut, to make the display not deformation, so, we should restore the whole preview the size of the interface, and then through the click on the location of the conversion into preview interface on the coordinate system of coordinates, and then get the corresponding focus area, And then set it up for the camera. Once the focus area is set, touch focus can be accomplished by calling the Camera’s autoFocus() method. The whole process is more code, please read the project source code.

4.6 Setting the Zoom

When a pinch is detected, we tend to want the camera to be zoomed as well, but this is relatively simple to implement. First of all, we need to add gesture recognition of zoom. When we recognize the gesture of zoom, we can scale the camera according to the size of the zoom. The code looks like this:

/** * Handles the pinch-to-zoom gesture */
private class ZoomGestureListener extends ScaleGestureDetector.SimpleOnScaleGestureListener {
    @Override
    public boolean onScale(ScaleGestureDetector detector) {
        if(! mIsFocusing) {float progress = 0;
            if (detector.getScaleFactor() > 1.0 f) {
                progress = CameraHolder.instance().cameraZoom(true);
            } else if (detector.getScaleFactor() < 1.0 f) {
                progress = CameraHolder.instance().cameraZoom(false);
            } else {
                return false;
            }
            if(mZoomListener ! =null) { mZoomListener.onZoomProgress(progress); }}return true; }}public float cameraZoom(boolean isBig) {
    if(mState ! = State.PREVIEW || mCameraDevice ==null || mCameraData == null) {
        return -1;
    }
    Camera.Parameters params = mCameraDevice.getParameters();
    if(isBig) {
        params.setZoom(Math.min(params.getZoom() + 1, params.getMaxZoom()));
    } else {
        params.setZoom(Math.max(params.getZoom() - 1.0));
    }
    mCameraDevice.setParameters(params);
    return (float) params.getZoom()/params.getMaxZoom();
}
Copy the code

4.7 Flashlight Operation

A camera may or may not have a corresponding flash, so make sure you have a corresponding flash before using it. The code to check whether the camera has a flash is as follows:

public static boolean supportFlash(Camera camera){
    Camera.Parameters params = camera.getParameters();
    List<String> flashModes = params.getSupportedFlashModes();
    if(flashModes == null) {
        return false;
    }
    for(String flashMode : flashModes) {
        if(Camera.Parameters.FLASH_MODE_TORCH.equals(flashMode)) {
            return true; }}return false;
}
Copy the code

The code for switching the flash is as follows:

public static void switchLight(Camera camera, Camera.Parameters cameraParameters) {
    if (cameraParameters.getFlashMode().equals(Camera.Parameters.FLASH_MODE_OFF)) {
        cameraParameters.setFlashMode(Camera.Parameters.FLASH_MODE_TORCH);
    } else {
        cameraParameters.setFlashMode(Camera.Parameters.FLASH_MODE_OFF);
    }
    try {
        camera.setParameters(cameraParameters);
    }catch(Exception e) { e.printStackTrace(); }}Copy the code

4.8 Starting a Preview

When the Camera is turned on and camera-related parameters are set, you can start the preview by calling the startPreview() method of the Camera. The surfaceHolder.callback can be set to open the camera and preview when the interface is displayed, stop the preview and close the camera when the interface is destroyed. That way, when the application goes back to the background, other applications can call the camera.

private SurfaceHolder.Callback mSurfaceHolderCallback = new SurfaceHolder.Callback() {
    @Override
    public void surfaceDestroyed(SurfaceHolder holder) {
        Log.d(SopCastConstant.TAG, "SurfaceView destroy");
        CameraHolder.instance().stopPreview();
        CameraHolder.instance().releaseCamera();
    }

    @TargetApi(Build.VERSION_CODES.GINGERBREAD)
    @Override
    public void surfaceCreated(SurfaceHolder holder) {
        Log.d(SopCastConstant.TAG, "SurfaceView created");
    }

    @Override
    public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
     Log.d(SopCastConstant.TAG, "SurfaceView width:" + width + " height:"+ height); CameraHolder.instance().openCamera(); CameraHolder.instance().startPreview(); }};Copy the code

5. Stop the preview

To stop the preview, just release the camera resources:

    public synchronized void releaseCamera(a) {
        if (mState == State.PREVIEW) {
            stopPreview();
        }
        if(mState ! = State.OPENED) {return;
        }
        if (mCameraDevice == null) {
            return;
        }
        mCameraDevice.release();
        mCameraDevice = null;
        mCameraData = null;
        mState = State.INIT;
    }
Copy the code

Audio coding

After AudioRecord collection, PCM data needs to be encoded in real time (software coding uses libfaac to cross-compile static library through NDK, hard coding uses Android SDK MediaCodec to encode).

Soft plait

Here we use libfaac to encode AAC speech data.

1. Compile libfaac

1.1 download libfaac

Wget HTTP: / / https://sourceforge.net/projects/faac/files/faac-src/faac-1.29/faac-1.29.9.2.tar.gzCopy the code

1.2 Writing cross-compilation scripts

#! /bin/bash

#Packaged address
PREFIX=`pwd`/android/armeabi-v7a
#Configure NDK environment variables
NDK_ROOT=$NDK_HOME
#Specify the CPU
CPU=arm-linux-androideabi
#Specify the Android API
ANDROID_API=17
#Compile toolchain directoryTOOLCHAIN = $NDK_ROOT/toolchains / $4.9 / prebuilt CPU/Linux - x86_64 FLAGS = "- isysroot $NDK_ROOT/sysroot - isystem $NDK_ROOT/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=$ANDROID_API -U_FILE_OFFSET_BITS -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -O0 -fPIC" CROSS_COMPILE=$TOOLCHAIN/bin/arm-linux-androideabi export CC="$CROSS_COMPILE-gcc --sysroot=$NDK_ROOT/platforms/android-17/arch-arm" export CFLAGS="$FLAGS" ./configure \ --prefix=$PREFIX \ --host=arm-linux \ --with-pic \ --enable-shared=no make clean make installCopy the code

2. CMakeLists. TXT configuration

cmake_minimum_required(VERSION 3.4.1)
# Voice encoder
set(faac ${CMAKE_SOURCE_DIR}/faac)
Load the faAC header directory
include_directories(${faac}/include)
Faac static library file directory
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -L${faac}/libs/${CMAKE_ANDROID_ARCH_ABI}")
# Batch add your own CPP files, do not add *. H
file(GLOB Push_CPP ${ykpusher}/*.cpp)
# Add your own CPP source files to generate dynamic library
add_library(ykpusher SHARED ${Push_CPP})
# find NDK log library in system
find_library(log_lib
        log)
Push flow so #
target_link_libraries(
        # play so
        ykpusher
Don't worry about adding ffmpeg lib order and causing application crash
# -Wl,--start-group
# avcodec avfilter avformat avutil swresample swscale
# -Wl,--end-group
# z
        # flow library
        rtmp
        # Video encoding
        x264
        # Speech coding
        faac
        # local library
        android
        ${log_lib}
        )

Copy the code

3. Set FAAC code parameters

// Set voice soft coding parameters
void AudioEncoderChannel::setAudioEncoderInfo(int samplesHZ, int channel) {
  	// If it is already initialized, it needs to be released
    release();
    // Channel default mono
    mChannels = channel;
    // Open the encoder
    // The maximum number of samples that can be input to the encoder at one time is also the number of data encoded (a sample is 16 bits 2 bytes).
    //4. The maximum number of possible bytes of encoded output data
    mAudioCodec = faacEncOpen(samplesHZ, channel, &mInputSamples, &mMaxOutputBytes);
    if(! mAudioCodec) {if (mIPushCallback) {
            mIPushCallback->onError(THREAD_MAIN, FAAC_ENC_OPEN_ERROR);
        }
        return;
    }

    // Set encoder parameters
    faacEncConfigurationPtr config = faacEncGetCurrentConfiguration(mAudioCodec);
    // Specify the mPEG4 standard
    config->mpegVersion = MPEG4;
    / / lc
    config->aacObjectType = LOW;
    / / 16
    config->inputFormat = FAAC_INPUT_16BIT;
    // Encode the raw data is neither ADTS nor ADIF
    config->outputFormat = 0;
    faacEncSetConfiguration(mAudioCodec, config);
    // The encoded data in the output buffer is stored in this buffer
    mBuffer = new u_char[mMaxOutputBytes];
  	// Sets a flag to enable encoding
    isStart = true;
}
Copy the code

4. Configure the AAC header

When sending RTMP audio and video packets, the voice packet header must be sent first

/** * audio header data * @return */
RTMPPacket *AudioEncoderChannel::getAudioTag() {
    if(! mAudioCodec) { setAudioEncoderInfo(FAAC_DEFAUTE_SAMPLE_RATE, FAAC_DEFAUTE_SAMPLE_CHANNEL);if(! mAudioCodec)return 0;
    }
    u_char *buf;
    u_long len;
    faacEncGetDecoderSpecificInfo(mAudioCodec, &buf, &len);
    int bodySize = 2 + len;
    RTMPPacket *packet = new RTMPPacket;
    RTMPPacket_Alloc(packet, bodySize);
    / / double track
    packet->m_body[0] = 0xAF;
    if (mChannels == 1) { / / single
        packet->m_body[0] = 0xAE;
    }
    packet->m_body[1] = 0x00;
    // Copy the header data to RTMPPacket
    memcpy(&packet->m_body[2], buf, len);
		// Whether to use absolute timestamp
    packet->m_hasAbsTimestamp = FALSE;
  	/ / package size
    packet->m_nBodySize = bodySize;
  	/ / package type
    packet->m_packetType = RTMP_PACKET_TYPE_AUDIO;
  	// Voice channel
    packet->m_nChannel = 0x11;
    packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
    return packet;
}
Copy the code

5. Start real-time coding

void AudioEncoderChannel::encodeData(int8_t *data) {
    if(! mAudioCodec || ! isStart)// Do not meet the encoding requirements, exit
        return;
    // Return the length of encoded data bytes
    int bytelen = faacEncEncode(mAudioCodec, reinterpret_cast<int32_t *>(data), mInputSamples,mBuffer, mMaxOutputBytes);
    if (bytelen > 0) {
        // Start packing RTMP
        int bodySize = 2 + bytelen;
        RTMPPacket *packet = new RTMPPacket;
        RTMPPacket_Alloc(packet, bodySize);
        / / double track
        packet->m_body[0] = 0xAF;
        if (mChannels == 1) {
            packet->m_body[0] = 0xAE;
        }
        // The encoded audio is 0x01
        packet->m_body[1] = 0x01;
        memcpy(&packet->m_body[2], mBuffer, bytelen);

        packet->m_hasAbsTimestamp = FALSE;
        packet->m_nBodySize = bodySize;
        packet->m_packetType = RTMP_PACKET_TYPE_AUDIO;
        packet->m_nChannel = 0x11;
        packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
        // Send the RTMP packet and send the callback to the RTMP send modulemAudioCallback(packet); }}Copy the code

6. Release the encoder

The encoder needs to be released actively to release native memory when no encoding is needed or the encoding exits. The operation of releasing the encoder can be realized through the following functions:

void AudioEncoderChannel::release() {
  	// Exit the encoding flag
    isStart = false;
    // Release the encoder
    if (mAudioCodec) {
      	// Close the encoder
        faacEncClose(mAudioCodec);
      	// Release the buffer
      	DELETE(mBuffer);
        mAudioCodec = 0; }}Copy the code

Hard make

The following uses the MediaCodec function of Android SDK to encode PCM into AAC format audio data. The use of MediaCodec encoding AAC is required for The Android system, which must be above 4.1, i.e. the Android version code must be above build.version_codes.jelly_bean (16). MediaCodec is a hardware encoder provided by The Android system. It can use the hardware of the device to complete the encoding, thus greatly improving the encoding efficiency and reducing the power consumption. However, it is not as compatible as the soft numbering, because the Locking chip of Android devices is too serious. So you can decide for yourself whether to use the hardware coding features of the Android platform in your application.

1. Create"audio/mp4a-latm"Type of hard encoder

 mediaCodec = MediaCodec.createEncoderByType(configuration.mime);    
Copy the code

2. Configure the audio encoder

    public static MediaCodec getAudioMediaCodec(AudioConfiguration configuration){
        MediaFormat format = MediaFormat.createAudioFormat(configuration.mime, configuration.frequency, configuration.channelCount);
        if(configuration.mime.equals(AudioConfiguration.DEFAULT_MIME)) {
            format.setInteger(MediaFormat.KEY_AAC_PROFILE, configuration.aacProfile);
        }
      	// Voice bit rate
        format.setInteger(MediaFormat.KEY_BIT_RATE, configuration.maxBps * 1024);
      	// Voice sampling rate 44100
        format.setInteger(MediaFormat.KEY_SAMPLE_RATE, configuration.frequency);
        int maxInputSize = AudioUtils.getRecordBufferSize(configuration);
        format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, maxInputSize);
        format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, configuration.channelCount);

        MediaCodec mediaCodec = null;
        try {
            mediaCodec = MediaCodec.createEncoderByType(configuration.mime);
          	// mediacodec. CONFIGURE_FLAG_ENCODE
            mediaCodec.configure(format, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        } catch (Exception e) {
            e.printStackTrace();
            if(mediaCodec ! =null) {
                mediaCodec.stop();
                mediaCodec.release();
                mediaCodec = null; }}return mediaCodec;
    }
Copy the code

3. Enable the audio encoder

void prepareEncoder(a) {
   mMediaCodec = AudioMediaCodec.getAudioMediaCodec(mAudioConfiguration);
   mMediaCodec.start();
}
Copy the code

4. Get hard coded input (PCM) output (AAC) ByteBufferer

At this point, the audio encoder is configured and enabled successfully. Now you can retrieve two buffers from your MediaCodec instance. One is the input buffer and the other is the output buffer. The input buffer is similar to the AVFrame in FFmpeg to store the PCM data to be encoded. The output buffer is similar to THE AAC data encoded by AVPacket of FFmpeg, and its code is as follows:

// Store PCM data
ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
// Store AAC data after encoding
ByteBuffer[] outputBuffers = mMediaCodec.getOutputBuffers();
Copy the code

5. Start hard coding PCM to AAC

At this point, all the initialization methods have been implemented, so let’s look at how MediaCodec works as shown below. The Client element on the left represents putting the PCM into a specific buffer in the inputBuffer. The Client element on the right represents extracting the encoded raw AAC data from a specific buffer in the outputBuffer, the little squares to the left of 👈 represent the individual inputBuffer elements, The small squares on the right represent each outputBuffer element. See the MediaCodec class introduction for more details.

The specific implementation of the code is as follows:

  //input:PCM  
	synchronized void offerEncoder(byte[] input) {
        if(mMediaCodec == null) {
            return;
        }
        ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
        ByteBuffer[] outputBuffers = mMediaCodec.getOutputBuffers();
        int inputBufferIndex = mMediaCodec.dequeueInputBuffer(12000);
        if (inputBufferIndex >= 0) {
            ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
            inputBuffer.clear();
            inputBuffer.put(input);
            mMediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, 0.0);
        }

        int outputBufferIndex = mMediaCodec.dequeueOutputBuffer(mBufferInfo, 12000);
        while (outputBufferIndex >= 0) {
            ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
            if(mListener ! =null) {
              	// Call back the AAC data
                mListener.onAudioEncode(outputBuffer, mBufferInfo);
            }
          	// Release the current internal encoding memory
            mMediaCodec.releaseOutputBuffer(outputBufferIndex, false);
            outputBufferIndex = mMediaCodec.dequeueOutputBuffer(mBufferInfo, 0); }}Copy the code

6. AAC is packaged as FLV

    @Override
    public void onAudioData(ByteBuffer bb, MediaCodec.BufferInfo bi) {
        if (packetListener == null| |! isHeaderWrite || ! isKeyFrameWrite) {return;
        }
        bb.position(bi.offset);
        bb.limit(bi.offset + bi.size);

        byte[] audio = new byte[bi.size];
        bb.get(audio);
        int size = AUDIO_HEADER_SIZE + audio.length;
        ByteBuffer buffer = ByteBuffer.allocate(size);
        FlvPackerHelper.writeAudioTag(buffer, audio, false, mAudioSampleSize);
        packetListener.onPacket(buffer.array(), AUDIO);
    }

    public static void writeAudioTag(ByteBuffer buffer, byte[] audioInfo, boolean isFirst, int audioSize) {
        // Write the audio header
        writeAudioHeader(buffer, isFirst, audioSize);

        // Write the audio message
        buffer.put(audioInfo);
    }
Copy the code

7. Release the encoder

After using the MediaCodec encoder, you need to stop running and release the encoder as follows:

    synchronized public void stop(a) {
        if(mMediaCodec ! =null) {
            mMediaCodec.stop();
            mMediaCodec.release();
            mMediaCodec = null; }}Copy the code

Video coding

After Camera acquisition, YUV data need to be encoded in real time (x264 is used for soft programming to cross-compile static library through NDK, and Android SDK MediaCodec is used for hard programming to encode).

Soft plait

Video software coding here we use the mainstream code library X264 to encode H264 video format data.

1. Cross-compile X264

1.1 download the x264

/ / a git clone 2 wget HTTP: / / https://code.videolan.org/videolan/x264.git / / way ftp://ftp.videolan.org/pub/x264/snapshots/last_x264.tar.bz2Copy the code

1.2 Compiling scripts

-werror =implicit-function-declaration =implicit-function-declaration

The cross-compile script is as follows:

#! /bin/bash

#Packaged address
PREFIX=./android/armeabi-v7a

#Configure NDK environment variables
NDK_ROOT=$NDK_HOME

#Specify the CPU
CPU=arm-linux-androideabi

#Specify the Android APIANDROID_API=17 TOOLCHAIN=$NDK_ROOT/toolchains/$CPU-4.9/prebuilt/linux-x86_64 FLAGS="-isysroot $NDK_ROOT/sysroot -isystem  $NDK_ROOT/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=$ANDROID_API -U_FILE_OFFSET_BITS -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -O0 -fPIC"
#--disable- CLI does not require command line tools
#- enable - static static library


./configure \
--prefix=$PREFIX \
--disable-cli \
--enable-static \
--enable-pic \
--host=arm-linux \
--cross-prefix=$TOOLCHAIN/bin/arm-linux-androideabi- \
--sysroot=$NDK_ROOT/platforms/android-17/arch-arm \
--extra-cflags="$FLAGS"

make clean
make install
Copy the code

2. CMakeList. TXT configuration

cmake_minimum_required(VERSION 3.4.1)

# Video encoder
set(x264 ${CMAKE_SOURCE_DIR}/x264)

Load the x264 header file directory
include_directories(${x264}/include)

X264 static library file directory
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -L${x264}/libs/${CMAKE_ANDROID_ARCH_ABI}")

# Batch add your own CPP files, do not add *. H
file(GLOB Player_CPP ${ykplayer}/*.cpp)
file(GLOB Push_CPP ${ykpusher}/*.cpp)
# Add your own CPP source files to generate dynamic library
add_library(ykpusher SHARED ${Push_CPP})

# find NDK log library in system
find_library(log_lib
        log)

Push flow so #
target_link_libraries(
        # play so
        ykpusher
Don't worry about adding ffmpeg lib order and causing application crash
# -Wl,--start-group
# avcodec avfilter avformat avutil swresample swscale
# -Wl,--end-group
# z
        # flow library
        rtmp
        # Video encoding
        x264
        # Speech coding
        faac
        # local library
        android
        ${log_lib}
        )
Copy the code

3. Configure and enable the X264 encoder

void VideoEncoderChannel::setVideoEncoderInfo(int width, int height, int fps, int bit) {
    pthread_mutex_lock(&mMutex);
    this->mWidth = width;
    this->mHeight = height;
    this->mFps = fps;
    this->mBit = bit;
    this->mY_Size = width * height;
    this->mUV_Size = mY_Size / 4;

    // If the encoder already exists, it needs to be released
    if (mVideoCodec || pic_in) {
        release();
    }
    // Open the X264 encoder
    // Attributes of the x264 encoder
    x264_param_t param;
    / / 2: fastest
    //3: no delay encoding
    x264_param_default_preset(&param, x264_preset_names[0], x264_tune_names[7]);
    //base_line 3.2 encoding specifications
    param.i_level_idc = 32;
    // Input data format
    param.i_csp = X264_CSP_I420;
    param.i_width = width;
    param.i_height = height;
    / / no b frames
    param.i_bframe = 0;
    // i_rc_method indicates bit rate control, CQP(constant mass), CRF(constant bit rate), ABR(average bit rate)
    param.rc.i_rc_method = X264_RC_ABR;
    // Bit rate (bit rate in Kbps)
    param.rc.i_bitrate = mBit;
    // Instantaneous maximum bit rate
    param.rc.i_vbv_max_bitrate = mBit * 1.2;
    // set i_vbv_max_bitrate This parameter must be set, bitrate control area size in KBPS
    param.rc.i_vbv_buffer_size = mBit;

    / / frame rate
    param.i_fps_num = fps;
    param.i_fps_den = 1;
    param.i_timebase_den = param.i_fps_num;
    param.i_timebase_num = param.i_fps_den;
// param.pf_log = x264_log_default2;
    // Use FPS instead of timestamps to calculate the distance between frames
    param.b_vfr_input = 0;
    // Frame distance (keyframe) 2s one keyframe
    param.i_keyint_max = fps * 2;
    // Whether to copy SPS and PPS before each keyframe this parameter is set so that each keyframe (I frame) has SPS/PPS attached.
    param.b_repeat_headers = 1;
    / / multi-threaded
    param.i_threads = 1;

    x264_param_apply_profile(&param, "baseline");
    // Open the encoder
    mVideoCodec = x264_encoder_open(&param);
    pic_in = new x264_picture_t;
    x264_picture_alloc(pic_in, X264_CSP_I420, width, height);
    // Restart the encoder
    isStart = true;
    pthread_mutex_unlock(&mMutex);
}
Copy the code

4. Start coding

void VideoEncoderChannel::onEncoder() {
    while (isStart) {
        if(! mVideoCodec) {continue;
        }
        int8_t *data = 0;
        mVideoPackets.pop(data);
        if(! data) { LOGE("Error obtaining YUV data");
            continue;
        }
        / / copy Y data
        memcpy(this->pic_in->img.plane[0], data, mY_Size);
        // Get the UV data
        for (int i = 0; i < mUV_Size; ++i) {
            // Get the u data
            *(pic_in->img.plane[1] + i) = *(data + mY_Size + i * 2 + 1);
            // Get v data
            *(pic_in->img.plane[2] + i) = *(data + mY_Size + i * 2);
        }
        // The encoded data
        x264_nal_t *pp_nal;
        // The number of frames encoded
        int pi_nal = 0;
        x264_picture_t pic_out;
        // Start encoding
        int ret = x264_encoder_encode(mVideoCodec, &pp_nal, &pi_nal, pic_in, &pic_out);
        if(! ret) { LOGE("Encoding failed");
            continue;
        }
        // If it is a keyframe
        int sps_len = 0;
        int pps_len = 0;
        uint8_t sps[100];
        uint8_t pps[100];
        for (int i = 0; i < pi_nal; ++i) {
            if (pp_nal[i].i_type == NAL_SPS) {
                // Exclude h264 interval 00 00 00 01
                sps_len = pp_nal[i].i_payload - 4;
                memcpy(sps, pp_nal[i].p_payload + 4, sps_len);
            } else if (pp_nal[i].i_type == NAL_PPS) {
                pps_len = pp_nal[i].i_payload - 4;
                memcpy(pps, pp_nal[i].p_payload + 4, pps_len);
                // PPS must follow SPS
                sendSpsPps(sps, pps, sps_len, pps_len);
            } else {
              	// Encode H264 data
                sendFrame(pp_nal[i].i_type, pp_nal[i].p_payload, pp_nal[i].i_payload, 0); }}}}/** * send SPS PPS * @param SPS codes the first frame data * @Param PPS codes the second frame data * @param SPs_len codes the first frame data length * @param PPs_len codes the second frame data length */
void VideoEncoderChannel::sendSpsPps(uint8_t *sps, uint8_t *pps, int sps_len, int pps_len) {
    int bodySize = 13 + sps_len + 3 + pps_len;
    RTMPPacket *packet = new RTMPPacket;
    //
    RTMPPacket_Alloc(packet, bodySize);
    int i = 0;
    / / fixed head
    packet->m_body[i++] = 0x17;
    / / type
    packet->m_body[i++] = 0x00;
    //composition time 0x000000
    packet->m_body[i++] = 0x00;
    packet->m_body[i++] = 0x00;
    packet->m_body[i++] = 0x00;

    / / version
    packet->m_body[i++] = 0x01;
    // Code specification
    packet->m_body[i++] = sps[1];
    packet->m_body[i++] = sps[2];
    packet->m_body[i++] = sps[3];
    packet->m_body[i++] = 0xFF;

    / / the whole SPS
    packet->m_body[i++] = 0xE1;
    / / SPS length
    packet->m_body[i++] = (sps_len >> 8) & 0xff;
    packet->m_body[i++] = sps_len & 0xff;
    memcpy(&packet->m_body[i], sps, sps_len);
    i += sps_len;

    //pps
    packet->m_body[i++] = 0x01;
    packet->m_body[i++] = (pps_len >> 8) & 0xff;
    packet->m_body[i++] = (pps_len) & 0xff;
    memcpy(&packet->m_body[i], pps, pps_len);

    / / video
    packet->m_packetType = RTMP_PACKET_TYPE_VIDEO;
    packet->m_nBodySize = bodySize;
    // Allocate a pipe randomly (avoid rtmp.c)
    packet->m_nChannel = 0x10;
    // SPS PPS have no timestamp
    packet->m_nTimeStamp = 0;
    // Do not use absolute time
    packet->m_hasAbsTimestamp = 0;
    packet->m_headerType = RTMP_PACKET_SIZE_MEDIUM;
    if (mVideoCallback && isStart)
        mVideoCallback(packet);
}

/** * send video frame -- keyframe * @param type * @param payload * @param i_playload */
void VideoEncoderChannel::sendFrame(int type, uint8_t *payload, int i_payload, long timestamp) {
    if (payload[2] = =0x00) {
        i_payload -= 4;
        payload += 4;
    } else {
        i_payload -= 3;
        payload += 3;
    }
    / / watch
    int bodySize = 9 + i_payload;
    RTMPPacket *packet = new RTMPPacket;
    //
    RTMPPacket_Alloc(packet, bodySize);

    packet->m_body[0] = 0x27;
    if (type == NAL_SLICE_IDR) {
        packet->m_body[0] = 0x17;
        LOGE("Keyframe");
    }
    / / type
    packet->m_body[1] = 0x01;
    / / timestamp
    packet->m_body[2] = 0x00;
    packet->m_body[3] = 0x00;
    packet->m_body[4] = 0x00;
    // Data length int 4 bytes
    packet->m_body[5] = (i_payload >> 24) & 0xff;
    packet->m_body[6] = (i_payload >> 16) & 0xff;
    packet->m_body[7] = (i_payload >> 8) & 0xff;
    packet->m_body[8] = (i_payload) & 0xff;

    // Image data
    memcpy(&packet->m_body[9], payload, i_payload);

    packet->m_hasAbsTimestamp = 0;
    packet->m_nBodySize = bodySize;
    packet->m_packetType = RTMP_PACKET_TYPE_VIDEO;
    packet->m_nChannel = 0x10;
    packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
    if (mVideoCallback && isStart)
        mVideoCallback(packet);// Callback to the RTMP module
}
Copy the code

5. Release the encoder

We need to release the encoder when we no longer need to encode, the code is as follows:

x264_encoder_close(mVideoCodec);
Copy the code

Hard make

After Android 4.3 system, video coding with MediaCodec has become the mainstream usage scenario. Although The fragmentation of Android is very serious, which may lead to some compatibility problems, the performance and speed of hardware encoder is very impressive. In addition, after 4.3 system, the input of the encoder can be configured through Surface, which greatly reduces the time used in the process of switching from video memory to memory, thus greatly improving the experience of the whole application. Now that the inputs and outputs are determined, the process of encoding video frames by MediaCodec will be written directly.

1. Createvideo/avcType of hard encoder

 mediaCodec = MediaCodec.createEncoderByType(videoConfiguration.mime);
Copy the code

2. Configure the video encoder

    public static MediaCodec getVideoMediaCodec(VideoConfiguration videoConfiguration) {
        int videoWidth = getVideoSize(videoConfiguration.width);
        int videoHeight = getVideoSize(videoConfiguration.height);
        MediaFormat format = MediaFormat.createVideoFormat(videoConfiguration.mime, videoWidth, videoHeight);
        format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
                MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
        format.setInteger(MediaFormat.KEY_BIT_RATE, videoConfiguration.maxBps* 1024);
        int fps = videoConfiguration.fps;
        // Set the camera preview frame rate
        if(BlackListHelper.deviceInFpsBlacklisted()) {
            SopCastLog.d(SopCastConstant.TAG, "Device in fps setting black list, so set mediacodec fps 15");
            fps = 15;
        }
        format.setInteger(MediaFormat.KEY_FRAME_RATE, fps);
        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, videoConfiguration.ifi);
        format.setInteger(MediaFormat.KEY_BITRATE_MODE, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR);
        format.setInteger(MediaFormat.KEY_COMPLEXITY, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_CBR);
        MediaCodec mediaCodec = null;

        try {
            mediaCodec = MediaCodec.createEncoderByType(videoConfiguration.mime);
            mediaCodec.configure(format, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        }catch (Exception e) {
            e.printStackTrace();
            if(mediaCodec ! =null) {
                mediaCodec.stop();
                mediaCodec.release();
                mediaCodec = null; }}return mediaCodec;
    }
Copy the code

3. Enable the video encoder

mMediaCodec.start();
Copy the code

4. Get the encoded data

	private void drainEncoder(a) {
		ByteBuffer[] outBuffers = mMediaCodec.getOutputBuffers();
		while (isStarted) {
			encodeLock.lock();
			if(mMediaCodec ! =null) {
				int outBufferIndex = mMediaCodec.dequeueOutputBuffer(mBufferInfo, 12000);
				if (outBufferIndex >= 0) {
					ByteBuffer bb = outBuffers[outBufferIndex];
					if(mListener ! =null) { // Call back the encoded H264 data
						mListener.onVideoEncode(bb, mBufferInfo);
					}
					mMediaCodec.releaseOutputBuffer(outBufferIndex, false);
				} else {
					try {
						// wait 10ms
						Thread.sleep(10);
					} catch (InterruptedException e) {
						e.printStackTrace();
					}
				}
				encodeLock.unlock();
			} else {
				encodeLock.unlock();
				break; }}}Copy the code

5. H264 is packaged as FLV

    // Receive H264 data
		@Override
    public void onVideoData(ByteBuffer bb, MediaCodec.BufferInfo bi) {
        mAnnexbHelper.analyseVideoData(bb, bi);
    }   
	/** * Process the hard-coded video data to generate each frame of video data, which is then passed to the FLV packer *@paramBb Hardcoded data buffer *@paramBi hardcoded BufferInfo */
    public void analyseVideoData(ByteBuffer bb, MediaCodec.BufferInfo bi) {
        bb.position(bi.offset);
        bb.limit(bi.offset + bi.size);

        ArrayList<byte[]> frames = new ArrayList<>();
        boolean isKeyFrame = false;

        while(bb.position() < bi.offset + bi.size) {
            byte[] frame = annexbDemux(bb, bi);
            if(frame == null) {
                LogUtils.e("annexb not match.");
                break;
            }
            // ignore the nalu type aud(9)
            if (isAccessUnitDelimiter(frame)) {
                continue;
            }
            // for pps
            if(isPps(frame)) {
                mPps = frame;
                continue;
            }
            // for sps
            if(isSps(frame)) {
                mSps = frame;
                continue;
            }
            // for IDR frame
            if(isKeyFrame(frame)) {
                isKeyFrame = true;
            } else {
                isKeyFrame = false;
            }
            byte[] naluHeader = buildNaluHeader(frame.length);
            frames.add(naluHeader);
            frames.add(frame);
        }
        if(mPps ! =null&& mSps ! =null&& mListener ! =null && mUploadPpsSps) {
            if(mListener ! =null) {
                mListener.onSpsPps(mSps, mPps);
            }
            mUploadPpsSps = false;
        }
        if(frames.size() == 0 || mListener == null) {
            return;
        }
        int size = 0;
        for (int i = 0; i < frames.size(); i++) {
            byte[] frame = frames.get(i);
            size += frame.length;
        }
        byte[] data = new byte[size];
        int currentSize = 0;
        for (int i = 0; i < frames.size(); i++) {
            byte[] frame = frames.get(i);
            System.arraycopy(frame, 0, data, currentSize, frame.length);
            currentSize += frame.length;
        }
        if(mListener ! =null) { mListener.onVideo(data, isKeyFrame); }}Copy the code

This method mainly obtains NALU from the encoded data, then determines the type of NALU, and finally calls back the data to FlvPacker for processing.

Processing spsPps:

    @Override
    public void onSpsPps(byte[] sps, byte[] pps) {
        if (packetListener == null) {
            return;
        }
        // Write the first video message
        writeFirstVideoTag(sps, pps);
        // Write the first audio message
        writeFirstAudioTag();
        isHeaderWrite = true;
    }

Copy the code

Processing video frames:

    @Override
    public void onVideo(byte[] video, boolean isKeyFrame) {
        if (packetListener == null| |! isHeaderWrite) {return;
        }
        int packetType = INTER_FRAME;
        if (isKeyFrame) {
            isKeyFrameWrite = true;
            packetType = KEY_FRAME;
        }
        // Make sure the first frame is a key frame to avoid the gray blur in the beginning
        if(! isKeyFrameWrite) {return;
        }
        int size = VIDEO_HEADER_SIZE + video.length;
        ByteBuffer buffer = ByteBuffer.allocate(size);
        FlvPackerHelper.writeH264Packet(buffer, video, isKeyFrame);
        packetListener.onPacket(buffer.array(), packetType);
    }
Copy the code

6. Release the encoder and release the Surface

	// Release the encoder
	private void releaseEncoder(a) {
		if(mMediaCodec ! =null) {
			mMediaCodec.signalEndOfInputStream();
			mMediaCodec.stop();
			mMediaCodec.release();
			mMediaCodec = null;
		}
		if(mInputSurface ! =null) {
			mInputSurface.release();
			mInputSurface = null; }}// Release OpenGL ES render, Surface
	public void release(a) {
		EGL14.eglDestroySurface(mEGLDisplay, mEGLSurface);
		EGL14.eglDestroyContext(mEGLDisplay, mEGLContext);
		EGL14.eglReleaseThread();
		EGL14.eglTerminate(mEGLDisplay);

		mSurface.release();

		mSurface    = null;
		mEGLDisplay = null;
		mEGLContext = null;
		mEGLSurface = null;
	}
Copy the code

The RTMP push flow

Note: The actual project RTMP needs to be connected before subsequent operations.

The RTMP module was compiled with FFmPEG when we were developing the player. So we directly use the last static library and header file can be, if you do not understand the RTMP protocol can refer to the last article, which also introduces the construction of RTMP live server.

At this point both the soft-coded and hard-coded data is ready to be sent to the RTMP module, which is native.

    /** * Packaged data, and raw stream data **@param data
     * @param type
     */
    @Override
    public void onData(byte[] data, int type) {
        if (type == RtmpPacker.FIRST_AUDIO || type == RtmpPacker.AUDIO) {// Audio AAC data, packaged
            mPusherManager.pushAACData(data, data.length, type);
        } else if (type == RtmpPacker.FIRST_VIDEO ||
                type == RtmpPacker.INTER_FRAME || type == RtmpPacker.KEY_FRAME) {//H264 video data, packed
            mPusherManager.pushH264(data, type, 0);
        } else if (type == RtmpPacker.PCM) { PCM raw stream data
            mPusherManager.pushPCM(data);
        } else if (type == RtmpPacker.YUV) { //YUV Raw stream datamPusherManager.pushYUV(data); }}/** * Send H264 data **@param h264
     */
    public native void pushH264(byte[] h264, int type, long timeStamp);
    / * * *@paramAudio Directly pushes the audio stream after the encoding is complete@param length
     * @param timestamp
     */
    public native void pushAACData(byte[] audio, int length, int timestamp);
    /** * Send PCM raw data **@param audioData
     */
    public native void native_pushAudio(byte[] audioData);
    /** * push video raw NV21 **@param data
     */
    public native void native_push_video(byte[] data);
Copy the code

1. The Rtmp links

Rtmp is TCP protocol, so you can use Java Socket for connection, you can also use c++ librtmp library to connect, let’s use librtmp to connect.

/** * the real RTMP connection function */
void RTMPModel::onConnect() {
		...

    / / 1. Initialization
    RTMP_Init(rtmp);
    //2. Set the RTMP address
    int ret = RTMP_SetupURL(rtmp, this->url)

  	//3. Confirm that RTMP is written
    RTMP_EnableWrite(rtmp);
		//4. Start the link
    ret = RTMP_Connect(rtmp, 0);
		//5. After the connection is successful, a stream needs to be connected
    ret = RTMP_ConnectStream(rtmp, 0); . }Copy the code

2. The Native audio module receives AAC Flv packaged data

/** * push AAC hardcoded * @param data */
void AudioEncoderChannel::pushAAC(u_char *data, int dataLen, long timestamp) {
    RTMPPacket *packet = (RTMPPacket *) malloc(sizeof(RTMPPacket));
    RTMPPacket_Alloc(packet, dataLen);
    RTMPPacket_Reset(packet);
    packet->m_nChannel = 0x05; / / audio
    memcpy(packet->m_body, data, dataLen);
    packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
    packet->m_hasAbsTimestamp = FALSE;
    packet->m_packetType = RTMP_PACKET_TYPE_AUDIO;
    packet->m_nBodySize = dataLen;
    if (mAudioCallback)
        mAudioCallback(packet); // Send to the RTMP module
}
Copy the code

3. Native video module receives H264 Flv package data

/** ** @param type Video frame type * @param buf H264 * @param Len H264 length */
void VideoEncoderChannel::sendH264(int type, uint8_t *data, int dataLen, int timeStamp) {
    RTMPPacket *packet = (RTMPPacket *) malloc(sizeof(RTMPPacket));
    RTMPPacket_Alloc(packet, dataLen);
    RTMPPacket_Reset(packet);

    packet->m_nChannel = 0x04; / / video

    if (type == RTMP_PACKET_KEY_FRAME) {
        LOGE("Video keyframe");
    }
    memcpy(packet->m_body, data, dataLen);
    packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
    packet->m_hasAbsTimestamp = FALSE;
    packet->m_packetType = RTMP_PACKET_TYPE_VIDEO;
    packet->m_nBodySize = dataLen;
    mVideoCallback(packet);// Send to the RTMP module
}
Copy the code

4. RTMP sends data

4.1 Sending the received data to the send queue

// Whether hard or soft coded, all sent data needs to be queued
void callback(RTMPPacket *packet) {
    if (packet) {
        if (rtmpModel) {
            // Set the timestamppacket->m_nTimeStamp = RTMP_GetTime() - rtmpModel->mStartTime; rtmpModel->mPackets.push(packet); }}}Copy the code

4.2 send

/** * where the stream is really pushed */
void RTMPModel::onPush() {
    RTMPPacket *packet = 0;
    while (isStart) {
      	// Get the audio and video data sent from the queue
        mPackets.pop(packet);
        if(! readyPushing) { releasePackets(packet);return;
        }
        if(! packet) { LOGE("Fetch failed");
            continue;
        }
        packet->m_nInfoField2 = rtmp->m_stream_id;
        int ret = RTMP_SendPacket(rtmp, packet, 1);
        if(! ret) { LOGE("Send failed")
            if (pushCallback) {
                pushCallback->onError(THREAD_CHILD, RTMP_PUSHER_ERROR);
            }
            return;
        }
    }
    releasePackets(packet);
    release();/ / release
}
Copy the code

5. Close the RTMP

You need to close the RTMP connection when you do not need to send audio and video data

void RTMPModel::release() {
    isStart = false;
    readyPushing = false;
    if (rtmp) {
        RTMP_DeleteStream(rtmp);
        RTMP_Close(rtmp);
        RTMP_Free(rtmp);
        rtmp = 0;
        LOGE("Release native resources");
    }
    mPackets.clearQueue();
}
Copy the code

Talk briefly about hard and soft codec

1. The difference between

Soft coding: Encoding using CPU. Hard coding: The GPU is used for encoding.

2. Compare

Soft coding: direct, simple, easy to adjust parameters, easy to upgrade, but the CPU load, performance is lower than hard coding, low bit rate quality is usually better than hard coding. Hard coding: with high performance and low bit rate, the quality is usually lower than that of soft encoder. However, some products transplant excellent soft coding algorithm (such as X264) on GPU hardware platform, and the quality is basically the same as that of soft coding.

3. Application scenarios

Soft coding: Suitable for short time operations, such as recording short videos.

Hard coding: Long coding or high requirements on video quality (VOIP real-time calls). Hardware coding is recommended (provided that the mobile phone performance is good).

conclusion

Here Android side soft push stream, hard push stream are respectively implemented. In the project can be based on the actual situation to choose after all is hard or soft.

I am based on the crazy open source project for secondary development:

Android push stream project address

Android pull flow project address

reference

• Let’s go live