preface

When many developers call the Android platform RTMP push or lightweight RTSP service interface, the video data collected are of various types, such as 420SP, I420, YV12, NV21, RGB, and some images are inverted. If the developers transfer the images to the underlying coding process after the upper level conversion, Undoubtedly, it increases the upper processing burden, and is easy to affect the experience due to low efficiency. This paper takes the Video data docking interface before pushing SDK coding of The Android platform RTMP of Daniu Live SDK as an example to see what commonly used data formats are. Related information, please refer to Github:

OnPreviewFrame() : OnPreviewFrame() : OnPreviewFrame() :

Android built-in camera camera data docking is the most basic, need to consider the camera direction, such as landscape, portrait, and some customized devices home button on the left side of the case, relatively easy to deal with, directly to the interface, no further details.

    @Override
    public void onPreviewFrame(byte[] data, Camera camera) {
        frameCount++;
        if (frameCount % 3000 == 0) {
            Log.i("OnPre", "gc+");
            System.gc();
            Log.i("OnPre", "gc-");
        }

        if (data == null) {
            Parameters params = camera.getParameters();
            Size size = params.getPreviewSize();
            int bufferSize = (((size.width | 0x1f) + 1) * size.height * ImageFormat.getBitsPerPixel(params.getPreviewFormat())) / 8;
            camera.addCallbackBuffer(new byte[bufferSize]);
        } else {
            if (isRTSPPublisherRunning || isPushingRtmp || isRecording || isPushingRtsp) {
                libPublisher.SmartPublisherOnCaptureVideoData(publisherHandle, data, data.length, currentCameraType, currentOrigentation);
            }

            camera.addCallbackBuffer(data);
        }
    }
Copy the code

Corresponding interface definition:

/** * Set live video data(no encoded data). * * @param cameraType: CAMERA_FACING_BACK with 0, CAMERA_FACING_FRONT with 1 * * @param curOrg: * PORTRAIT = 1; // LANDSCAPE = 2; * * LANDSCAPE_LEFT_HOME_KEY = 3; / / landscape home button on the left of the @ * * return {0} if successful * / public native int SmartPublisherOnCaptureVideoData (long handle. byte[] data, int len, int cameraType, int curOrg);Copy the code

2. Some customized devices only support YV12 data:

YV12 and NV21 are in the majority of third-party cameras. Some of them need to be rotated.

/** * @param data: YV12 data ** @param width: width ** @param height: height ** @param y_stride: Y * * @param v_stride: v * * @param u_stride: u * * rotation_degree: * * @return {0} if successful */ public native int SmartPublisherOnYV12Data(long handle, SmartPublisherOnYV12Data) byte[] data, int width, int height, int y_stride, int v_stride, int u_stride, int rotation_degree);Copy the code

3. Support NV21 data interface:

Nv21 data interface, in addition to the conventional camera data access, part of the customized camera out of the data flip, this interface also supports.

/** * @param data: NV21 data ** @param len: data length ** @param width: width ** @param height: * * @param uv_stride: y stride * * @param uv_stride: uv stride * * rotation_degree: * * @return {0} if successful */ public native int SmartPublisherOnNV21Data(long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree); /** * @param data: NV21 data ** @param len: data length ** @param width: width ** @param height: * * @param uv_stride: y stride * * @param uv_stride: uv stride * * rotation_degree: Rotate clockwise, must be 0, 90, 180, 270 * * @param is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flips * * @param is_horizontal_flip: Whether the flip horizontal, 0 not flip, flip * * @ 1 return {0} if successful * / public native int SmartPublisherOnNV21DataV2 (long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree, int is_vertical_flip, int is_horizontal_flip);Copy the code

4. Support YUV data access:

Supports standard I420 data interface interconnection.

    /**
    * Set live video data(no encoded data).
    *
    * @param data: I420 data
    * 
    * @param len: I420 data length
    * 
    * @param yStride: y stride
    * 
    * @param uStride: u stride
    * 
    * @param vStride: v stride
    *
    * @return {0} if successful
    */
    public native int SmartPublisherOnCaptureVideoI420Data(long handle,  byte[] data, int len, int yStride, int uStride, int vStride);
Copy the code

5. Support RGBA data access (support tailored data access, mainly used in the same screen scenario) :

RGBA is mainly used in screen sharing scenarios, in order to facilitate the push part of the screen, we have added a friendly cropping parameter.

    /**
    * Set live video data(no encoded data).
    *
    * @param data: RGBA data
    * 
    * @param rowStride: stride information
    * 
    * @param width: width
    * 
    * @param height: height
    *
    * @return {0} if successful
    */
    public native int SmartPublisherOnCaptureVideoRGBAData(long handle,  ByteBuffer data, int rowStride, int width, int height);

    /** * Drop clipped RGBA data **@param data: RGBA data
     *
     * @param rowStride: stride information
     *
     * @param width: width
     *
     * @param height: height
     *
     * @paramClipedLeft: left; ClipedTop:; Clipedwidth: clipped width; ClipedHeight: clipped height; Make sure that the width and height of the cut down are even * *@return {0} if successful
     */
    public native int SmartPublisherOnCaptureVideoClipedRGBAData(long handle,  ByteBuffer data, int rowStride, int width, int height, int clipedLeft, int clipedTop, int clipedWidth, int clipedHeight);

    /**
     * Set live video data(no encoded data).
     *
     * @paramData: ABGR flip vertical * *@param rowStride: stride information
     *
     * @param width: width
     *
     * @param height: height
     *
     * @return {0} if successful
     */
    public native int SmartPublisherOnCaptureVideoABGRFlipVerticalData(long handle,  ByteBuffer data, int rowStride, int width, int height);
Copy the code

6. RGB565 data access (mainly used in the same screen scenario) :

Most of the scenes are on the same screen.

    /**
     * Set live video data(no encoded data).
     *
     * @param data: RGB565 data
     *
     * @param row_stride: stride information
     *
     * @param width: width
     *
     * @param height: height
     *
     * @return {0} if successful
     */
    public native int SmartPublisherOnCaptureVideoRGB565Data(long handle,ByteBuffer data, int row_stride, int width, int height);
Copy the code

7. Camera data access (mainly used for Camera 2 interface docking) : Compatible with Camera 2 data acquisition mode for higher efficiency.

/ * * specifically for android. Media. The Image of android. Graphics. ImageFormat. YUV_420_888 format provided interfaces * * @ param width: * * @param height: must be a multiple of 8 * * @param crop_left: Left upper corner of the shear horizontal coordinates, generally based on android. The media. Image. GetCropRect () fill @ * * param crop_top: Shear vertical coordinates of the upper left corner, generally based on android. Media. Image. GetCropRect () fill @ * * param crop_width: Must be in multiples of 8, 0 will ignore this parameter, generally based on android. Media. Image. GetCropRect () fill @ * * param crop_height: Must be a multiple of 8, 0 will ignore this argument, Generally based on android. Media. Image. GetCropRect () fill * * @ param y_plane corresponding android. Media. Image. The Plane [0]. GetBuffer () * * @ param Y_row_stride corresponding android. Media. Image. The Plane [0]. GetRowStride () * * @ param u_plane corresponding android. Media. Image. The Plane [1]. The getBuffer () * * @ param v_plane corresponding android. Media. Image. The Plane [2]. The getBuffer () * * @ param uv_row_stride . Corresponding android. Media. The Image Plane. [1] getRowStride () * * @ param uv_pixel_stride corresponding android. Media. Image. The Plane [1]. The getPixelStride () * * @param rotation_degree: rotate clockwise, must be 0, 90, 180, 270 * * @param is_vertical_flip: @param is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flips * * @param scale_width: Scale width, must be a multiple of 8, 0 is not scaled * * @param scale_height: scale height, must be a multiple of 8, 0 is not scaled * * @param scale_filter_mode: Return {0} if successful */
    public native int SmartPublisherOnImageYUV420888(long handle, int width, int height,
                                                     int crop_left, int crop_top, int crop_width, int crop_height,
                                                     ByteBuffer y_plane, int y_row_stride,
                                                     ByteBuffer u_plane, ByteBuffer v_plane, int uv_row_stride, int uv_pixel_stride,
                                                     int rotation_degree, int is_vertical_flip, int is_horizontal_flip,
                                                     int scale_width, int scale_height, int scale_filter_mode);
Copy the code

Conclusion:

It can be seen that some companies or developers mentioned that an Android platform RTMP push module only has several interfaces, it is almost impossible to simplify complexity, unless.