In the first two chapters, we introduced the RGB and YUV raw formats of video and how to collect camera data. In this chapter, we will introduce how to encode YUV data

1. Video coding

In the previous chapter, introduced how to collect the camera data, tried friends may find that the NV21 data, a few seconds down, the file is hundreds of M, we usually see MP4, a few seconds only a few M, what is going on

This is because we are storing raw, uncompressed data, so it is extremely large to store

In fact, not only the large storage, the general original format of the video, but also can not play, because the playback can not know your a lot of data is what, the player generally need to know the video frame rate, resolution, bit rate and other parameters, in order to correctly parse the video and decode the playback

So what we’re going to do is we’re going to encode the data from the camera

Common encoding format H26x, general Mp4 video encoding format is H264, this encoding format can greatly reduce the storage size

To review the video capture process in the previous chapter:

  1. Turn on the camera
  2. Initializing the Camera
  3. Set the preview callback
  4. To preview
  5. In the preview callback, getNV21Data and write it to a file
  6. Stop previewing and release the camera

Then, for encoding, it needs to be modified in step 5 by adding encoding logic, and the step becomes:

  1. Turn on the camera
  2. Initializing the Camera
  3. Set the preview callback
  4. To preview
  5. In the preview callback, getNV21Data, convert toNV12Data, passed into the encoder
  6. In the encoder output, get the encoded data and write it to a file orMediaMuxer
  7. Stop previewing and release the camera

Note that when we preview the callback, we get NV21 data, but when we pass NV12 data to the encoder, we need NV12 data (the encoder supports NV12 data best), so there is a conversion process in the middle, which we will discuss in detail below

Another is that in step 6, we have two options. If we need to make our encoded video playable in any player, we can use MediaMuxer to package it as AN Mp4

So, let’s implement the above

YUV rotation, mirror

2.1 YUV rotation

In preview callback, we obtain the original NV21 data, but need to note that the image is been rotating, timely when we initialize camera Angle to its rotation, preview picture showed normal, but the preview callback or selected 90 or 270 degrees, all we convert them into NV12, Also pay attention to the rotation Angle

Rotation Angle

When we initialize the camera, we set the rotation Angle

camera.setDisplayOrientation(orientation = CameraUtils.getDisplayOrientation(activity, facing));
Copy the code

Because NV21 to NV12 is required, so we have a variable here

Now let’s rotate it at various angles

Rotate 90 degrees

First of all, let’s say the graph looks something like this

Then rotate 90 degrees clockwise and convert to NV12 format, you can get

So let’s look at the numbers for Y first

The first row of NV12 takes the first column of NV21, the second row takes the second column of NV21, and so on, to get the data of Y

When looking at the UV data, since NV12 and NV21 are in YUV420sp format, we need to rotate the UV as a whole

NV12 first and second data are taken from U3 and V3, originally stored in the order of V3->U3, but because NV12 UV order and NV21 UV order is just opposite, so you need to reverse the two, and so on, you get the data storage form of the figure above. How to see the code to achieve

/** * nv21 to nv12, and rotate 90 degrees */
private static byte[] nv21ToNv12AndRotate90(byte[] inputData, int width, int height) {
    if (inputData == null) {
        return null;
    }
    int size = inputData.length;
    if(size ! = (width * height *3 / 2)) {
        return null;
    }
    byte[] outputData = new byte[size];
    int k = 0;
    for (int i = 0; i < width; i++) {
        for (int j = height - 1; j >= 0; j--) { outputData[k++] = inputData[width * j + i]; }}int start = width * height;
    for (int i = 0; i < width; i += 2) {
        for (int j = height / 2 - 1; j >= 0; j--) {
            outputData[k++] = inputData[start + width * j + i + 1]; outputData[k++] = inputData[start + width * j + i]; }}return outputData;
}
Copy the code

In the code, there are two for loops, the first for Y data and the second for UV data

From the above diagram and code, it is not difficult to find the law of rotation at other angles

Rotate 180 degrees

/** * nv21 to nv12, and rotate 180 degrees */
private static byte[] nv21ToNv12AndRotate180(byte[] inputData, int width, int height) {
    if (inputData == null) {
        return null;
    }
    int size = inputData.length;
    if(size ! = (width * height *3 / 2)) {
        return null;
    }
    byte[] outputData = new byte[size];
    int k = 0;
    for (int i = height - 1; i >= 0; i--) {
        for (int j = width - 1; j >= 0; j--) { outputData[k++] = inputData[width * i + j]; }}int start = width * height;
    for (int i = height / 2 - 1; i >= 0; i--) {
        for (int j = width - 1; j >= 0; j -= 2) {
            outputData[k++] = inputData[start + width * i + j];
            outputData[k++] = inputData[start + width * i + j - 1]; }}return outputData;
}
Copy the code

Rotate 270 degrees

/** * nv21 to NV12, and 270 degrees */
private static byte[] nv21ToNv12AndRotate270(byte[] inputData, int width, int height) {
    if (inputData == null) {
        return null;
    }
    int size = inputData.length;
    if(size ! = (width * height *3 / 2)) {
        return null;
    }
    byte[] outputData = new byte[size];
    int k = 0;
    for (int i = width - 1; i >= 0; i--) {
        for (int j = 0; j < height; j++) { outputData[k++] = inputData[width * j + i]; }}int start = width * height;
    for (int i = width - 1; i >= 0; i -= 2) {
        for (int j = 0; j < height / 2; j++) {
            outputData[k++] = inputData[start + width * j + i];
            outputData[k++] = inputData[start + width * j + i - 1]; }}return outputData;
}
Copy the code

This includes all the angles of the camera rotation. However, it is important to note that generally, the proactive image is the opposite, that is, we need to mirror it when we process the proactive image

2.2 YUV mirror

/** * Mirror processing */
private static byte[] yuvMirror(byte[] inputData, int width, int height) {
    if (inputData == null) {
        return null;
    }
    int size = inputData.length;
    byte[] outputData = new byte[size];
    int k = 0;
    for (int i = 0; i < height; i++) {
        for (int j = width - 1; j >= 0; j--) { outputData[k++] = inputData[width * i + j]; }}int start = width * height;
    for (int i = 0; i < height / 2; i++) {
        for (int j = width - 1; j >= 0; j -= 2) {
            outputData[k++] = inputData[start + width * i + j - 1]; outputData[k++] = inputData[start + width * i + j]; }}return outputData;
}
Copy the code

Mirror processing is relatively simple, is to store each row of data in reverse order, but the need to pay attention to UV as a whole, when processing do not forget

Let’s do another method based on incoming data, size, camera type, and rotation Angle

/** * nv12 */
public static byte[] cameraNv21ToNv12(
        byte[] data,
        int width,
        int height,
        int facing,
        int orientation) {
    byte[] outputData;
    Log.d(TAG, "cameraNv21ToNv12: " + orientation + " facing:" + facing);
    int rotate = orientation;
    int w = width;
    int h = height;
    if (facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        rotate = 360 - orientation;
    }
    switch (rotate) {
        case 90:
            // After rotation, the width and height are interchangeable
            w = height;
            h = width;
            outputData = nv21ToNv12AndRotate90(data, width, height);
            break;
        case 180:
            // The width and height remain the same after rotation
            outputData = nv21ToNv12AndRotate180(data, width, height);
            break;
        case 270:
            // After rotation, the width and height are interchangeable
            w = height;
            h = width;
            outputData = nv21ToNv12AndRotate270(data, width, height);
            break;
        default:
            outputData = data;
            break;
    }
    return cameraNv21ToNv12WidthFacing(outputData, w, h, facing);
}
/** * Obtain NV12 data by facing */
private static byte[] cameraNv21ToNv12WidthFacing(
        byte[] data,
        int width,
        int height,
        int facing) {
    if (facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        // Proactive is mirrored, so it needs to be mirrored once
        return yuvMirror(data, width, height);
    }
    return data;
}
Copy the code

As you can see from the above, we only expose a single cameraNv21ToNv12() method to external calls

3. MediaCodec coding

Now that you’ve learned how to pre-encode YUV, let’s get to the actual coding, okay

Initialize MediaCodec

private void initMediaCodec(a) {
    int width = this.height;
    int height = this.width;
    try {
        MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height);
        format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
        format.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
        format.setInteger(MediaFormat.KEY_BIT_RATE,
                width * height * 4);
        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
        // Set the default compression level to baseline
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
            format.setInteger(MediaFormat.KEY_PROFILE, MediaCodecInfo.CodecProfileLevel.AVCProfileMain);
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
                format.setInteger(MediaFormat.KEY_LEVEL, MediaCodecInfo.CodecProfileLevel.AVCLevel3);
            }
        }
        mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
        mediaCodec.setCallback(this);
        mediaCodec.configure(format, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        mediaCodec.start();
    } catch(IOException e) { e.printStackTrace(); }}Copy the code

For the use of MediaCodec, we have already known in the previous audio coding, but we have used the synchronous encoding of MediaCodec, now we will use the asynchronous encoding of MediaCodec

After setCallback, MediaCodec needs to implement several callback methods

  • onInputBufferAvailable()

    Callback when the input buffer is available, where data can be inserted

  • onOutputBufferAvailable()

    Callback when the encoded data is finished, which is already encoded H264 data that can be written to MediaMuxer

  • onError()

    Returns when an error occurred

  • onOutputFormatChanged()

    At this point, you can manipulate the MediaFormat of the output, such as calling the mediamuxer.addTrack method to get a TrackId to write to MediaMuxer

After initializing MediaCodec, we can initialize MediaMuxer. MediaMuxer can help us generate Mp4 and check whether the video is encoded properly

private void initMediaMuxer(String path) {
    if (TextUtils.isEmpty(path)) {
        return;
    }
    File file = new File(path);
    if (file.exists()) {
        file.delete();
    }
    try {
        mediaMuxer = new MediaMuxer(path, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    } catch (IOException e) {
        e.printStackTrace();
        mediaMuxer = null; }}Copy the code

From the above description, we need to put the camera and encoding operations into the child thread, and the camera preview callback, encoding insertion and extraction data are asynchronous, so what should we do

We can use a queue to store the data of the preview callback, and then at encoding time, pull the queue data into the encoder to achieve the desired effect

For queues, we can use LinkedBlockingQueue, which is a thread-safe queue

private final BlockingQueue<byte[]> queue = new LinkedBlockingQueue<>(10);
Copy the code

Preview data processing

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
    if (this.data == null) {
        this.data = new byte[width * height * 3 / 2];
    }
    camera.addCallbackBuffer(this.data);
    queue.offer(YuvUtils.cameraNv21ToNv12(this.data, width, height, facing, orientation));
}
Copy the code

In the preview callback, we call the previously described method cameraNv21ToNv12 to convert the NV21 data into NV12 data and pass it into the queue

Insert encoder

@Override
public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
    ByteBuffer buffer = codec.getInputBuffer(index);
    buffer.clear();
    int size = 0;
    byte[] data = queue.poll();
    if(data ! =null) {
        buffer.put(data);
        size = data.length;
    }
    codec.queueInputBuffer(index, 0, size, System.nanoTime() / 1000.0);
}
Copy the code

In the onInputBufferAvailable() callback, we take the data and insert it into the encoder. Note that we need to pass a timestamp to insert it. Instead, we pass system.nanotime () / 1000

Add a video track

Before we can write to MediaMuxer, we have to write the output format of the video to MediaMuxer so that we can have a TrackId to write to

@Override
public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) {
    if(trackIndex ! = -1) {
        return;
    }
    if (mediaMuxer == null) {
        return;
    }
    trackIndex = mediaMuxer.addTrack(format);
    mediaMuxer.start();
}
Copy the code

Get the encoded data and write to MediaMuxer

@Override
public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info) {
    ByteBuffer buffer = codec.getOutputBuffer(index);
    if(buffer ! =null && info.size > 0) {
        if(mediaMuxer ! =null&& trackIndex ! = -1) {
            mediaMuxer.writeSampleData(trackIndex, buffer, info);
        }
        buffer.clear();
    }
    codec.releaseOutputBuffer(index, false);
}
Copy the code

In this way, we have completed the whole video collection to the encoding, and finally to the output Mp4 process

Finally, don’t forget to release resources

private void closeCamera(a) {
    if (camera == null) {
        return;
    }
    camera.stopPreview();
    camera.release();
    camera = null;
}
​
private void stopMediaMuxer(a) {
    if (mediaMuxer == null) {
        return;
    }
    mediaMuxer.stop();
    mediaMuxer.release();
    mediaMuxer = null;
}
​
private void stopMediaCodec(a) {
    if (mediaCodec == null) {
        return;
    }
    mediaCodec.stop();
    mediaCodec.release();
    mediaCodec = null;
}
Copy the code

Fourth, making

YuvEncoder.java

YuvActivity.java