Camera preview direction

The orientation of the image captured by the camera sensor and displayed on the screen is the camera preview orientation. By default, the orientation is the same as that of the Camera sensor, which can be changed through the Camera API. Camaer1 can set the preview orientation using setDisplayOrientation, and Camera2 can set the preview orientation using TextureView. Orientation is different for different camera positions. Orientation is the Angle that the camera sensor rotates clockwise to the natural orientation of the screen.

The rear

The rear orientation90

Pre –

Orientation270 in front of orientation270, after the image is collected (without mirror processing), it must be displayed on the screen according to the coordinate system of the natural direction of the screen. It needs to rotate 270 degrees clockwise to be consistent with the natural direction of the device. When I preview, I mirror it, so I just need to rotate it 90 degrees clockwise to match the natural orientation. Camera1 and Camera2 specify the preview direction of the code from CameraView:

Camera1

private int calcDisplayOrientation(int screenOrientationDegrees) {
    if (mCameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        return (360 - (mCameraInfo.orientation + screenOrientationDegrees) % 360) % 360;    // compensate the mirror
    } else {  // back-facing
        return (mCameraInfo.orientation - screenOrientationDegrees + 360) % 360; }}Copy the code

The code distinguishes between front and rear cameras.

• The rear:(mCameraInfo.orientation - screenOrientationDegrees + 360) % 360To return to the natural direction, clockwise rotation is required, while the counterclockwise rotation of the screen just counterbalances the rotation of the camera. The difference between the two is +360.
• Lead:(mCameraInfo.orientation + screenOrientationDegrees) % 360The vertical view of the screen is a mirror image, 360-(mCameraInfo.orientation + screenOrientationDegrees) % 360If you rotate the difference clockwise, it goes in the natural direction, except it’s a mirror image, flipped left and right

Camera2 uses TextureView’s setTransform for rotation and differentiates between vertical and horizontal screens.

/**
* Configures the transform matrix for TextureView based on {@link #mDisplayOrientation} and
* the surface size.
*/
void configureTransform(a) {
    Matrix matrix = new Matrix();
    if (mDisplayOrientation % 180= =90) {
        final int width = getWidth();
        final int height = getHeight();
        // Rotate the camera preview when the screen is landscape.
        matrix.setPolyToPoly(
                new float[] {0.f, 0.f, // top left
                        width, 0.f, // top right
                        0.f, height, // bottom left
                        width, height, // bottom right
                }, 0,
                mDisplayOrientation == 90 ?
                        // Clockwise
                        new float[] {0.f, height, // top left
                                0.f, 0.f, // top right
                                width, height, // bottom left
                                width, 0.f, // bottom right
                        } : // mDisplayOrientation == 270
                        // Counter-clockwise
                        new float[]{
                                width, 0.f, // top left
                                width, height, // top right
                                0.f, 0.f, // bottom left
                                0.f, height, // bottom right
                        }, 0.4);
    } else if (mDisplayOrientation == 180) {
        matrix.postRotate(180, getWidth() / 2, getHeight() / 2);
    }
    mTextureView.setTransform(matrix);
}
Copy the code

Photo direction

Setting the preview direction does not change the direction of the resulting photo. For the rear-mounted camera, the image captured by the camera is the same as the image previewed by the camera, only the orientation degree of the rear-mounted camera needs to be rotated. For front-facing cameras, the image previewed by the camera and the image captured by the camera are mirror images. Captured image: rotated 270 degrees clockwise, consistent with the natural orientation of the screen. Preview image: rotate 90 degrees clockwise to match the natural orientation of the screen. Finally, steal a picture to illustrate:

Camera1

Using mCameraParameters. SetRotation () set after the photo image direction:

mCameraParameters.setRotation(calcCameraRotation(displayOrientation)); ./**
* Calculate camera rotation
*
* This calculation is applied to the output JPEG either via Exif Orientation tag
* or by actually transforming the bitmap. (Determined by vendor camera API implementation)
*
* Note: This is not the same calculation as the display orientation
*
* @param screenOrientationDegrees Screen orientation in degrees
* @return Number of degrees to rotate image in order for it to view correctly.
*/
private int calcCameraRotation(int screenOrientationDegrees) {
    if (mCameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        return (mCameraInfo.orientation + screenOrientationDegrees) % 360;
    } else {  // back-facing
        final int landscapeFlip = isLandscape(screenOrientationDegrees) ? 180 : 0;
        return (mCameraInfo.orientation + screenOrientationDegrees + landscapeFlip) % 360; }}Copy the code

The image collected by the camera only needs to rotate the orientation degree of the camera.

Camera2

According to CameraCharacteristics SENSOR_ORIENTATION, set up a JPEG image using captureRequest direction of rotation.

// Calculate JPEG orientation.
@SuppressWarnings("ConstantConditions")
int sensorOrientation = mCameraCharacteristics.get(
        CameraCharacteristics.SENSOR_ORIENTATION);
captureRequestBuilder.set(CaptureRequest.JPEG_ORIENTATION,
        (sensorOrientation +
                mDisplayOrientation * (mFacing == Constants.FACING_FRONT ? 1 : -1) +
                360) % 360);
Copy the code

Image data

The default data format returned by Android Camera is NV21. Through Camera1 mParameters. SetPreviewFormat () Settings, Camera2 through ImageReader. NewInstance () setting. NV21 and YV12 are officially recommended formats.NV21 and YV12 belong to YUV formats, which can also be represented as YCbCr. Cb and Cr are the same as U and V.

YUV

YUV is a color coding method, and it is similar to RGB color coding method, mainly used in the field of TV systems and analog video. YUV represents three components, Y represents brightness, U and V represent chroma, which in turn defines two aspects of color: hue and saturation. By separating Y from UV, a complete image can be displayed without UV information, but only grayscale images can be displayed.

YUV sampling format

The main sampling methods of YUV images are as follows:

• YUV 4:4:4 sampling: Each Y corresponds to a set of UV components
• YUV 4:2:2 sampling: every two Y’s share a set of UV components
• YUV 4:2:0 sampling: each of the four Y’s share a set of UV components

The black dot represents the Y component of the pixel sampled, and the hollow circle represents the UV component of the pixel

YUV storage format

There are two storage formats, planar and Packed.

• Planar: stores the Y of all pixels successively, then the U of all pixels, then the V of all pixels
• Packed: Y,U and V of each pixel are stored continuously and alternately

YUV Pixel formats vary according to sampling methods and storage formats. The most common YUV formats are:

YUVY format

YUVY format belongs to Packed storage format, and two adjacent Y’s share two adjacent U and V’s

Y0 UO Y1 V0 Y2 U2 Y3 V2
Copy the code

Y0 and Y1 share U0 and V0. Y2 and Y3 share U2 and V2

UYVY format

UYVY format also belongs to Packed storage format. Different from YUYV format, UYVY format has a different order of UV

YUV422P format

YUV422P format belongs to planar storage format, which stores Y of all pixels successively first, then U of all pixels, and then V of all pixels

Format: YV12, YU12

YU12 and YV12 formats both belong to YUV420P format, YUV420P is planar storage format. First store all the Y’s, then store U and V’s. The difference between YU12 and YV12 is that YU12 is Y, then U, then V, and YV12 is Y, then V, then U.

NV12 or NV21

NV12, NV21 YUV420SP format, YUV420SP is planar storage format. Store all Y first, then store in alternating UV or VU order. The NV12 format stores Y first, then UV and then interchangeably. In NV21 format, Y is stored first, and THEN VU is stored alternately. Finally steal a summary of the data format:

YV21: YYYYYYYY UU VV => YUV420P   
YV12: YYYYYYYY VV UU => YUV420P   
NV12: YYYYYYYY UV UV => YUV420SP   
NV21: YYYYYYYY VU VU => YUV420SP
Copy the code

Android Camera default data format is NV21, directly set mParameters Camera1. SetPreviewFormat (ImageFormat. NV21), then take photos in the callback raw data is NV21 data back. Imagereader.newinstance () : imageformat.nv21: imageformat.nv21

if (format == ImageFormat.NV21) {
    throw new IllegalArgumentException(
            "NV21 format is not supported");
}
Copy the code

It is explained in the latest imageformat.nv21:

YCrCb format used for images, which uses the NV21 encoding format.   
This is the default format for android.hardware.Camera preview images,
when not otherwise set with android.hardware.Camera.Parameters.setPreviewFormat(int).
For the android.hardware.camera2 API, the YUV_420_888 format is recommended for YUV output instead.
Copy the code

It is suggested to use YUV_420_888 instead of Camera2, so data transformation is required to obtain NV21 data. For details, please refer to the Image class analysis (combined with YUV_420_888).

reference

• Android camera development and encountered pits
• Android Camera2 tutorial chapter 3 preview
• Size and orientation issues in Android camera development
• Android camera preview direction and adaptation exploration
• Understand the sampling and format of YUV
• YUV420 data format
• Brief analysis of Image class (combined with YUV_420_888)