Android custom Camera2 camera

Writing in the front

Camera2(Android.hardware. Camera2) replaces Camera1(Android.Hardware.Camera) Camera framework with android 5.0L (API 21).

Camera2 is a complete departure from the previous Camera1 architecture and is more complex to use, but at the same time very powerful.

This blog post will help you quickly build and understand the key steps for customizing Camera2 cameras.

For the complete code, please move:Github.com/zhijunhong/…

Advantages of using Camera2

Through the transformation and optimization of the design framework, Camera2 has the following advantages:

  1. Improved performance of new hardware with more advanced API architecture;
  2. You can get more frame (preview/photo) information and manually control the parameters of each frame;
  3. More complete Camera control (such as adjusting focus Distance, clipping preview/photo);
  4. Support more picture formats (YUV/RAW) and high-speed continuous shooting;
  5. .

Custom Camera2 camera

Some of the concepts

1. Pipeline

The Camera2 API model is designed as a Pipeline that processes each frame of requests sequentially and returns the request results to the client. The official diagram below shows the Pipeline workflow and will be explained in detail with a simple example.

20210224001.png

Pipeline schematic

To illustrate the diagram above, suppose we want to take two images of different sizes at the same time, and the flash must be on during the shooting. The whole shooting process is as follows:

  1. Create a CaptureRequest to fetch images from Pipeline;
  2. Modify the flash configuration of CaptureRequest so that the flash lights up during photo taking.
  3. Create two surfaces of different sizes to receive image data and add them to CaptureRequest;
  4. Send the configured CaptureRequest to the Pipeline and wait for it to return the photo result.

A new CaptureRequest is placed into a Queue called a Pending Request Queue to be executed. In-flight Capture Queue When the in-flight Capture Queue is idle, the System obtains several Pending Capturerequests from the Pending Request Queue, and Capture operations are performed based on the configuration of each CaptureRequest. Finally, we get the picture data from the Surface of different sizes and get a CaptureResult containing a lot of information related to this photo taking. The end of the process.

2. Supported Hardware Level

Whether a camera is powerful or not depends on the Hardware. Different vendors have different levels of support for Camera2, so Camera2 defines an important concept called Supported Hardware Level. The idea is to divide Camera2 on different devices into different levels based on their functionality so that developers can get an idea of what Camera2 is currently supported on their devices. As of Android P, there are four levels of LEGACY, LIMITED, FULL, and LEVEL_3 from low to high:

  1. LEGACY: Backward compatible level. A device at this level only supports Camera1 features and does not have any Camera2 advanced features.
  2. LIMITED: Support for some levels of Camera2 advanced features in addition to basic Camera1 features;
  3. FULL: supports all advanced Camera2 features;
  4. LEVEL_3: Added more advanced Camera2 features, such as YUV data post-processing.

3. Capture

All camera operations and parameter configurations ultimately serve the purpose of image capture, such as focusing to make an area of the image sharper, adjusting the exposure compensation to adjust the brightness of the image, etc. Therefore, all camera operations and Settings in Camera2 are abstracted to Capture, so do not simply think of Capture as taking a picture, because the Capture operation may only be used to focus the preview image more clearly. If you’re familiar with Camera, you might be asking where is setFlashMode()? SetFocusMode ()? TakePicture ()? For your information, they are all implemented through Capture.

Capture is divided into three execution modes: single mode, multiple mode, and repeat mode.

  • One-shot: Capture operations are performed only once, such as setting the flash mode, focusing mode, and taking a photo. Multiple one-time modes of Capture are queued and executed in sequence.
  • Burst mode: This mode does not allow any other Capture operations to be performed consecutively. For example, if 100 photos are taken in a row, any new Capture requests will be queued. Until you’ve taken 100 pictures. Multiple groups of Multiple modes of Capture are queued and executed in sequence.
  • Repeating: Capture is a Capture operation that is continuously executed. When a Capture in another mode is submitted, the execution of the Capture in another mode is suspended. When the execution of the Capture in another mode is complete, the execution of the Capture in this mode is automatically resumed. For example, to display a preview screen is to Capture each frame continuously. The schema Capture is globally unique, which means that the newly submitted duplicate schema Capture overwrites the old duplicate schema Capture.

Key to the API

| CameraManager | CameraManager is a system to set up the camera connection query and services, its function is not much, here are a few CameraManager key features: 1. The camera information is encapsulated in CameraCharacteristics, and the way to obtain the instance of CameraCharacteristics is proposed. 2. Connect the camera device according to the specified camera ID. 3. Provide a shortcut to set the flash to flashlight mode. | | — – | — – | | CameraCharacteristics | CameraCharacteristics is a read-only cameras information provider, its internal carry plenty of camera information, including on behalf of the camera toward LENS_FACING; FLASH_INFO_AVAILABLE to determine whether the flash is available; Get CONTROL_AE_AVAILABLE_MODES and so on for all available AE modes. If you are familiar with Camera1, CameraCharacteristics is a bit like Camera1 Camera.CameraInfo or Camera.parameters. | | CameraDevice | CameraDevice represents the current connection camera equipment, it is the responsibility of the following four: 1. Create CameraCaptureSession according to the specified parameters; 2. Create a CaptureRequest based on the specified template. 3. Turn off the camera. 4. Monitor the status of the camera, such as disconnection, startup success, and startup failure. CameraDevice is the Camera class of Camera1, but it’s not. The Camera class is responsible for almost all Camera operations, while CameraDevice is very simple. The more detailed camera operation is handed over to CameraCaptureSession, which will be introduced later. | | Surface | Surface is a piece of used to populate image data memory space, for example, you can use a SurfaceView Surface receives each frame preview data is used to display the preview screen, You can also use ImageReader’s Surface to receive JPEG or YUV data. Each Surface has its own size and data format. You can get a list of sizes supported by a particular data format from Camera Acteristics. | | CameraCaptureSession | CameraCaptureSession actually is configured with the target Surface Pipeline as an example, We must create an instance of CameraCaptureSession before using the camera functionality. CameraDevice can only start one CameraCaptureSession at a time. Most camera operations are implemented by submitting aCapture request to CameraCaptureSession. For example, take photos, shoot in tandem, set flash mode, touch focus, display preview screen and so on. | | CaptureRequest | CaptureRequest is submitted to CameraCaptureSession the Capture request information carrier, the interior includes the Capture of Surface parameters configuration and receive image data. CaptureRequest can be configured with a lot of information, including image format, image resolution, sensor control, flash control, 3A control and so on. It can be said that most of the camera parameters are configured through CaptureRequest. It is important to note that each Capture Capture represents one frame of action, which means that you have precise control over the Capture action for each frame. | | CaptureResult | CaptureResult are the result of every time the Capture operation, including a lot of state information, including flash state, focus, timestamp, etc. For example, you can use CaptureResult to get the focus status and timestamp of the photo when it is finished. Note that CaptureResult does not contain any image data, as we mentioned earlier in the Surface section that the image data is obtained from the Surface. | | ImageReader | used to read from the camera open channel to the format of the original image data, you can set multiple ImageReader. |

The development process

20210224002.png

1. Obtain CameraManager

private val cameraManager: CameraManager by lazy { getSystemService(CameraManager::class.java) }

Copy the code

2. Obtain camera information

val cameraIdList = cameraManager.cameraIdList
cameraIdList.forEach { cameraId ->
    val cameraCharacteristics = cameraManager.getCameraCharacteristics(cameraId)
    if (cameraCharacteristics.isHardwareLevelSupported(CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL_FULL)) {
        if (cameraCharacteristics[CameraCharacteristics.LENS_FACING] == CameraCharacteristics.LENS_FACING_FRONT) {
            frontCameraId = cameraId
            frontCameraCharacteristics = cameraCharacteristics
        } else if (cameraCharacteristics[CameraCharacteristics.LENS_FACING] == CameraCharacteristics.LENS_FACING_BACK) {
            backCameraId = cameraId
            backCameraCharacteristics = cameraCharacteristics
        }
    }
}

Copy the code

Through CameraManager access to all the cameras cameraId, cycle through judgment is camera (CameraCharacteristics. LENS_FACING_FRONT) before or after the camera (CameraCharacteristics LENS_FA CING_BACK)

3. Initialize ImageReader

private var jpegImageReader: ImageReader? = null jpegImageReader = ImageReader.newInstance(imageSize.width, imageSize.height, ImageFormat.JPEG, 5) jpegImageReader? .setOnImageAvailableListener(OnJpegImageAvailableListener(), cameraHandler) ...... private inner class OnJpegImageAvailableListener : ImageReader.OnImageAvailableListener { private val dateFormat: DateFormat = SimpleDateFormat("yyyyMMddHHmmssSSS", Locale.getDefault()) private val cameraDir: String = "${Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DCIM)}/Camera" @WorkerThread override fun onImageAvailable(imageReader: ImageReader) { val image = imageReader.acquireNextImage() val captureResult = captureResults.take() if (image ! = null && captureResult ! = null) { image.use { val jpegByteBuffer = it.planes[0].buffer// Jpeg image data only occupy the planes[0]. val jpegByteArray = ByteArray(jpegByteBuffer.remaining()) jpegByteBuffer.get(jpegByteArray) val width = it.width val height = it.height saveImageExecutor.execute { val date = System.currentTimeMillis() val title = "IMG_${dateFormat.format(date)}"// e.g. IMG_20190211100833786 val displayName = "$title.jpeg"// e.g. IMG_20190211100833786.jpeg val path = "$cameraDir/$displayName"// e.g. /sdcard/DCIM/Camera/IMG_20190211100833786.jpeg val orientation = captureResult[CaptureResult.JPEG_ORIENTATION] val location = captureResult[CaptureResult.JPEG_GPS_LOCATION] val longitude = location?.longitude ?: 0.0val latitude = location?.latitude?: 0.0 // Write the jpeg data into the specified file.file (path). WriteBytes (jpegByteArray) // Insert the image information  into the media store. val values = ContentValues() values.put(MediaStore.Images.ImageColumns.TITLE, title) values.put(MediaStore.Images.ImageColumns.DISPLAY_NAME, displayName) values.put(MediaStore.Images.ImageColumns.DATA, path) values.put(MediaStore.Images.ImageColumns.DATE_TAKEN, date) values.put(MediaStore.Images.ImageColumns.WIDTH, width) values.put(MediaStore.Images.ImageColumns.HEIGHT, height) values.put(MediaStore.Images.ImageColumns.ORIENTATION, orientation) values.put(MediaStore.Images.ImageColumns.LONGITUDE, longitude) values.put(MediaStore.Images.ImageColumns.LATITUDE, latitude) contentResolver.insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values) // Refresh the thumbnail of image. val thumbnail = getThumbnail(path) if (thumbnail ! . = null) {runOnUiThread {thumbnailView setImageBitmap (thumbnail) thumbnailView. ScaleX = 0.8 F thumbnailView. ScaleY = 0.8f ThumbnailView.animate ().setDuration(50).scalex (1.0f).scaley (1.0f).start()}}}}}}}Copy the code

ImageReader is an important way to obtain image data, through which you can obtain different formats of image data, such as JPEG, YUV, RAW and so on. Create an ImageReader object using ImageReader. NewInstance (int Width, int height, int format, int maxImages) with 4 parameters:

  • Width: indicates the width of the image data
  • Height: Height of the image data
  • Format: Format of image data, for exampleImageFormat.JPEG.ImageFormat.YUV_420_888Etc.
  • MaxImages: specifies the maximum number of images that can be retrieved from the ImageReader. Too many images may result in OOM, so it is best to set this value to minimize the need

Other ImageReader related methods and callbacks:

  • ImageReader.OnImageAvailableListener: callback with new image data
  • acquireLatestImage(): Retrieves the latest Image from the ImageReader queue, removes the old Image, and returns null if no Image is available
  • acquireNextImage(): Retrieves the next latest available Image, or null if none is available
  • close(): Frees all resources associated with this ImageReader
  • getSurface(): Gets the Surface that generates the Image for the current ImageReader

4. Turn on the camera

val cameraStateCallback = CameraStateCallback() cameraManager.openCamera(cameraId, cameraStateCallback, mainHandler) ...... private inner class CameraStateCallback : CameraDevice.StateCallback() { @MainThread override fun onOpened(camera: CameraDevice) { cameraDeviceFuture!! .set(camera) cameraCharacteristicsFuture!! .set(getCameraCharacteristics(camera.id)) } @MainThread override fun onClosed(camera: CameraDevice) { } @MainThread override fun onDisconnected(camera: CameraDevice) { cameraDeviceFuture!! .set(camera) closeCamera() } @MainThread override fun onError(camera: CameraDevice, error: Int) { cameraDeviceFuture!! .set(camera) closeCamera() } }Copy the code

cameraManager.openCamera(@NonNull String cameraId,@NonNull final CameraDevice.StateCallback callback, @nullable Handler Handler).

  • CameraId: Unique identifier of the camera
  • Callback: callback for changes in device connection status
  • Handler: The handler object to which the callback is executed. If null is passed, the current main thread handler is used

The CameraStateCallback callback:

  • OnOpened: If the camera is opened successfully, you can use the camera and create a Capture session
  • OnDisconnected: This method is called back when the camera is disconnected, requiring the camera to be released
  • OnError: Release the camera when the camera fails to open
  • OnClosed: callback method after camera.close () is called

5. Create Capture Session

val sessionStateCallback = SessionStateCallback() ...... val cameraDevice = cameraDeviceFuture? .get() cameraDevice? .createCaptureSession(outputs, sessionStateCallback, mainHandler) ...... private inner class SessionStateCallback : CameraCaptureSession.StateCallback() { @MainThread override fun onConfigureFailed(session: CameraCaptureSession) { captureSessionFuture!! .set(session) } @MainThread override fun onConfigured(session: CameraCaptureSession) { captureSessionFuture!! .set(session) } @MainThread override fun onClosed(session: CameraCaptureSession) { } }Copy the code

This piece of code core method is mCameraDevice createCaptureSession () to create the Capture session, it accepts three parameters:

  • Outputs: Collection of surfaces used to receive image data, in which case a Preview surface is passed in
  • The callback: listens for the Session state CameraCaptureSession. StateCallback object
  • Handler: used to perform CameraCaptureSession. StateCallback handler object, pass in null use the main thread of the current handler

6. Create CaptureRequest

CaptureRequest is the information carrier when submitting aCapture request to CameraCaptureSession. The CaptureRequest contains the parameters of Capture and the Surface to receive image data

if (cameraDevice ! = null) { previewImageRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW) captureImageRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE) } ...... val cameraDevice = cameraDeviceFuture? .get() val captureSession = captureSessionFuture? .get() val previewImageRequestBuilder = previewImageRequestBuilder!! val captureImageRequestBuilder = captureImageRequestBuilder!! if (cameraDevice ! = null && captureSession ! = null) { val previewSurface = previewSurface!! val previewDataSurface = previewDataSurface previewImageRequestBuilder.addTarget(previewSurface) // Avoid missing preview frame while capturing image. captureImageRequestBuilder.addTarget(previewSurface) if (previewDataSurface ! = null) { previewImageRequestBuilder.addTarget(previewDataSurface) // Avoid missing preview data while capturing image. captureImageRequestBuilder.addTarget(previewDataSurface) } val previewRequest = previewImageRequestBuilder.build() captureSession.setRepeatingRequest(previewRequest, RepeatingCaptureStateCallback(), mainHandler) } ...... private inner class RepeatingCaptureStateCallback : CameraCaptureSession.CaptureCallback() { @MainThread override fun onCaptureStarted(session: CameraCaptureSession, request: CaptureRequest, timestamp: Long, frameNumber: Long) { super.onCaptureStarted(session, request, timestamp, frameNumber) } @MainThread override fun onCaptureCompleted(session: CameraCaptureSession, request: CaptureRequest, result: TotalCaptureResult) { super.onCaptureCompleted(session, request, result) } }Copy the code

In addition to mode configuration, CaptureRequest can also be configured with many other information, such as image format, image resolution, sensor control, flash control, 3A(autofocus -AF, auto exposure -AE, and auto white balance -AWB) control. This can be set in the createCaptureSession callback, and finally the CaptureRequest object is generated through the build() method.

7. Preview

In Camera2, Capture is used repeatedly to preview images. Each time, Capture will display the preview images on the corresponding Surface. Consecutively repeated Capture operations pass. Procedure

captureSession.setRepeatingRequest(previewRequest, RepeatingCaptureStateCallback(), mainHandler)

This method takes three parameters:

  • Request: CaptureRequest object
  • Listener: Listens for a callback to the Capture status
  • Handler: used to perform CameraCaptureSession. CaptureCallback handler object, pass in null use the main thread of the current handler

Stop use preview mCaptureSession. StopRepeating () method.

8. Photo

After setting up the request session above, you can actually start taking photos

val captureImageRequest = captureImageRequestBuilder.build() captureSession.capture(captureImageRequest, CaptureImageStateCallback(), mainHandler) ...... private inner class CaptureImageStateCallback : CameraCaptureSession.CaptureCallback() { @MainThread override fun onCaptureStarted(session: CameraCaptureSession, request: CaptureRequest, timestamp: Long, frameNumber: Long) { super.onCaptureStarted(session, request, timestamp, frameNumber) // Play the shutter click sound. cameraHandler? .post { mediaActionSound.play(MediaActionSound.SHUTTER_CLICK) } } @MainThread override fun onCaptureCompleted(session: CameraCaptureSession, request: CaptureRequest, result: TotalCaptureResult) { super.onCaptureCompleted(session, request, result) captureResults.put(result) } }Copy the code

CaptureSession. The capture () method also has three parameters, and mCaptureSession setRepeatingRequest:

  • Request: CaptureRequest object
  • Listener: Listens for a callback to the Capture status
  • Handler: used to perform CameraCaptureSession. CaptureCallback handler object, pass in null use the main thread of the current handler

9. Turn off the camera

As with other hardware resources, call the cameraDevice.close () method to turn off the camera recycling resource when the camera is no longer needed. Turning off the camera is very important, because if you keep occupying the camera resources, other camera-based functions will not work properly, and in serious cases, other camera-related apps will not work properly. When the camera is fully closed by CameraStateCallback. OnCllosed () method to inform you that the camera has been closed. So when is the best time to turn the camera off? My personal advice is to always turn off the camera during onPause(), since the camera page is no longer the user’s focus at this point, and in most cases the camera can already be turned off.

cameraDevice? .close() previewDataImageReader? .close() jpegImageReader? .close()Copy the code

Close CaptureSession, CameraDevice and ImageReader successively to release resources.

Recommendations for migrating from Camera1 to Camera2

If your project is using Camera1 and you are planning to migrate from Camera1 to Camera2, I hope the following suggestions will help you:

  1. Camera2 abstracts these two processes into Capture behavior. Therefore, it is recommended that you do not use Camera2 with too much thinking. You should not be able to make full use of the flexible API of Camera2.
  2. As with Camera1, some of the CAMERA API calls are time-consuming, so it is recommended that you use a separate thread to perform all camera operations. Try to avoid calling the Camera API directly from the main thread. HandlerThread is a good choice.
  3. You can think of Camera1 as a subset of Camera2, that is, what Camera1 can do, Camera2 can do, and vice versa.
  4. If your application needs to be compatible with both Camera1 and Camera2, I recommend maintaining it separately, because Camera1’s poor API design is likely to prevent the use of Camera2’s flexible API. The pain of mixing two things that are completely incompatible in design may outweigh the convenience, and you may be happier writing redundant code.
  5. Officials say Camera2 performance will be better, but performance on earlier machines is not much better;
  6. Camera1 is recommended when the Supported Hardware Level of the device is lower than FULL. Camera2 provides almost the same functions as Camera1, so it is better to use the more stable Camera1.

Full code: github.com/zhijunhong/…

Finally, if this post helps you, don’t forget to give it a thumbs up

Reference:

The Android Camera – Camera2 use

Android Camera2 tutorial