Webrtc-android source code guide (a) : camera acquisition analysis

blog.piasy.com
Blog.piasy.com/2017/07/24/…

The front shared a set of WebRTC development environment out of the box, hope to bring help to friends interested in WebRTC. But with the development environment is just the first step in the long march, the things behind have to carefully read the source code, so here to bring WebRTC Android Demo project – APPRTC-Android source guide series. (Dagger2 will have to wait, Haihan haihan…)

The amount of WebRTC code is too large to read all at once. In this series, I’m going to try to figure out three things:

1. How to establish a connection between clients?
2. How to transfer data between clients?
3. The complete process of audio and video data collection, preview, coding, transmission, decoding and rendering.

This article is the first, I will start from the most familiar acquisition, analysis of the implementation of weBRTC-Android camera acquisition.

Webrtc-android camera collection is expected of Enumerator, CapErator, Session, And SurfaceTextureHelper. That’s what’s expected of you.

Expected Of Women, Caperator generates Session, which is expected of Women. SurfaceTextureHelper receives SurfaceTexture data.

Enumerator

The CameraEnumerator interfaces are as follows:

public interface CameraEnumerator {
  public String[] getDeviceNames();
  public boolean isFrontFacing(String deviceName);
  public boolean isBackFacing(String deviceName);
  public List<CaptureFormat> getSupportedFormats(String deviceName);
  public CameraVideoCapturer createCapturer(
      String deviceName, CameraVideoCapturer.CameraEventsHandler eventsHandler);
}
Copy the code

It’s getting the device list, checking the orientation, creating capWomen. That’s what’s expected of you.

This is a typical abstraction that consolidates (and encapsulates) the different apis of Camera1 and Camera2 with apI-independent interfaces, and deviceName is at the heart of this unification. Map all camera devices under different apis to their own deviceName, and subsequent operations can specify deviceName. You can also use an integer ID instead of a string, but the readability becomes worse.

Capturer

initialize

Initialize is simple and simply holds the related objects passed in.

startCapture

stopCapture

StopCapture determines whether a session is being created. If a session is being created, wait until it is created. After this check, close the session in the camera thread if there is one currently running.

changeCaptureFormat

switchCamera

SwitchCamera will also stop the old session and create a new session, but also need to check the number of cameras, switch status notification logic.

There should be a slight problem with this code: StartCapture sets openAttemptsRemaining to MAX_OPEN_CAMERA_ATTEMPTS, but only sets it to 1 when switching the camera. This asymmetry should not make sense, so I think it should be consistent.

Session

Camera1

• createCameraObject:Camera.open;
• Set the preview SurfaceTexture to receive frame data (in video memory) :camera.setPreviewTexture;
• Select the appropriate camera preview parameters (size, frame rate, focus) :Parameters 和 camera.setParameters;
• If you need to get a memory data callback, you need to set buffer and listener:camera.addCallbackBuffer 和 camera.setPreviewCallbackWithBuffer;
• If you need the camera service to adjust the data direction for us, you can set the rotation Angle:camera.setDisplayOrientation;
• Enable preview:camera.startPreview;
• Stop preview:camera.stopPreview 和 camera.release;

Camera2

Image direction

Usually the front camera outputs an image orientation of 270° counterclockwise and the rear camera 90°, but there are a few exceptions, such as the Nexus 5X’s 270° front and back.

We can through the camera in Camera1. SetDisplayOrientation interface to control the output image of the camera Angle, but in fact whether access to memory data, or access to memory data (SurfaceTexture), the call will change the data, It only affects the direction of the transformation matrix carried by the camera as it outputs data. There is no corresponding interface in Camera2, but the camera service automatically adjusts the orientation of the transformation matrix for us, so it is equivalent to calling a similar interface correctly.

If using the camera. SetPreviewDisplay or camera. SetPreviewTexture realize preview, the camera. The setDisplayOrientation do let preview image direction of change, Because the camera service applies a transformation matrix when rendering to SurfaceView/TextureView, the preview screen is rotated.

Besides the problems the direction as well as a mirror, Camera1 in front camera automatically flip the picture for us (of course) transformation matrix is modified, for example, the front-facing camera output image direction is counterclockwise, 270 °, the image upside down, would have been if we set up a rotated 90 °, the image is, That’s the equivalent of flipping left to right, and it’s a mirror image. In the front camera, we touch the left side of the face with our left hand, and the preview also shows it on the left side of the screen. .

To set the camera. SetPreviewDisplay parameter so that the direct preview can be oriented correctly, you can use the following code:

private static int getRotationDegree(int cameraId) { int orientation = 0; WindowManager wm = (WindowManager) applicationContext .getSystemService(Context.WINDOW_SERVICE); switch (wm.getDefaultDisplay().getRotation()) { case Surface.ROTATION_90: orientation = 90; break; case Surface.ROTATION_180: orientation = 180; break; case Surface.ROTATION_270: orientation = 270; break; case Surface.ROTATION_0: default: orientation = 0; break; } if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) { return (720 - (cameraInfo.orientation + orientation)) % 360; } else { return (360 - orientation + cameraInfo.orientation) % 360; }}Copy the code

SurfaceTextureHelper

// The onFrameAvailable() callback will be executed on the SurfaceTexture ctor thread. // See: http://grepcode.com/file/repository.grepcode.com/java/ext/com.google.android/ // Android / 5.1.1 _r1 / android/graphics/SurfaceTexture java# 195. / / Therefore, in order to control the callback thread on API lvl < 21, // the SurfaceTextureHelper is constructed on the |handler| thread.Copy the code

What are the pit

• On earlier (pre-5.0) systems, when Camera1 stops previewing, do not manually call the following interface to set null values:setPreviewDisplay/setPreviewCallback/setPreviewTexture(The documentation does say not to call…) Otherwise, all system services may crash and the phone may restart:

• There may be a deadlock in stopping Camera1 preview (not resolved) :

// Note: stopPreview or other driver code might deadlock. Deadlock in
// android.hardware.Camera._stopPreview(Native Method) has been observed on
// Nexus 5 (hammerhead), OS version LMY48I.
camera.stopPreview();
Copy the code

• Camera2 related code encounters VerifyError on systems prior to 4.4.2:

try {
    return cameraManager.getCameraIdList();
    // On Android OS pre 4.4.2, a class will not load because of VerifyError if it contains a
    // catch statement with an Exception from a newer API, even if the code is never executed.
    // https://code.google.com/p/android/issues/detail?id=209129
} catch (/* CameraAccessException */ AndroidException e) {
    Logging.e(TAG, "Camera access exception: " + e);
    return new String[] {};
}
Copy the code

• Using SurfaceTexture receive frame data, some models may access to data is black screen (MX5 met) : you need to set up SurfaceTexture of buffer size, SurfaceTexture. SetDefaultBufferSize
• Using SurfaceTexture to receive frame data throughSurfaceTexture.getTimestampInterface to get the timestamp and the timestamp is relative to time, and there will be a few frames in front of the value is 0, the problem of relative time can be in the first frame and the physical time difference, and then calculate the subsequent physical time stamp of each frame, but the head of a few frames timestamp is 0, so we have to wait until write down the difference is not zero, while the first few frames can be used directly physical time as a timestamp;
• surfaceTexture.updateTexImage 和 eglSwapBuffersDeadlocks occur and we need to lock them ourselves:

// SurfaceTexture.updateTexImage apparently can compete and deadlock with eglSwapBuffers,
// as observed on Nexus 5. Therefore, synchronize it with the EGL functions.
// See https://bugs.chromium.org/p/webrtc/issues/detail?id=5702 for more info.
synchronized (EglBase.lock) {
  surfaceTexture.updateTexImage();
}

synchronized (EglBase.lock) {
  EGL14.eglSwapBuffers(eglDisplay, eglSurface);
}
Copy the code

• On some models, useTextureViewImplementation preview,onSurfaceTextureAvailableThe callback will not be called and the preview will not be enabled, which may be resolved by enabling hardware acceleration (see.StackOverflowBut chances are that doesn’t work either, so you’ll need to expend some brain cells. This happened to me on an OPPO 4.3 phone and I had to wait a while to reset itLayoutParamsIt would trigger it, so I did it first;

Memory jitter Optimization

Running apprtC-Android, we will find that memory jitter is very serious:

We can use Allocation Tracker to analyze and optimize this part. We will arrange and release the specific content and other basic content after release. Please look forward to it

Subsequent articles