preface
With the acceleration of the online and digitalization of the global industrial chain, real-time screen sharing on mobile terminals has been widely applied in all walks of life, such as online education, video conferencing, remote business consulting and mobile game live broadcasting. Screen acquisition is the first step in the process of realizing real-time screen sharing. This technology share will tell you about The experience and practice of Screen acquisition of PaileYun in Andorid terminal.
background
Android has provided mobile screen recording methods since 4.0, but requires root permission. As of 5.0, Google has opened the system recording API: MediaProjection and MediaProjectionManager do not require root permission, but will pop up a screen recording permission application box, and users can only start recording after they agree, similar to the permission application process after Android6.0. In view of the low proportion of Android phones below 5.0 on the market and the complexity of screen acquisition requiring root permission, we will mainly introduce the principle of screen acquisition of Android5.0 and above. Imagine what a complete screen capture process would look like. Screen data sources (producers) generate data in the buffer, and screen data consumers extract data from the buffer for use. Different consumers can realize different functions, such as screen saving and screen live recording (screen sharing). Who plays these key roles on Android? VirtualDisplayVirtualDisplay is on Android virtual display. In this article, the function of VirtualDisplay is to grab the content displayed on the screen. It is the producer of screen data. In the Window implementation of Android, Surface corresponds to a screen data buffer. Screen data producers can produce data on the Surface, and consumers can extract data from the Surface for use. Screen capture process
After introducing the above key roles, we can roughly draw a screen capture flow chart:
The code implementation is described step by step.
Access to MediaProjection
The MediaProjectionManager service is used to obtain an Intent that applies for screen capture permission and launch the screen capture permission request interface: mediaProjectionManager = (MediaProjectionManager) getSystemService(MEDIA_PROJECTION_SERVICE); Intent intent = mediaProjectionManager.createScreenCaptureIntent(); startActivityForResult(intent, SCREEN_CAPTURE_REQUEST_CODE); The screen capture permission application page is as follows:
When the user allows (click Start now) to retrieve MediaProjection in the onActivityResult callback based on the returned resultCode and data:
protected void onActivityResult(int requestCode, int resultCode, Intent data) { super.onActivityResult(requestCode, resultCode, data);
if (requestCode == SCREEN_CAPTURE_REQUEST_CODE && resultCode == Activity.RESULT_OK) { mediaProjection = mediaProjectionManager.getMediaProjection(resultCode, data); }} If the targetSdkVersion is greater than or equal to 29, the screen capture is restricted, and the corresponding foreground Service must be started before the getMediaProjection method can be called. Otherwise, an exception will be thrown: java.lang.SecurityException: Media projections require a foreground service of type ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION If (REQUIRE_FG_SERVICE_FOR_PROJECTION //1) if (REQUIRE_FG_SERVICE_FOR_PROJECTION //1) The default is true && requiresForegroundService () / / 2. The current APP needs to start foreground Service &&! mActivityManagerInternal.hasRunningForegroundService( //3. The foreground service UID is not enabled for the current application. ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION)) { throw new SecurityException(“Media projections require a foreground service” + ” of type ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION”); }
/ / APP TargetSdkVersion greater than or equal to 29 and not privilege application (usually privilege application), then returns true (need to start the service reception) Boolean requiresForegroundService () {return mTargetSdkVersion >= Build.VERSION_CODES.Q && ! mIsPrivileged; } The foreground Service configuration reference is as follows:
2. Structure Surface
1. If the screen capture data is used to record video, then the consumer can be MediaRecoder, and the corresponding Surface is provided by MediaRecoder: Surface Surface = mediaRecorder.getSurface(); 2. If the screen capture data is used for screen sharing (recording screen live), then the consumer can be an encoder like MediaCodec, and the Surface is provided by MediaCodec accordingly: Surface surface = mediaCodec.createInputSurface(); 3. If the screen data needs to be displayed on the UI SurfaceView, the Surface can be generated in the following ways: SurfaceView surfaceView = (SurfaceView) findViewById(R.id.surface); Surface surface = surfaceView.getHolder().getSurface(); SurfaceTexture = new SurfaceTexture(textureId) SurfaceTexture = new SurfaceTexture(textureId) surfaceTexture.setOnFrameAvailableListener(new OnFrameAvailableListener() {
@Override public void onFrameAvailable(SurfaceTexture surfaceTexture) {
} }, handler); Surface surface = new Surface(surfaceTexture); SurfaceTexture is briefly introduced here. SurfaceTexture can be used to capture image frames in a video stream. When data is updated in the SurfaceTexture, the onFrameAvailable callback is triggered. The updateTexImage method can then be called to update the current data frame from the video stream data.
Create VirtualDisplay
MediaProjection has a ready-made API to call: public VirtualDisplay createVirtualDisplay(String name, int width, int height, int dpi, int flags, Surface surface, VirtualDisplay.Callback callback, Handler handler) {
DisplayManager dm = (DisplayManager) mContext.getSystemService(Context.DISPLAY_SERVICE); return dm.createVirtualDisplay(this, name, width, height, dpi, surface, flags, callback, handler, null /* uniqueId */); } Parameters are described as follows:
Each parameter of Android has a detailed description in the official documentation, among which flag and surface are given an additional description:
Flag is the flag bit of VirtualDisplay. Generally, the value is VIRTUAL_DISPLAY_FLAG_PUBLIC. The surface, also known as the on-screen data buffer mentioned above, is typically supplied by consumers.
4. Screen acquisition data processing
Take the Surface generated from the SurfaceTexture in step 2. When data is updated in the SurfaceTexture, the onFrameAvailable callback is triggered, and we can perform specific processing on the data in that callback. @Override public void onFrameAvailable(SurfaceTexture surfaceTexture) { dealTextureFrame(); }
private void dealTextureFrame() { … surfaceTexture.updateTexImage(); float[] transformMatrix = new float[16]; surfaceTexture.getTransformMatrix(transformMatrix); . }
V. Resolution and frame rate control
In screen sharing, high resolution means clarity, and high frame rate means fluency. It is often impossible to have your cake and eat it, especially in the case of limited network and device performance.
When the mobile phone screen is still in an interface or the interface moves at a low speed, we grab the screen at a low frame rate so that the receiver will not feel stuck and lose frames when watching. At this time, the screen acquisition resolution can be appropriately improved to make the picture quality clearer. On the contrary, if the scene is a game live broadcast and the screen interface moves rapidly, the screen content needs to be captured at a high frame rate to make the receiver have a smooth viewing experience. However, under the circumstance of limited resources, some clarity may need to be sacrificed.
Screen capture resolution control is relatively simple, in the third step to create VirtualDisplay, pass in the required width and height values.
The upper limit of the screen capture frame rate is determined by the screen refresh rate of the Android device, and the lower limit is 0, that is, all returned data is discarded and not processed. Capture frame rate is not the higher the better, enough on the line. For example, on low-end computers, even if the screen data is collected at a higher frame rate, the frame rate transmitted on the screen is not up to the frame rate collected due to the limitations of the machine’s encoding and decoding ability, which will consume too much system resources and lead to heating and stalling. In this case, the acquisition frame rate needs to be appropriately reduced. In the onFrameAvailable callback, a specific algorithm regularly dismisses some data to reduce the capture frame rate.
Six, horizontal and vertical screen switching
Switching between vertical and horizontal screens is common in live games. For example, when the host of King of Glory switches the account, he needs to kill the King of Glory APP to return to the main interface of the mobile phone, and then open the King of Glory to log in again, after switching from landscape to portrait and then back to landscape.
Screen capture also needs to be dynamically adjusted according to the different horizontal and vertical screen modes. The premise of adjustment is how to sense changes in vertical and horizontal modes.
Use OrientationEventListener if you want to listen for the physical flip of the phone. However, for some apps with mandatory landscape display, such as King of Glory, you can directly open these apps by placing your phone flat on the horizontal desktop. The interface after entering the APP is displayed in landscape, and the Angle changes detected by OrientationEventListener cannot determine whether the APP interface is displayed in landscape.
In fact, we need to be aware of the current vertical display of the screen rather than the physical flip of the phone.
The rotation value of Display has the following values:
public static final int ROTATION_0 = 0; Public static final int ROTATION_90 = 1; Public static final int ROTATION_180 = 2; Public static final int ROTATION_270 = 3; ROTATION_0 and ROTATION_180 represent two portrait states, ROTATION_90 and ROTATION_270 represent two landscape states. We only care about whether the interface has been switched between horizontal and vertical screen. As for left landscape screen or right landscape screen, it does not affect the acquisition effect.
private boolean checkRotationChange() { int currentRotation = display.getRotation(); boolean rotationChange = false; if ((currentRotation + lastRotation) % 2 == 1) { rotationChange = true; } lastRotation = currentRotation; return rotationChange; }
conclusion
In this paper, the screen data producers and data buffers involved in Android screen acquisition are briefly introduced. In fact, the processing of original screen data by consumers is a key step in the whole screen sharing process. In addition to the screen acquisition of resolution, frame rate control, horizontal and vertical screen switching adaptation and other issues are only theoretical elaboration, specific code implementation or a lot of details need to pay attention to.