WebRTC is a very good project, can support the Web, iOS, Android, Mac, Windows, Linux and other platforms of the API, ensure the consistency of the API across all platforms. However, WebRTC’s performance on mobile terminals is not so satisfactory compared with PC, especially on Android system, which has been criticized for its fragmentation for a long time. Every time the Android system is upgraded, each chip manufacturer and mobile phone manufacturer will make some customization based on the Android system, resulting in different performance of different devices even if the version of Android system is the same and the standard API of Android is called. Therefore, without adaptation for different models, it is difficult to achieve a unified user experience and ensure the stability of functions.

In RTC scenarios, to achieve high reliability, stability, low latency, and device generality on Android devices, audio and video compatibility is essential. The refined configuration and intelligent configuration of relevant parameters of Android devices are the focus of this paper.

How to refine the configuration

The mainstream SOC (System On Chip) schemes are developed based On Linux/Android System, but each has its own advantages. Qualcomm SOC is mainly used in mobile phone devices, with strong codec performance and stability, and good control of video over-editing problems; MTK and MStar after the merger, in addition to mobile phone products, set-top box and large screen products are widely used, often through MTK own parameters, open some functions. In addition, different chip versions have different computing performance, which will affect the opening of audio and video functions, such as video processing before and after. The current mainstream chip scheme is shown as follows:

Based on the SOC platforms mentioned above, various device manufacturers will customize different Android smart devices, and each type of device has different features.

Such as:

  • Android Phone: Mobile phone battery consumption requirements are relatively strict, the device needs to meet the 360 Angle rotation experience, etc.
  • Android TV Large Screen device:
    • Some use an external camera;
  • In the sound acquisition scene, the distance between human and equipment is often far, which poses a challenge to audio data acquisition with high sound quality.
  • Android Watch: Small screen, weak CPU performance, etc.

Mobile devices are the most common and highly used devices. Most phone manufacturers have customized Android, so each has its own ROM. Mobile phone manufacturers usually customize the Android Framework, HAL, and Linux driver. As a result, different mobile phones will perform differently even if they use the same SOC. Mainstream Android phone brands and customized ROM are shown below:

Therefore, according to the above analysis, there are so many compatibility problems. For products with audio and video as the main scene, both audio and video modules need to be refined and compatible. You can refer to the following configuration parameters (currently, not all of them can be configured) :

  • Audio: Basic features, sound effects, audio strategies, etc
  • Video: basic functions, video processing and so on

Detailed parameters are shown as follows:

How to Configure intelligently

For the compatibility configuration of audio and video, parameters should be modified, timeliness, flexibility, automation and rollback. The current Configuration schemes for Delivering Android parameters can be divided into the following four types:

Below, let’s talk about each of these four points in detail and analyze their pros and cons.

Compatibility code BuiltIn

The compatibility solution is to do it directly in the code logic. This solution only covers a few scenarios, such as compatibility for different Android SDK versions and configurations that have been clearly defined in testing.

  • Benefits: It is directly written in the SDK and does not need to be delivered from the server. It does not occupy network bandwidth and is efficient.
  • Disadvantages: But for special models with problems online, the Settings are often not flexible enough. When there are a lot of problems with different users online, the SDK needs to be changed to solve the problems, which often feels like far water can not solve the near thirst.

Local File Configuration

The local file configuration mode is that the parameters take effect by reading the configuration file in the specified local directory. This step is usually performed during SDK initialization.

  • Benefits: You do not need to repackage the SDK. You can directly save the configuration file to the specified directory for the parameters to take effect. You do not need to modify the parameters to the server for the parameters to take effect. This is suitable for local parameter setting scenarios.
  • Cons: But can’t solve online customer problems remote modification.

Delivering Server Parameters

Set server delivery parameters. You can control device function parameters by delivering parameters at any time.

  • Benefits: If online users encounter compatibility problems, they can directly modify parameters through the server. The problem can be solved in a few minutes.
  • Disadvantages: If the parameter file is delivered through the server, it occupies server resources. If the parameter file is too large, the SDK initialization time will be affected.

Automation policy selection

The automatic strategy selection of parameters is to form several sets of parameter templates, such as low performance requirement parameter templates or quality priority parameter templates, based on the combination of parameters of different functional modules by comprehensively considering different system versions, different CPU computing capabilities, chip platforms and other factors. If improper parameter Settings cause problems, you can roll back to basic parameter Settings to ensure that basic functions work properly.

Compatibility rules

The preceding two chapters describe common parameters of refined configuration and solutions of intelligent configuration. How to distinguish different device models, service scenarios, and system versions?

Now, let’s take a closer look at the dimensions of making compatibility rules.

Adapt to individual devices

It will be difficult for phone makers to be completely consistent when customizing based on the new Android operating system, especially in audio and video. The uniqueness of each device can be identified by the hardware manufacturer, mainboard, device version, and device parameters.

This adaptation depends on the DEVICE CPU

Adaptation is performed based on the CPU computing capability, for example, video pre – and post-processing. After this mode is selected, all devices with the same CPU model can be adapted.

Adaptation based on the system version

Each update to the Android system involves changes to the audio and video apis. You need to adapt the device based on the Android version.

Adaptation based on different application services

Different services have different emphases on audio and video adaptation. For example, the multi-person meeting scenario has high requirements on real-time performance of audio and video, stability under weak network, and audio noise reduction and echo cancellation. In the cloud game scenario, the video resolution and delay are highly required. Music playback in entertainment services, such as the cloud music listening function, has high requirements on sound quality, and has special requirements on audio device routing policies. For example, music is played from Bluetooth, but audio data is collected from mobile mic. These scenarios require different parameters to meet the requirements.

Applicable to the IOT device type

Android system is applied in various IOT devices, resulting in a lot of adaptation scenarios. For example, TV large screen, because the screen is relatively large, so the video content is required to be high resolution, high frame rate, so the ability to collect and decode relatively high requirements; And there are some TV that can rotate 360 degrees in the market, which needs to be adapted in the display Angle; In terms of the sound quality of the large screen, people are often far away from the TV, so the processing of sound collection and echo cancellation is not quite the same as that of mobile phones.

Compatibility Common audio and video problems are addressed

Let’s talk about common audio and video compatibility problems, their causes and how we solve them.

Audio Compatibility FaQs

Audio compatibility often involves problems with sound quality, volume, and audio strategy. Here are a few of them and a brief analysis of the solutions to these problems.

  • Problem: The audio delay is too large or unstable

    • Solution: When creating Android self-collection, the Buffer size set in the AudioRecord affects the collection end delay, thus affecting the accuracy of AEC algorithm.

  • Problem encountered: In Bluetooth A2DP mode, playback from Bluetooth cannot be collected from the mic of the device

    • Solution: Distinguish the types of AudioSource and AudioMode in SCO/A2DP mode

  • Problem: The volume of the sending end is low

    • Solution 1: The volume of the collected audio data source is relatively low, and the volume is enhanced by the software AGC algorithm.
    • Solution 2: Modify the AudioSource. In some mobile phones, especially old ones, the voice in AudioSource.VOICE_COMMUNICATION mode is low or the audio channel processing is faulty.

  • Problem: The audio playback quality is low

    • AudioTrack streamType audioSystem.stream_music;

  • Error: Audio 3A (AEC AGC, ANS); The phenomenon is that the sound is up and down, the voice is suppressed, the background noise is serious, the volume is too high or too low
    • The solution: It’s usually a combination of hardware 3A and software algorithm 3A.

There are also some problems such as calling system API Freeze and not being able to play with OpenSL ES. We have also explored corresponding solutions, which will not be described here.

Common Video Compatibility Problems

Video compatibility often has problems such as stuttering, abnormal display picture and codec failure. Here we select several of them and briefly analyze the solutions to these problems.

  • Problems encountered: There are red bars in the collection screen:

  • Solution: Specific resolution, frame rate, resulting in acquisition with red stripes

  • Problems encountered: Collecting image black:

  • Solution: Certain frame rates cause exposure problems.

  • Encountered problems: occasionally collection screen split

  • Solution: Texture capture format results

  • Problems encountered: some mobile phone Camera2 collection has a green edge

    • Solution: Camera2 is not compatible

  • Problem encountered: the actual code rate of the hardware coding with the code rate difference

    • Solution: On some phones, even if Mediacodec is set to CBR, bit rate fluctuation is still large

  • Problem encountered: hardware-encoded stream with black edge and the latter with green edge:

    • Solution: The length and width of the input data are not aligned according to the stride, so the encoder cannot be compatible

  • Problem encountered: Hardware decoder cannot be created:

    • Solution 1: Set the wrong format for the decoder, such as color space, encoder name, whether to render to suface, etc.;
    • Solution 2: Exceed the maximum number of decoders supported by the SOC;

There are also some call system API Freeze, MTK chip special problem processing, acquisition frame rate is not stable, decoding failure, we also explore the corresponding solutions, and will not be described here.

Common Problems with screen sharing compatibility

Screen sharing is a special kind of video capture, which also has some problems related to equipment. There are common problems such as picture jam and black edge in data collection. The following is a simple analysis of the solutions to such problems.

  • Problem encountered: When the mobile phone screen is still, the frame rate is 0

    • Solution: Cache a frame of data and send it periodically

  • Problem encountered: Data collection black edge

    • Solution: If the resolution set does not match the screen resolution, the system will fill it with black data. Change the collection resolution.

At the end

Netease YUNxin AUDIO and video SDK is committed to realizing high-definition, stable, easy-to-use and low-latency services for every user. Through the introduction of this paper, netease Yunxin.com has a very perfect compatibility and adaptation scheme to make up for the lack of experience brought by the fragmentation of Android for users, and has also accumulated a lot of compatibility and adaptation experience to meet the various odd-shaped problems brought by different use scenarios and different device types and models.

The adaptation of each piece of equipment, is the heart of the artisan pour;

Every step of product grinding, is excellence pay.

The authors introduce

Iven, senior Android audio and video development engineer of netease, has been engaged in the development of Android audio and video SDK functions. During this period, HE was responsible for the research and development of G1 and G2 of netease Yunxin, as well as the audio and video adaptation based on Android intelligent hardware. Adaptation products include mobile phones, TVS, watches, set-top boxes, and smart audio.

More technical dry goods, welcome to pay attention to [netease Smart enterprise technology +] wechat public number