1. Set media constraints

  • Obtain audio and Video permissions

  • Add videoSource audioSource

  • Add audioTrack and videoTrack

  • Set the captureSession

  • Set the video format device FPS

Mobile phone you’d better set down adaptation AVCaptureSessionPreset640x480 15 FPS setting, the general Settings is AVCaptureSessionPreset1280x720 even higher.

The pit point here is the value, which will affect the resolution and sharpness of the video, including sharing the screen, stretching, and zooming all will also stretch. Our own processing is the size of the resolution rate and which value to set

Now, notice that this is important here, because this is probably what’s causing your black screen

How the screen fills

2. Switch between front and rear cameras

  • CGAffineTransformMakeScale (1.0, 1.0) set up a camera before and after switching

  • AVCaptureDevicePositionFront  | AVCaptureDevicePositionBack

3. Switch between the receiver (earpiece) and the speaker

4. Disable all remote audio

5. Turn off an audio

6. Encoding and decoding

  • Hardcoding benefits: very low CPU consumption, very high decoding efficiency Disadvantages: use the private interface VideoToolBox
  • Soft code: disadvantages: CPU consumption is too large, such as iphone4-6 basic mobile phone hot to explode sometimes will be stuck

Hard-coded: do not use the CPU coding, the use of graphics hardware acceleration (GPU), special DSP, FPGA and ASIC chip hardware encoding (IOS AudioToolbox | Video ToolBox)

Soft coding: Coding using the CPU. Encoding framework FFMPEG + X264

Because the mobile phone is hot, hard coding and decoding have to be carried out. If the technology is more powerful, H265 can also be the best. Its advantages are as follows:

  • High compression ratio, two times higher than JPEG at the same image quality
  • Can increase such as picture depth information, transparent channel and other auxiliary pictures.
  • Support to store multiple pictures, similar to albums and collections. (Achieve the effect of multiple exposures)
  • Support multiple images to achieve GIF and livePhoto animation effects.
  • There is no maximum pixel limit similar to JPEG
  • Support for transparent pixels
  • Block loading mechanism
  • Support for thumbnails

H264Decode

H264Encode

Hard code is recommended otherwise the phone will be horribly hot haha.

7 PeerConnection:

offer Anwer ICE  DTLS STUN TURN 

DTLS:

8 Pull and push flow (operation UI should be operated on the main thread)

9 determine whether there is a camera in the fetch or you will crash the camera may not be the other side such as PC

  • Shared screen pull stream and push stream shared screen background run remote click webpage and other APP influence

  • Video recording

  • The socket connection was reconnected

  • The socket keep alive

  • Sharing does not release the problem

  • Sharing remote screen stretch iPad iOS PC adaptation

  • VOIP PushKit 

  • UserNotifications

Echo cancellation

The library used is mainly AEC (C, C ++) to be mixed

The AEC algorithm of WebRTC mainly includes the following important modules:

  1. Echo delay estimation module

  2. NLMS algorithm

  3. NLP nonlinear filtering

  4. CNG(comfort nosie generation)