General process of audio and video processing Data collection → data coding → data transmission (streaming media server) → decoding data → playing display 1. Data collection: Video and audio data are collected by camera and pickup, and the original data is obtained at this time. Technologies or protocols are involved: camera: CCD, CMOS pickup: acoustic electric conversion device (microphone), audio amplifier circuit.
2. Data encoding: use relevant hardware or software to encode and process audio and video raw data (digitalization) (such as audio and video mixing, packaging and packaging, etc.) to obtain available audio and video data involving technology or protocol: Encoding method: CBR, VBR encoding format video: H.265, H.264, MPEG-4, packaging containers TS, MKV, AVI, MP4 and other audio: G.711μ, AAC, Opus, packaging MP3, OGG, AAC and so on
3. Data transmission: the audio and video data after coding are transmitted. In the early stage, audio and video data are transmitted through coaxial cables and other cables. Control signaling including RTP and RTCP, RTSP, RTMP, HTTP, and HTTP Live Streaming (HLS) : SIP, SDP, and SNMP.
4. Decoding data: use relevant hardware or software to decode the received encoded audio and video data, and get directly displayed images/sounds involved in technology or protocol: generally, the corresponding encoder will have corresponding decoders, and there are also some third-party decoding plug-ins. The technology or protocol involved in displaying the corresponding image or sound in the monitor (TV, monitor screen, etc.) or speaker (headset, speaker, etc.) is the monitor, speaker, 3D glasses, etc.
1. First of all, collection refers to the collection of images and sounds. In a nutshell, it can connect the network video live broadcasting system with the anchor camera, so that the software can obtain audio and video information from the outside world. Then comes the pre-processing stage: this stage is also commonly known as beauty stage, which requires us to add beauty, special effects and cute face effects into the network video live broadcast system, and this algorithm needs to involve GPU programming. The difficulty in this stage is not beauty, because beauty is usually solved by SDK, but the proportion of GPU.
2. The hardware configuration of mobile phones on the market is getting higher and higher at present, but power consumption is still a big problem for some old models. Too high GPU occupancy will lead to hot phone, camera frame loss and even software flash back.
3, the next step is coding, coding is to compress the data in order to better decoding, usually we will choose to use hard-coded to balance efficiency between the coding speed and coding quality than, but soft plait is still higher image quality, the choice of users are willing to use soft coding way, so the original system design, We should design both soft and hard coding methods simultaneously.
4. The last operation of the streaming end is to push the video to the server, so as to facilitate the distribution and streaming of live video.