What is Web multimedia technology

In recent years, douyin, Kuaishou, B-station and other audio and video apps have entered thousands of households, enriching our entertainment life. The development and learning of audio and video technology has also set off a new wave of upsurge. Driven by business needs, the audio and video technology of Web products has also achieved great development and breakthrough.

Web front end

Interactive experience, front-end engineering, cross-end capabilities…

Digital pair media

Audio and video principles, packaging containers, encoding and decoding algorithms…

Web multimedia technology is mainly divided into the following directions:

  • On demand

It is mainly divided into short video, medium video and long video.

  • live

On-site audio and video data will be collected by the equipment and pushed to the server, which will process the data and then use its client to pull the live stream to watch the live broadcast.

  • The picture

We need to monitor and diagnose the distribution and downloaded network links of images. Different image formats are different in different browsers. For example, WebP files are not displayed in Safari, and then Web multimedia engineers need to decode the WebP files and display them in Safari.

  • Real-time communication

Provide high quality low latency audio and video data communication capabilities, mainly used in video conferencing, online education and some interactive entertainment scenarios.

  • Cloud game

The game runs on the server, reducing the client hardware requirements.

  • The video clip

Audio and video editing in the Web end, do not need to operate in their own editing software.

2. Basic knowledge of audio and video

Assume that 8bit represents a sub-pixel with resolution of 1280720, frame rate of 25fpx, duration of 60s, and uncompressed video size = 8BIT3128025 *60 = 3.9GB

After H264 compression, the video size is 11MB, and the compression ratio is 360:1

Space/intra-frame compression

We look at the picture above, the similarity of each small piece of blue sky high, show that there is a lot of redundancy between them, can use a small piece of approximation instead of the other three small pieces and only save a small piece of information, can save a lot of traffic, the algorithm to eliminate every small piece of information redundancy between the can greatly compressed video files.

Time/interframe compression

In the above two pictures, only the players, referees and fans are different. The changes are all the same, so there is redundancy. This situation can also be compressed.

Coding format development

Container format

We will get the of the encoded video through encoder is naked flow, such as the use of 264 encoder, got is 264 naked flow, if direct broadcast 264 naked flow, it must be played from the beginning to the end, we are unable to operate, so we need to wrap the audio and video naked flow, inside the fixed format of the file, These fixed files are called audio and video containers.

<video> and <audio> elements

  • Function: Decode + render
  • Support video formats: MP4, MP3

<video> and <audio> element methods

<body>
  <button onclick="playVid()">Play the video</button>
  <button onclick="pauseVid()">Stop video</button>
  <button onclick="loadVid()">Reload video</button>

  <video id="vs" src="demo.mp4"></video>

  <script>
    const myVideo = document.getElementById("vs");
    function playVid(){
      myVideo.play();
    }
    function pauseVid(){
      myVideo.pause();
    }
    function loadVid(){
      myVideo.load();
    }
  </script>
</body>
Copy the code

<video> and <audio> element attributes


<body>
  <button onclick="getVolume()">What is the volume</button>
  <button onclick="setVolume()">Set the video volume to 20%</button>
  <button onclick="getCurTime()">Gets the current playback time</button>
  <button onclick="setCurTime()">Set the playback time to 5 seconds</button>


  <video id="vs" src="demo.mp4"></video>

  <script>
    const myVideo = document.getElementById("vs");
    function getVolume(){
      alert(myVideo.volume);
    }
    function setVolume(){
      myVideo.volume = 0.2;
    }
    function getCurTime(){
      alert(myVideo.currentTime);
    }
    function setCurTime(){
      myVideo.currentTime = 5;
    }
  </script>
</body>
Copy the code

<video> and <audio> element events

<body>

  <video id="vs" src="demo.mp4"></video>

  <script>
    const myVideo = document.getElementById("vs");
    myVideo.addEventListener('canplay'.function(){
      alert('canplay');
    })
    myVideo.addEventListener('pause'.function(){
      alert('pause');
    })
    myVideo.addEventListener('play'.function(){
      alert('play');
    })
    myVideo.addEventListener('waiting'.function(){
      alert('waiting');
    })
  </script>
</body>
Copy the code

<video> and <audio> element defects

  • Video formats such as HLS and FLV are not supported

  • Requests and loads of video resources cannot be controlled by code

    • Segmented loading (save traffic)
    • Clarity seamless switching
    • Precise preloading

3. Native multimedia capabilities provided by browsers

MediaSouce

  • Extension of browser video playback capability, support video fragment loading (support for fMP4 fragments), instead of Flash player
  • Support playback MP4 (streaming playback), HLS, FLV and so on
  • It can realize segmental video loading, seamless resolution switching, accurate preloading and so on

Mp4 Video file structure

Fmp4 video file structure

Use MSE to realize MP4 streaming playback

Encrypted audio and video playback

Do anti-theft chain processing for audio and video

Adaptive bit rate

Automatically select high-definition playback based on network conditions

barrage

Collision – free and overlap – free projectile barrage needs a certain algorithm to achieve

How does the blocking portrait work here?

When the original video transcoding, are actually using computer vision algorithm, each frame of video detection, detect the portrait of the range, and then generates a mask file, when we began on the client, will be issued this mask file at the same time, players can then use this mask to achieve as anti blocking effect.

Soft solution: To play H265 format on the Web and prevent hijacking by domestic browser

Push page flow

At ordinary times, if we want to push the stream in the live broadcast, we should download and install the OBS software. Now, we have realized the function of opening a new page on the Web to achieve a push stream, mainly by using the Web RTC technology to achieve it.

Image decoding

Cloud games and features

Bytedance Web multimedia technology development

W3C new standard for audio and video technology

  • WebCodecs

A more efficient API that calls the browser’s audio and video codec capabilities

  • WebGPU

The ability to call this underlying GPU graphics computation on the Web side

  • WebVR, WebXR

Rendering capabilities for virtual reality, augmented reality devices

The last

⚽ this article leads you to learn Web multimedia technology, I believe you must harvest full ~ ⚾ if you are interested in this article welcome to like attention + collection, more wonderful knowledge is waiting for you! 😘 🏀GitHub blogs at github.com/Awu1227. 🏉 I have other columns, please read ~ 🏐 play the beauty of CSS 🎱Vue from giving up to getting started 🎳 simple JavaScript