What is Web multimedia technology?

The following excerpt is the official definition of W3School: multimedia is everything we can see and hear: text, books, pictures, music, sounds, CDS, videos, DVDS, files, movies, etc. Multimedia exists in many ways, and many elements are found embedded in web pages on the Internet.

  • On demand
  • live
  • The picture
  • Real-time communication
  • Cloud game
  • Video editing

Basic knowledge of audio and video

Assume that 8bit represents a sub-pixel with resolution of 1280720, frequency of 25fps and duration of 60s. The uncompressed video size =8bit312807202560=3.9GB, and the video size after H264 compression is 11MB.

Coding format development

  • 1999: mpeg-4
  • 2003: h. 264
  • 2010: VP8
  • 2013: h. 265, VP9
  • 2015: VP10
  • 2018: AV1

Native multimedia capabilities provided by the browser

Video and Audio element methods

  • Play () : Starts playing audio/video
  • Paused () : pauses the current play
  • Load () : Reloads the current element
  • CanPlayType () : checks whether the browser can play the specified type
  • AddTextTrack () : Adds a new text track to the target file
  • Autoplay () : Sets or returns whether audio is played immediately after loading
  • Controls () : Sets or returns whether to display a control

Defects in the video and audio elements

  • Video formats such as HLS and FLV are not supported
  • Requests and loads of video resources cannot be controlled by code

MediaSource

  • Expand video playback capabilities of the browser and support video loading in segments instead of Flash player
  • Supports playing MP4, HLS, FLV and so on.
  • It can realize segmental loading, seamless resolution switching, adaptive bit rate, accurate preloading and so on.

The development and breakthrough of Web multimedia technology

  • Encrypted audio and video playback
  • Adaptive bit rate
  • Barrage: no collision, interactive barrage, anti – blocking portrait.
  • Push page flow
  • Web decoding
  • Cloud game principle and characteristics: no installation, the first hardware requirements, low latency, strong compatibility requirements.

Summary and Prospect

Bytedance Web Multimedia Technology Development:

  • Independently developed video player
  • Xgplayer open source
  • flv
  • hls
  • webrtc
  • Video editing
  • web heif
  • Encrypt video
  • H265 with anti-hijack
  • Interactive video
  • Cloud game