Front-end streaming media playback from entry to pit

Recently, the player of huggies, a mainstream video website, seems not to be the URL of an MP4 player with the SRC attribute and the video tag we are used to. Basically, it is a pure front-end address starting with blob:. In view of this, I have done some research on the front-end streaming media, and wrote this article. By the way, I have added some interesting phenomena found in the process of research and write them together. I hope you have a basic understanding of streaming media after reading them. This article does not involve a lot of obscure and complex codes, and we only take what we can see on the video website as an example to explain 😂. OK, let’s go…

Video blob address basic introduction

When you go to YouTube/ YouTube/Netflix and check the SRC address on their video TAB, you’ll see that it’s almost always in the form of Blob URLS:

Address of Station B:

<video src="blob:https://www.bilibili.com/c21f2f47-52fb-46e6-a943-f02e708bceac"></video>
Copy the code

Tubing Address:

<video src="blob:https://www.youtube.com/250d7dd2-a387-4024-9325-ed4842c5c3d0"></video>
Copy the code

Introduced by Blob on MDN above, toblog:The leading address is generated as follows:

// blob:https://www.youtube.com/250d7dd2-a387-4024-9325-ed4842c5c3d0
const objectURL = URL.createObjectURL(object)
Copy the code

The object argument above can be: File,Blob, or MediaSource(not supported on IOS, however, We’ll look at how these sites are handled on IOS later. For streaming media, Object urls are created using MediaSource Extension API.

Video container format and video encoding

After understanding the basic creation method, I think we should review the basic knowledge of video first, which will help us understand the different streaming media formats that need to be adopted by different browsers. Simply speaking, the video container format is the suffix of the video files we usually see, such as.mp4,.mov,.mkv and so on. Basically, different forms of boxes, as to what is inside the different encoding, such as H.264 (AVC), H.265 (HEVC), VP9, AV1, ETC., H.264 should be the most used video encoding, Of course, in addition to video encoding, there are corresponding audio encoding such as AAC, which corresponds to the previous AVC video encoding, also under MPEG-4. More video formats and encoding can refer to this MDN address.

Insert a Mac where Safari cannot play YouTube 4K video 🌰(of course this issue has been fixed on macOS 11 Big Sur 😎) :

As you can see in the figure above, Safari 10.15 and before did not choose the resolution above 1080P, but in Google’s son Chrome, there is no problem. The problem is the choice of video encoding, YouTube 1080P and above such as 4K encoding mainly vp8/ VP9, AV1 encoding (WebM format). These encoders were not supported on Safari before, so there was no 4K option, but after Big Sur Apple supported WebM, so the new Version of Safari can play the above 4K video as well;

The flip side is that when you shoot a video on your iPhone (h.265 by default) and play it on a Mac, Safari works fine, but Chrome doesn’t, because Chrome doesn’t support H.265 yet. The above two phenomena are caused by different video encoding in different browsers.

Common streaming media protocols

When we use local players play a local several G video file is really do not have what problem, but as more and more people choose online video watching films, we can’t wait for the browser to download after a few G video to start playing again, this speed is slower than 🐌 simply, all came streaming protocol, also is the so-called edge below, This greatly improves the online viewing experience, but of course the content itself is still based on these encoding formats no matter what your streaming protocol changes.

Here are some common streaming protocols:

The FLV container format that came along with Flash seems to be alive and well in browsers, thanks to the open source flv.js. Even if there is no Flash in the browser, we can still play the video content in FLV container format normally through FLv. js. FLV is still used a lot in the live broadcast of domestic websites, and the main reason is that the low latency of HTTP-FLV is still very good. HLS(Http Live Streming), apple’s protocol, and apple said, well, since we’re going to go with Steve Jobs and stop supporting Flash, we’re going to have to build a new wheel for streaming, and we don’t want to build a lower wheel, we’re going to do it over Http, because Http is pretty mature, And the compatibility is not too good, then came the HLS streaming media protocol, based on the usage of iPhone, HLS in streaming media share is quite high:

Mpeg-dash, the MPEG equivalent of HLS, thought apple, why do you always like to build your own wheels and not open source? Well, in order not to be monopolized by you, we need to create an open source streaming protocol similar to HLS, and then MPEG-Dash came out. From the above figure, it seems that the share is not very high, but after all, open source, all the video manufacturers have already used B station in China, foreign tubing, Netflix, etc., the future can be expected.

As shown in the table above, HLS and MPEG-DASH support adaptive bitstreaming. To put it simply, for the two streaming media protocols, a source video file will be cut into different small video files according to the different resolutions that need to be supported. Generally, the video files are between several hundred K and a few megabytes. In this way, when the network speed changes, the corresponding slice file can be quickly cut to the resolution that meets the current network speed, which can ensure the seamless switching of the playback smoothness and greatly improve the user experience. You can see the following figure to feel it:

A live link

HLS, MPEG-Dash is mainly used for on-demand, you watch up main video in B station, go to YouTube to watch a TV drama, these are on-demand, that is, all videos are transcoded in advance to generate different resolutions, and select HLS or DASH to generate corresponding slice files; So how do in the case of live, actually when the client viewing basic about the same, the main difference is that the flow end, mainly user upload video on demand case, then cloud video transcoding operation, such as the live quite so side edge recording process, video source can be a video camera, computer camera, Of course, the easiest way to live broadcast is to directly hold a live broadcast with a mobile app, which is basically similar to a video conference.

Here is a picture to illustrate the whole process of live broadcasting: First of all, in the stream side, that is, the host side, the host may open the camera through the software on the computer (commonly used, such as open source OBS), set up the stream connection and protocol, now the most commonly used is RTMP, which is also the product of Flash at that time, without Flash, RTMP is basically unusable in the browser side. But in pushing the flow end is widely used, can back to look at the figure of the above share, RTMP there is a third major share, when the host of the video information by pushing flow agreement was pushed to the platform after the cloud server, will be conducted in real-time transcoding format of the operation, after the transfer real-time pushed to the CDN, and then use the CDN speed, regional, Users can watch live broadcasts with less delay.

Flash Video(.flv file) + FLVJS

As mentioned above, FLV format videos are still widely used at least in China. If you look at the network requests of the live broadcast page, there are many requests ending in. FLV, including the live broadcast of STATION B (not all of course), which can be seen in the live broadcast page of the anchor:

Livestreaming in IQiyi is also FLV. Even some livestreaming in Toutiao app, if you go to grab a bag, you will find a lot of FLV. All these livestreaming with FLV are basically used in the browserflv.jsSo below we might as well take a look at the general workflow of FLv.js, the first figure (from flv.js official Github) :

When we enter the live broadcast page, flVPlayer will first request the address of FLV. First, it will select an appropriate loader to load the video data according to the current browser through IO Loaders. Let’s see the live broadcast address in the figure above (fetch type in network is). FetchStreamLoader should be used, because FETCH supports stream and can obtain data piece by piece. Every time it gets a complete piece of data, it will be processed by FlvDemuxer, that is, unpack the FLV container format and get the original AVC/AAC audio and video data inside. It is then repackaged into a legitimate MP4 format via MP4Remuxer, passed up to Transmuxer, fed to the video TAB via the browser’s own MediaSource, and played in FLV.

So if the browser doesn’t support MediaSource, flv.js is useless. As stated at the beginning of this article, MediaSource is not supported on iOS (iPad supports 🤪, iPhone only), so flv.js is not available in safari on iOS. So if we open bilibili.com in Safari on ios, we will find that it uses the MP4 address directly, without using the BLOB connection, like the following image:

Take a look at the situation of IQiyi. Since iQiyi’s playback is mainly based on HLS, the HLS protocol can also be perfectly used on ios:

You can see the address of. M3u8 directly used by IQiyi on H5 (of course, whether directly MP4, or the original HLS, their core logic on H5 is to make you download or jump to the app 🤪), M3U8 is the entry file of HLS protocol, it is a text file. This file system allows you to retrieve individual slices of the video. Thanks to apple’s protocol, all m3U8 text files can be thrown directly into the SRC tag of the video. If you’re interested, you can drop the M3U8 address directly into the Safari address bar on your Mac and it will start playing, but in Chrome it will tell you to download the file because native support is not available. Unless you have the HLS plugin installed.

Generate HLS, MPEG-DASH compliant tools and front-end players

Pull so much, then how we can put a movie files, generate support these streaming protocols slice file, already has a variety of tools available, of course, as well as our front end can be used to play the stream media file library, here is a simple list, how to do is look at the document 😇 thing

Bonus: Download a Bilibili video manually

Finally, let’s go to 🖐. We already know that the streaming protocol used by STATION B is MPEG-Dash, and Dash normally uses an XML file ending in.mpd (similar to the M3U8 file used by HLS). However, we found that there was no MPD request in the browser’s web request. You can open a video page, right-click the source code, and search for Dash. You’ll find JS objects like the one below, with dash fields containing links to the video, supported resolutions, and more:

Above we can find a separate video and audio field, also he took apart the video and audio data, will use a separate url to load, the advantage is av independent each other, when we switch the low resolution, we can still request high quality of audio data, avoid because of lower resolution, sound quality is also decreased.

Let’s go straight to the Network request in the browser and filter it with M4S. M4s is the dash slicing file suffix, but the content is actually part of an MP4. In theory, you just slice all the M4S into an MP4 and it’s ready to play without any extra processing. M4s and XXX-30280. m4s. You can guess one path is video and one path is audio

If we look at the following figure, we can see that for the video file, it is not cut into different file names, but uses the Range Header in the Header to obtain different parts of the file. We can know the total size of the file through the content-range in response, and the number after /. The Range in the curl request is 0 to 181545880, and the end of the curl request needs to be subtracted by 1, because it starts from 0.

And then you can put that curl on the command line, or you can put it on any tool that sends a request, but in Insomnic’s case, you can paste that curl in, and it automatically converts, and then you can adjust the Range, and send the request, Save the original response data as an MP4 file (do the same for audio) :

When all the audio and video files are played, you can actually play the two files separately, but one only has video information and the other only has audio information. Some players can directly import audio information (such as IINA), so you don’t need to merge them. Of course, for completeness, We still use FFMPEG to combine audio and video files into a complete MP4 file:

ffmpeg -i video.mp4 -i audio.mp4 -c:v copy -c:a copy output.mp4
Copy the code

In this way, the final output.mp4 can play directly 🎉, so we have completed a manual download of streaming video, give yourself 👏!!

conclusion

Above is the introduction to the basic of front-end streaming, we through some practical examples, and in the different video web site to see some of the different processing, the front-end video introduction to the understanding of relevant knowledge, hoped everybody through the introduction to this article can do, and even into the pit of the front end of streaming media, interested students can 💪 to do in-depth research.