@ (av) [Audio | Video | MSE]
Audio and video With the development of the Internet, the demand for audio and video is more and more, but audio and video without confusion is playing or decoding, packaging performance requirements are relatively high, then at the present stage of the front-end audio and video field can do what?
[TOC]
Audio or video playback
html5 audio
When it comes to audio and video playback, THE first thing I think of is HTMLMediaElement. Here’s an example:
<audio controls autoplay loop="true" preload="auto" src="audio.mp3"></audio>
Copy the codecontrolsSpecify that the browser renders to HTML5audio.autoplayProperty tells the browser to play automatically when it has finished loading.loopProperty plays in loop.preloadWhen the audio element is rendered, the audio file is loaded.- Mobile browsers do not support it
autoplayandpreloadProperty, that is, audio files are not automatically loaded, only triggered by events such astouch,clickEvents and so on trigger loading and then play. - There are also some changes to the volume, the completion of an audio playback event, etc. Read HTMLMediaElement.
- Of course, if your web page is running in a WebView, you can ask the client to set some properties for preloading and auto-playing.
AudioContext
It can use HTML 5 audio playback of audio, but as you can see a lot of problems, at the same time I ‘can’t to good control of audio playback, such as access to the audio in binary data from the network, sometimes I want to order playback of audio, also is not to use audio element, processing is simple elegance. Here’s an example:
function queuePlayAudio(sounds) {
let index = 0;
function recursivePlay(sounds, index) {
if(sounds.length == index) return;
sounds[index].play();
sounds[index].onended = recursivePlay.bind(this, sounds, ++index); }}Copy the codeListen for onended events on the Audio element, played sequentially.
For better control of audio playback, I need an AudioContext.
The AudioContext interface represents an audio processing diagram connected by audio modules, each of which corresponds to an AudioNode. The AudioContext can control the creation of the nodes it contains, as well as the execution of audio processing and decoding operations. Create the AudioContext object before you do anything, because that’s where everything happens.
It may be difficult to understand, but in simple terms, an AudioContext is like a factory. For the playback of an audio, from the audio source to the sound control, to the implementation of the linked playback hardware, all the modules are responsible for processing, and the process is controlled through CONNECT.
Now I can control audio playback, such as from the network. Using AJAX to obtain arrayBuffer type data, through decoding, and then the audio binary data to the BufferSourceNode created by AudioContext, finally through the link destination module to achieve audio playback.
export default class PlaySoundWithAudioContext {
constructor() {
if(PlaySoundWithAudioContext.isSupportAudioContext()) {
this.duration = 0;
this.currentTime = 0;
this.nextTime = 0;
this.pending = [];
this.mutex = false;
this.audioContext = new (window.AudioContext || window.webkitAudioContext)(); }}static isSupportAudioContext() {
return window.AudioContext || window.webkitAudioContext;
}
play(buffer) {
var source = this.audioContext.createBufferSource();
source.buffer = buffer;
source.connect(this.audioContext.destination);
source.start(this.nextTime);
this.nextTime += source.buffer.duration;
}
addChunks(buffer) {
this.pending.push(buffer);
let customer = (a)= > {
if(!this.pending.length) return;
let buffer = this.pending.shift();
this.audioContext.decodeAudioData(buffer, buffer => {
this.play(buffer);
console.log(buffer)
if(this.pending.length) {
customer()
}
}, (err) => {
console.log('decode audio data error', err);
});
}
if(!this.mutex) {
this.mutex = true;
customer()
}
}
clearAll() {
this.duration = 0;
this.currentTime = 0;
this.nextTime = 0; }}Copy the codeThe AJAX call
function xhr() {
var XHR = new XMLHttpRequest();
XHR.open('GET'.'//example.com/audio.mp3');
XHR.responseType = 'arraybuffer';
XHR.onreadystatechange = function(e) {
if(XHR.readyState == 4) {
if(XHR.status == 200) {
playSoundWithAudioContext.addChunks(XHR.response);
}
}
}
XHR.send();
}
Copy the codeUsing Ajax to play is fine for small audio files, but for large audio files, it is not practical to wait until the download is complete to play. Can you download and play at the same time? The fetch implementation is used to load the stream.
fetch(url).then((res) = > {
if(res.ok && (res.status >= 200 && res.status <= 299)) {
readData(res.body.getReader())
} else {
that.postMessage({type: constants.LOAD_ERROR})
}
})
function readData(reader) {
reader.read().then((result) = > {
if(result.done) {
return;
}
console.log(result); playSoundWithAudioContext.addChunks(result.value.buffer); })}Copy the codeTo put it simply, fetch’s response returns a readableStream interface, from which the stream is read and continuously fed to audioContext for playback. The test finds that the mobile end cannot play smoothly, but the PC end browser can.
PCM audio
Implement audioContext play, I need of decoding, using decodeAudioDataapi decoding, I know of, general audio will be compressed into mp3, aac encoding format, I need first of decoding as PCM data to play, the PCM? I knew that sound was generated by the vibration of objects, but such sound waves could not be stored and calculated by a computer. I needed some way to characterize the sound, so I had data in PCM format, which indicated the frequency of the sound picked up by the microphone, the number of bits picked up and the number of channels, stereo or mono.
Media Source Extensions
Media Source Extensions can dynamically create streams for Audio and Video to play. In simple terms, they can be used to control playback, such as seek, and can be used to convert certain formats to play. Not all formats are supported.
By appending data to SourceBuffer, the MSE stores the data into the buffer and decodes it for playback. Here’s a simple example of using MSE to play audio:
export default class PlaySoundWithMSE{
constructor(audio) {
this.audio = audio;
if(PlaySoundWithMSE.isSupportMSE()) {
this.pendingBuffer = [];
this._mediaSource = new MediaSource();
this.audio.src = URL.createObjectURL(this._mediaSource);
this._mediaSource.addEventListener('sourceopen', () = > {this.sourcebuffer = this._mediaSource.addSourceBuffer('audio/mpeg');
this.sourcebuffer.addEventListener('updateend'.this.handleSourceBufferUpdateEnd.bind(this));
})
}
}
addBuffer(buffer) {
this.pendingBuffer.push(buffer);
}
handleSourceBufferUpdateEnd() {
if(this.pendingBuffer.length) {
this.sourcebuffer.appendBuffer(this.pendingBuffer.shift());
} else {
this._mediaSource.endOfStream(); }}static isSupportMSE() {
return!!!!!window.MediaSource; }}Copy the codeHTML 5 player
Speaking of HTML5 players, you probably know Bilibili’s Flv.js, which relies on Media Source Extensions to package flV-encoded video into MP4 and play it back.
As can be seen from the flow chart, IOController implements the loading of video streams. Here, the stream capability of FETCH is supported, WebSocket, etc. The video streams to be obtained are in FLV format and encapsulated in MP4 format. Finally, data in MP4 format is fed to the MSE through appendBuffer for playback.
In the future
This is all about video playback. As you can see, even though there are many limitations to playback, MSE browser support is still limited. Can the front-end do anything in the performance demanding areas of video encoding and decoding? There are a number of reasons why front-end performance is not high. In a sandbox environment like a browser, and js as a dynamic language, performance is not high. So one of the big guys came up with the idea of compiling c++ into js and improving performance. I would like to consider compiling some video coding libraries to JS to run to improve performance, including FFmpeg, can consider compiling it to ASM, and then codec the video.
Write in the last
I can see that the front end is treading on thin ice in terms of audio and video processing for a number of reasons, but with browser support for video playback, there is still room for improvement.
Recruit wise men
Toutiao recruits a large number of front-end engineers in Beijing, Shenzhen, Shanghai, Xiamen and other cities for a long time. Please send your resume to [email protected]/[email protected]