Make your web page speak – audioContext API

The original link

Browsers can use CSS3, SVG, and Canvas for graphics and painting, making it a simple Version of Photoshop. Recently I studied audioContext and found that the water depth is as deep as they are! Although I was born in communication major, I also learned speech signal processing, but I still know little about the basic principles, I feel sorry for the training of my Alma mater…

preface

There are so many apis designed by AudioContext, many of which are still experimental and not fully supported by all browsers, that this article is not going to cover everything, but just a few commonly used apis. First know, then know why, but also can be regarded as a learning method. (actually a lot of I really don’t understand 😂)

Introduction to the

First come a simple chestnut 🌰↓, feel what it can do —

demo1


Note that this is not an audio file with the audio tag, but with js control browser sound. You can adjust the volume, frequency and other parameters to try again.

How do you do that? Take a look at the graph belowThis is the flow chart necessary to implement the sound. First you need an audio context, then you need an input of audio data, then you need a handler, then you need an output of audio, then you need a connection

The input of audio data can be custom data, audio of the audio element on the page, audio or video input from the user’s device, remote audio files, and so on.

The handler can be a profiler, processor, etc., or it may not be anything, that is, connecting the sound source directly to the speaker.

The audio output is typically the speaker of the user’s device.

Create an AudioContext instance, which is the environment in which the audio processor runs

let audioCtx = new AudioContext(); 
Copy the code

Create an oscillator, which is the source of the sound

let oscillator = audioCtx.createOscillator();
Copy the code

Create a gain node (volume node) to adjust the volume changes

let gainNode = audioCtx.createGain();
Copy the code

Set the volume and oscillator parameters

GainNode. Gain. Value = 0.5; // Volume 0~1 oscillator. Type = 'sine'; / / oscillator output sine wave oscillator. The frequency. The value = 200; // The oscillation frequency is 200HzCopy the code

For various connections, refer to the flow chart above

oscillator.connect(gainNode); // Source oscillator connection volume gainNode.connect(Audioctx.destination); // The volume is connected to the speakerCopy the code

Began to speak

oscillator.start();
Copy the code

The end of the voice

oscillator.stop(audioCtx.currentTime + FADING_TIME); // Now FADING_TIME finishes after seconds. If there is no FADING_TIME, it finishes immediatelyCopy the code

In the following demo, the overall flow is roughly the same: Create the sound source → intermediate processor → speaker. Of course, if you don’t need to output sound, you don’t need to connect to audioctx.destination.

In addition, you can also set a voice change curve, the API provides two methods: linearRampToValueAtTime and exponentialRampToValueAtTime, linear changes and index respectively.

Complete code:

Const FADING_TIME = 0.5; Export default {data () {return {started: false, queryParams: {gain: 0.5, gainChangeType: 'linearRampToValueAtTime', frequency: 196, waveform: 'sine' } } }, methods: { init () { this.oscillator = this.audioCtx.createOscillator(); this.gainNode = this.audioCtx.createGain(); }, onSet () { this.oscillator.type = this.queryParams.waveform; this.gainNode.gain.value = this.queryParams.gain; this.oscillator.frequency.value = this.queryParams.frequency; this.oscillator.connect(this.gainNode); this.gainNode.connect(this.audioCtx.destination); }, onStart () { this.init(); this.onSet(); this.oscillator.start(); this.started = true; }, onStop () {// Change to 0.001 in 0.5 seconds, And stop this. GainNode. Gain [this. QueryParams. GainChangeType] (0.001, this. AudioCtx. CurrentTime + FADING_TIME); this.oscillator.stop(this.audioCtx.currentTime + FADING_TIME); this.started = false; } }, mounted () { if (! AudioContext && ! WebkitAudioContext) {alert(' Your browser does not support audioContext! '); return; } this.audioCtx = new (AudioContext || webkitAudioContext)(); this.init(); }}Copy the code

demo2

If you can control the frequency of sounds, so can simple notes like “Do re mi”. Check the frequency table of piano keys, select a frequency, can be made into a simple piano. See chestnut below 🌰↓

The code is just set to a fixed number of frequencies compared to Demo1.

Const FADING_TIME = 0.5; export default { data () { return { FREQUENCY_LIST: [261.626, 277.183, 293.665, 311.127, 329.628, 349.228, 369.994, 391.995, 415.305, 440.000, 466.164, 493.883] QueryParams: {Gain: 0.5, frequency: 196, Waveform: 'sine'}}}, methods: { init () { this.oscillator = this.audioCtx.createOscillator(); this.gainNode = this.audioCtx.createGain(); this.oscillator.type = this.queryParams.waveform; this.gainNode.gain.value = this.queryParams.gain; this.oscillator.connect(this.gainNode); this.gainNode.connect(this.audioCtx.destination); }, onPlay (item) { this.init() this.oscillator.frequency.value = item; this.oscillator.start(this.audioCtx.currentTime); this.oscillator.stop(this.audioCtx.currentTime + FADING_TIME); } }, mounted () { if (! AudioContext && ! WebkitAudioContext) {alert(' Your browser does not support audioContext! '); return; } this.audioCtx = new (AudioContext || webkitAudioContext)(); this.init(); }}Copy the code

As mentioned above, the input of audio data can be custom data, audio of the audio element on the page, audio or video input from the user’s device, remote audio files, and so on. Here are a few demos to show different situations of sound sources and the frequency variation of audio on canvas.

Pay attention to

NotSupportedError: Failed to construct ‘AudioContext’: The number of Hardware Contexts provided (6) is greater than or equal to The maximum bound (6).”, The reason is that some versions limit the number of audioContext instances that can exist simultaneously on a web page (limit 6?). Because at least one instance will be created in each demo, errors may be reported if the limit is exceeded.

Chromium (58+) I use will report errors, Chrome (68+) does not, specific support has not been studied. If you encounter an error, try upgrading your browser.

In addition to the above methods, there are two more important methods: createAnalyser and createScriptProcessor.

The createAnalyser() method creates an AnalyserNode that can be used to capture audio time and frequency data, as well as data visualization.

The createScriptProcessor() method creates a ScriptProcessorNode to process audio through JavaScript.

The following code snippets will be used in the following demos

InitAnalyser () {/ / create a parser enclosing analyser = this. AudioCtx. CreateAnalyser (); // The fast Fourier transform parameter this.analyser.fftsize = 256; This. / / bufferArray length bufferLength = this. Analyser. FrequencyBinCount; DataArray = new Uint8Array(this.bufferLength); }, initScriptProcessor () {// create a processor, Parameters are respectively buffer size, enclosing the number of input channels, output the number of channels scriptProcessor = this. AudioCtx. CreateScriptProcessor (2048, 1, 1); // The parser connects to the processor, and the processor connects to the speaker this.analyser.connect(this.scriptprocessor); this.scriptProcessor.connect(this.audioCtx.destination); }Copy the code

A scriptProcessor can bind an audioProcess to an event that is fired when the scriptProcessor connects to the analyser and has audio data.

bindDrawEvent () {
    this.scriptProcessor.onaudioprocess = this.draw;
}
Copy the code

Canvas drawing code

draw () { let cWidth = this.canvas.width, cHeight = this.canvas.height, barWidth = parseInt(.5 * cWidth / this.bufferLength), barHeight, x = 0; this.canvasCtx.clearRect(0, 0, cWidth, cHeight); / / analyzer for audio data "slices" enclosing analyser. GetByteFrequencyData (enclosing dataArray); For (var I = 0; i < this.bufferLength; I ++) {barHeight = parseInt(0.4 * this. DataArray [I]); this.canvasCtx.fillRect(x, cHeight - barHeight, barWidth, barHeight); x += barWidth + 3; }}Copy the code

🌰 comes from the audio element

Click Play:

Core code:

this.audioElement = document.querySelector('.audio');
this.audioElement.crossOrigin="anonymous";
this.audioSource = this.audioCtx.createMediaElementSource(this.audioElement);
this.audioSource.connect(this.analyser);
this.audioSource.connect(this.gainNode);

this.bindDrawEvent();
Copy the code

🌰 from the user device

Core code:

navigator.mediaDevices.getUserMedia({audio: true}).then(stream => { this.audioSource = this.audioCtx.createMediaStreamSource(stream); this.audioSource.connect(this.analyser); this.audioSource.connect(this.gainNode); this.bindDrawEvent(); }, error => {alert(' error, please make sure browser is allowed to get audio rights '); });Copy the code

🌰 from remote audio

  • Remote music

Core code:

initSource () { this.audioSource = this.audioCtx.createBufferSource(); this.audioSource.connect(this.analyser); this.audioSource.connect(this.gainNode); Replayed}, onPlay () {/ / need to create a buffer enclosing audioSource = this. AudioCtx. CreateBufferSource (); this.audioSource.connect(this.analyser); this.audioSource.connect(this.audioCtx.destination); this.audioSource.buffer = this.buffer; this.audioSource.loop = true; this.audioSource.start(0, this.playResume); this.playStart = new Date().getTime() - this.playResume * 1000; this.playing = true; }, onPause () { this.playResume = new Date().getTime(); this.playResume -= this.playStart; this.playResume /= 1000; this.playResume %= this.audioSource.buffer.duration; this.audioSource.stop(); this.playing = false; }, play (url, index) { if(this.index == index) return; this.index = index; this.requestSong(url); }, requestSong (url) { let request = new XMLHttpRequest(); request.open("GET", url, true); request.responseType = "arraybuffer"; this.loading = true; request.onload = () => { this.audioCtx.decodeAudioData(request.response, buffer => { this.loading = false; this.playing = true; this.buffer = buffer; this.playSound(buffer); this.bindDrawEvent(); }); }; request.send(); }, playSound (buffer) { this.audioSource.buffer = buffer; this.audioSource.loop = true; this.audioSource.start(0); this.playStart = new Date().getTime(); }Copy the code

🌰 from a local audio file

Core code:

onProcessFile (e) { this.file = e.target.files[0]; let stream = URL.createObjectURL(this.file), audio = new Audio(); audio.src = stream; this.init(); audio.oncanplay = this.initAudioSource(audio); this.showCancel = true; }, onCancel () { this.audioCtx.state ! = 'closed' && this.audioCtx.close(); URL.revokeObjectURL(this.file); this.fileInput.value = ''; this.showCancel = false; }, initAudioSource (audio) { this.fileInput = this.$refs['fileInput']; this.audioSource = this.audioCtx.createMediaElementSource(audio); this.audioSource.connect(this.analyser); this.audioSource.connect(this.gainNode); audio.play(); this.bindDrawEvent(); }Copy the code

use

What can it be used for?

  • Game sound?
  • A music website?
  • H5 promotional page?
  • Embellish your blog with sound?
  • , etc.

You can use it wherever you want.

The resources

  • Web Audio API
  • Add sound to Web JS interactions using the HTML5 Web Audio API
  • Introduction to Web Audio API
  • The piano keys correspond to frequencies
  • Visualizations with Web Audio API