H5 Recording Practice Summary (Preact)

directory

• To obtainPCMdata
• To deal withPCMdata
  • Float32 转 Int16
  • ArrayBuffer 转 Base64
  • PCMFile playback
  • resampling
  • PCMturnMP3
  • PCMturnWAV
  • Short time energy calculation
  • Web Worker optimizes performance
• Audio Storage (IndexedDB)
• WebViewopenWebRTC

To obtainPCMdata

You are advised to open DEMO in the browser or wechat. You need to configure the WebRTC in the Webview to authorize DEMO

Github.com/deepkolos/p…

Example code:

const mediaStream = await window.navigator.mediaDevices.getUserMedia({
    audio: {
		// sampleRate: 44100, // If the sampling rate is not valid, manually resampling is required
        channelCount: 1./ / track
        // echoCancellation: true,
        // noiseSuppression: true, // real noiseSuppression is good}})const audioContext = new window.AudioContext()
const inputSampleRate = audioContext.sampleRate
const mediaNode = audioContext.createMediaStreamSource(mediaStream)

if(! audioContext.createScriptProcessor) { audioContext.createScriptProcessor = audioContext.createJavaScriptNode }// Create a jsNode
const jsNode = audioContext.createScriptProcessor(4096.1.1)
jsNode.connect(audioContext.destination)
jsNode.onaudioprocess = (e) = > {
    // e.inputBuffer.getChannelData(0) (left)
    / / dual channel through e.i nputBuffer. GetChannelData (1) obtain (right)
}
mediaNode.connect(jsNode)
Copy the code

The brief process is as follows:

Start =>start: start. GetUserMedia => Operation: Obtain MediaStream. AudioContext => Operation: Create audioContext. Create scriptNode and associate AudioContext onAudioProcess =>operation: Set onAudioProcess and process data end=>end: The end of the start - > getUserMedia - > audioContext - > scriptNode - > onaudioprocess - > endCopy the code

To stop recording, simply uninstall the node where the audioContext is hanging, and then merge each frame stored to produce PCM data

jsNode.disconnect()
mediaNode.disconnect()
jsNode.onaudioprocess = null
Copy the code

PCMThe data processing

The PCM data obtained via WebRTC is in Float32 format and a merge channel is required for dual-channel recording

const leftDataList = [];
const rightDataList = [];
function onAudioProcess(event) {
  // One frame audio PCM data
  let audioBuffer = event.inputBuffer;
  leftDataList.push(audioBuffer.getChannelData(0).slice(0));
  rightDataList.push(audioBuffer.getChannelData(1).slice(0));
}

// Cross-merge the left and right channels
function interleaveLeftAndRight(left, right) {
  let totalLength = left.length + right.length;
  let data = new Float32Array(totalLength);
  for (let i = 0; i < left.length; i++) {
    let k = i * 2;
    data[k] = left[i];
    data[k + 1] = right[i];
  }
  return data;
}
Copy the code

Float32 转 Int16

const float32 = new Float32Array(1)
const int16 = Int16Array.from(
	float32.map(x= > (x > 0 ? x * 0x7fff : x * 0x8000)))Copy the code

arrayBuffer 转 Base64

Note: In the browser there is a btoa() function that can also be converted to Base64, but the input arguments must be strings. Passing buffer arguments will be toString() followed by Base64. Using ffplay to play the deserialized Base64 will be harsh

This is done using base64-arrayBuffer

import { encode } from 'base64-arraybuffer'

const float32 = new Float32Array(1)
const int16 = Int16Array.from(
	float32.map(x= > (x > 0 ? x * 0x7fff : x * 0x8000)))console.log(encode(int16.buffer))
Copy the code

To verify that Base64 is correct, convert the output Base64 to Int16 PCM file under node and then play it with FFPlay to see if the audio plays properly

PCMFile playback

#Single channel sampling rate :16000 Int16
ffplay -f s16le -ar 16k -ac 1 test.pcm

#Dual-channel sampling rate :48000 Float32
ffplay -f f32le -ar 48000 -ac 2 test.pcm
Copy the code

Resampling/adjusting the sampling rate

Although the getUserMedia parameter can set the sampling rate, it does not work on the latest Chrome, so you need to manually resampling

const mediaStream = await window.navigator.mediaDevices.getUserMedia({
    audio: {
    	// sampleRate: 44100, // The sampleRate rate setting is not valid
        channelCount: 1./ / track
        // Cancellation: true, // Cancellation: cancellation
        // noiseSuppression: true, // pression is good for noise reduction}})Copy the code

This is done using wave-resampler

import { resample } from 'wave-resampler'

const inputSampleRate =  44100
const outputSampleRate = 16000
const resampledBuffers = resample(
    // Requires onAudioProcess to merge the array of buffers for each frame
	mergeArray(audioBuffers),
	inputSampleRate,
	outputSampleRate,
)
Copy the code

PCMturnMP3

import { Mp3Encoder } from 'lamejs'

let mp3buf
const mp3Data = []
const sampleBlockSize = 576 * 10 // The working cache, a multiple of 576
const mp3Encoder = new Mp3Encoder(1, outputSampleRate, kbps)
const samples = float32ToInt16(
  audioBuffers,
  inputSampleRate,
  outputSampleRate,
)

let remaining = samples.length
for (let i = 0; remaining >= 0; i += sampleBlockSize) {
  const left = samples.subarray(i, i + sampleBlockSize)
  mp3buf = mp3Encoder.encodeBuffer(left)
  mp3Data.push(new Int8Array(mp3buf))
  remaining -= sampleBlockSize
}

mp3Data.push(new Int8Array(mp3Encoder.flush()))
console.log(mp3Data)

// Utility functions
function float32ToInt16(audioBuffers, inputSampleRate, outputSampleRate) {
  const float32 = resample(
    // Requires onAudioProcess to merge the array of buffers for each frame
    mergeArray(audioBuffers),
    inputSampleRate,
    outputSampleRate,
  )
  const int16 = Int16Array.from(
    float32.map(x= > (x > 0 ? x * 0x7fff : x * 0x8000)))return int16
}
Copy the code

Lamejs is fine, but the size is large (160+KB), and WAV format is available if there is no storage requirement

> ls -alh-rwxrwxrwx 1 root root 95K April 22 12:45 12s.mp3* -rwxrwxrwx 1 root root 1.1m April 22 12:44 12s.wav* -rwxrwxrwx 1 root root 235K 4月 22 12:41 30s.mp3* -rwxrwxrwx 1 root root 2.6m 4月 22 12:40 30s.wav* -rwxrwxrwx 1 root root 63K 4月 22 12:49 8s.mp3* -rwxrwxrwx 1 root root 689K 4月 22 12:48 8s.wav*Copy the code

PCMturnWAV

function mergeArray(list) {
  const length = list.length * list[0].length
  const data = new Float32Array(length)
  let offset = 0
  for (let i = 0; i < list.length; i++) {
    data.set(list[i], offset)
    offset += list[i].length
  }
  return data
}

function writeUTFBytes(view, offset, string) {
  var lng = string.length
  for (let i = 0; i < lng; i++) {
    view.setUint8(offset + i, string.charCodeAt(i))
  }
}

function createWavBuffer(audioData, sampleRate = 44100, channels = 1) {
  const WAV_HEAD_SIZE = 44
  const buffer = new ArrayBuffer(audioData.length * 2 + WAV_HEAD_SIZE)
  // We need a view to manipulate the buffer
  const view = new DataView(buffer)
  // Write the wav header
  // RIFF chunk descriptor/identifier
  writeUTFBytes(view, 0.'RIFF')
  // RIFF chunk length
  view.setUint32(4.44 + audioData.length * 2.true)
  // RIFF type
  writeUTFBytes(view, 8.'WAVE')
  // format chunk identifier
  // FMT sub-chunk
  writeUTFBytes(view, 12.'fmt')
  // format chunk length
  view.setUint32(16.16.true)
  // sample format (raw)
  view.setUint16(20.1.true)
  // stereo (2 channels)
  view.setUint16(22, channels, true)
  // sample rate
  view.setUint32(24, sampleRate, true)
  // byte rate (sample rate * block align)
  view.setUint32(28, sampleRate * 2.true)
  // block align (channel count * bytes per sample)
  view.setUint16(32, channels * 2.true)
  // bits per sample
  view.setUint16(34.16.true)
  // data sub-chunk
  // data chunk identifier
  writeUTFBytes(view, 36.'data')
  // data chunk length
  view.setUint32(40, audioData.length * 2.true)

  // Write PCM data
  let index = 44
  const volume = 1
  const { length } = audioData
  for (let i = 0; i < length; i++) {
    view.setInt16(index, audioData[i] * (0x7fff * volume), true)
    index += 2
  }
  return buffer
}

// Requires onAudioProcess to merge the array of buffers for each frame
createWavBuffer(mergeArray(audioBuffers))
Copy the code

WAV is basically PCM plus some audio information

Simple short-term energy calculation

function shortTimeEnergy(audioData) {
  let sum = 0
  const energy = []
  const { length } = audioData
  for (let i = 0; i < length; i++) {
    sum += audioData[i] ** 2

    if ((i + 1) % 256= = =0) {
      energy.push(sum)
      sum = 0
    } else if (i === length - 1) {
      energy.push(sum)
    }
  }
  return energy
}
Copy the code

Because the calculation results will vary greatly due to the recording gain of the equipment, the calculated data will also be large, so the ratio is used to simply distinguish between human voice and noise

To view the DEMO

const NoiseVoiceWatershedWave = 2.3
const energy = shortTimeEnergy(e.inputBuffer.getChannelData(0).slice(0))
const avg = energy.reduce((a, b) = > a + b) / energy.length

const nextState = Math.max(... energy) / avg > NoiseVoiceWatershedWave ?'voice' : 'noise'
Copy the code

Web WorkerOptimize performance

The volume of audio data is large, so it can be optimized using a Web Worker without blocking the UI thread

The Web Worker is relatively simple in the Webpack project. Just install the worker-Loader

preact.config.js

export default (config, env, helpers) => {
    config.module.rules.push({
        test: /\.worker\.js$/.use: { loader: 'worker-loader'.options: { inline: true}}})}Copy the code

recorder.worker.js

self.addEventListener('message', event => {
  console.log(event.data)
  // to MP3/ Base64/ WAV etc
  const output = ' '
  self.postMessage(output)
}
Copy the code

The use of the Worker

async function toMP3(audioBuffers, inputSampleRate, outputSampleRate = 16000) {
  const { default: Worker } = await import('./recorder.worker')
  const worker = new Worker()
  // Easy to use, the project can create worker instance when the Recorder is instantiated, there is a method needs to create multiple instances

  return new Promise(resolve= > {
    worker.postMessage({
      audioBuffers: audioBuffers,
      inputSampleRate: inputSampleRate,
      outputSampleRate: outputSampleRate,
      type: 'mp3',
    })
    worker.onmessage = event= > resolve(event.data)
  })
}
Copy the code

Audio storage

There are LocalStorage and IndexedDB. LocalStorage is more common, but can only store strings. IndexedDB can store bloBs directly. Therefore, IndexedDB is preferred. If LocalStorage is used, it needs to be converted to Base64. The volume will be larger

So in order to avoid taking up too much space for users, so choose MP3 format for storage

> ls -alh-rwxrwxrwx 1 root root 95K April 22 12:45 12s.mp3* -rwxrwxrwx 1 root root 1.1m April 22 12:44 12s.wav* -rwxrwxrwx 1 root root 235K 4月 22 12:41 30s.mp3* -rwxrwxrwx 1 root root 2.6m 4月 22 12:40 30s.wav* -rwxrwxrwx 1 root root 63K 4月 22 12:49 8s.mp3* -rwxrwxrwx 1 root root 689K 4月 22 12:48 8s.wav*Copy the code

IndexedDB is encapsulated as follows, familiar with the background of students can find an ORM library to facilitate data reading and writing

const indexedDB =
  window.indexedDB ||
  window.webkitIndexedDB ||
  window.mozIndexedDB ||
  window.OIndexedDB ||
  window.msIndexedDB

const IDBTransaction =
  window.IDBTransaction ||
  window.webkitIDBTransaction ||
  window.OIDBTransaction ||
  window.msIDBTransaction

const readWriteMode =
  typeof IDBTransaction.READ_WRITE === 'undefined'
    ? 'readwrite'
    : IDBTransaction.READ_WRITE

const dbVersion = 1
const storeDefault = 'mp3'

let dbLink

function initDB(store) {
  return new Promise((resolve, reject) = > {
    if (dbLink) resolve(dbLink)

    // Create/open database
    const request = indexedDB.open('audio', dbVersion)

    request.onsuccess = event= > {
      const db = request.result

      db.onerror = event= > {
        reject(event)
      }

      if (db.version === dbVersion) resolve(db)
    }

    request.onerror = event= > {
      reject(event)
    }

    // For future use. Currently only in latest Firefox versions
    request.onupgradeneeded = event= > {
      dbLink = event.target.result
      const { transaction } = event.target

      if(! dbLink.objectStoreNames.contains(store)) { dbLink.createObjectStore(store) } transaction.oncomplete =event= > {
        // Now store is available to be populated
        resolve(dbLink)
      }
    }
  })
}

export const writeIDB = async (name, blob, store = storeDefault) => {
  const db = await initDB(store)

  const transaction = db.transaction([store], readWriteMode)
  const objStore = transaction.objectStore(store)

  return new Promise((resolve, reject) = > {
    const request = objStore.put(blob, name)
    request.onsuccess = event= > resolve(event)
    request.onerror = event= > reject(event)
    transaction.commit && transaction.commit()
  })
}

export const readIDB = async (name, store = storeDefault) => {
  const db = await initDB(store)

  const transaction = db.transaction([store], readWriteMode)
  const objStore = transaction.objectStore(store)

  return new Promise((resolve, reject) = > {
    const request = objStore.get(name)
    request.onsuccess = event= > resolve(event.target.result)
    request.onerror = event= > reject(event)
    transaction.commit && transaction.commit()
  })
}

export const clearIDB = async (store = storeDefault) => {
  const db = await initDB(store)

  const transaction = db.transaction([store], readWriteMode)
  const objStore = transaction.objectStore(store)
  return new Promise((resolve, reject) = > {
    const request = objStore.clear()
    request.onsuccess = event= > resolve(event)
    request.onerror = event= > reject(event)
    transaction.commit && transaction.commit()
  })
}
Copy the code

WebViewopenWebRTC

See WebView WebRTC Not working

webView.setWebChromeClient(new WebChromeClient(){
	@TargetApi(Build.VERSION_CODES.LOLLIPOP)
	@Override
	public void onPermissionRequest(final PermissionRequest request) { request.grant(request.getResources()); }});Copy the code

The last

Above is a simple summary of personal recording project practice, hope to have a recording needs of students to help, the following will share the DEMO of the wave effect practice

reference

Juejin. Cn/post / 684490…

Blog.addpipe.com/recording-a…

Stackoverflow.com/questions/3…

Juejin. Cn/post / 684490…

www.cnblogs.com/xingshansi/…

Blog.csdn.net/qq_39516859…

www.cnblogs.com/CoderTian/p…

Blog.csdn.net/qcyfred/art…