“This is the fifth day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”
Because the company is engaged in voice translation, there is a trial function on the official website, which requires recording on the web page and uploading it to the background for translation and voice synthesis.
The business requirement is to use the recording function on the web page, communicate once every 100ms, transmit the data to the back end for a set of link processing,
The original business logic is to use HTTP to communicate every 1s, but we want to optimize it to use Websocket to communicate and reduce the translation delay to 100ms, so we optimize it.
1. Project problems
The front-end framework used in the project is Vue, but SINCE I am a back-end developer, I do not understand Vue. The plug-in used for front-end development is recorderX. When sending data per second is normal, but at the time of adjust the timer for 100 ms, data is always can’t normal to send, or an empty package, or the amount of data is large, more than the length of the back end cap, lead to can’t handle, business also cannot meet the demand, because the front there are other things to deal with, in the study after a few days, Gave up and left the problem lying around
You can see that the first one is only 44 bytes, and then it stays the same for a few frames, and then it suddenly gets bigger.
2, the path to solve the problem
Because I am not very busy with the project recently, I temporarily put the recording problem in my hand for some time. If there is any problem, I will solve it. The following is my path to solve the problem.
1. Learn VUE grammar
Front-end I am not quite understand, only in ancient times wrote JSP and JS, Vue this advanced framework WHERE I understand, so have to learn. I have to say that the front end has evolved quite well, although Vue still does not understand, but probably know that vue module is divided into three parts
The first part is the interface configuration
The second part is the logical code
The third part is web style
<template>
<div id="app">
<img alt="Vue logo" src="./assets/logo.png">
<HelloWorld msg="Welcome to Your Vue.js App"/>
</div>
</template>
<script>
import HelloWorld from './components/HelloWorld.vue'
export default {
name: 'App',
components: {
HelloWorld
}
}
</script>
<style>
#app {
font-family: Avenir, Helvetica, Arial, sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-align: center;
color: #2c3e50;
margin-top: 60px;
}
</style>
Copy the code2. Build an environment
The environment of Vue is quite complicated. The IDE I choose is Webstorm, but it is said that VScode is used in the front end. As a heavy user of IDEA, I resolutely choose WebStorm.
Install nodejs,
Deploy the VUE environment
After setting up the environment, I still can’t understand why I use the command to start NPM run serve every time, but it doesn’t affect me to solve the problem
3. Code issues
The front end of the problem is shortened to 100ms in the timer, the recorder at each time to obtain data can not normally obtain data, when the timer is slightly longer when it is normal
I have a few guesses based on this question
First: VUE refresh mechanism leads to the recorder data can not be technically into the buffer, (should not be)
Second: not using recorderX correctly, not clear buffer when fetching (no exception found after reading recorderX source code, try to call recorder. Clear (), then nothing useful)
Third: Writing problems (after trying several writing methods decisively give up)
Fourth: The plugin has a problem, change the plugin (in desperation, I gave up, directly changed the plugin, finally solved the problem perfectly)
3. Code solution
The final solution is to change the plug-in, will be replaced by the original plug-in, HZRecorder
Here is the source code, I did some optimization. The file name is hzorio.js
export function HZRecorder(stream, config) {
config = config || {};
config.sampleBits = config.sampleBits || 16; //采样数位 8, 16
config.sampleRate = config.sampleRate || 16000; //采样率16khz
var context = new (window.webkitAudioContext || window.AudioContext)();
var audioInput = context.createMediaStreamSource(stream);
var createScript = context.createScriptProcessor || context.createJavaScriptNode;
var recorder = createScript.apply(context, [4096, 1, 1]);
var audioData = {
size: 0 //录音文件长度
, buffer: [] //录音缓存
, inputSampleRate: context.sampleRate //输入采样率
, inputSampleBits: 16 //输入采样数位 8, 16
, outputSampleRate: config.sampleRate //输出采样率
, outputSampleBits: config.sampleBits //输出采样数位 8, 16
, input: function (data) {
this.buffer.push(new Float32Array(data));
this.size += data.length;
}
, compress: function (clearBuff) { //合并压缩
//合并
var data = new Float32Array(this.size);
var offset = 0;
for (var i = 0; i < this.buffer.length; i++) {
data.set(this.buffer[i], offset);
offset += this.buffer[i].length;
}
//压缩
var compression = parseInt(this.inputSampleRate / this.outputSampleRate);
var length = data.length / compression;
var result = new Float32Array(length);
var index = 0, j = 0;
while (index < length) {
result[index] = data[j];
j += compression;
index++;
}
// TODO 每次获取清空缓冲区
if (clearBuff){
this.buffer = []
this.size = 0
}
return result;
}
, encodeWAV: function (clearBuff) {
var sampleRate = Math.min(this.inputSampleRate, this.outputSampleRate);
var sampleBits = Math.min(this.inputSampleBits, this.outputSampleBits);
var bytes = this.compress(clearBuff);
var dataLength = bytes.length * (sampleBits / 8);
var buffer = new ArrayBuffer(44 + dataLength);
var data = new DataView(buffer);
var channelCount = 1;//单声道
var offset = 0;
var writeString = function (str) {
for (var i = 0; i < str.length; i++) {
data.setUint8(offset + i, str.charCodeAt(i));
}
}
// 资源交换文件标识符
writeString('RIFF'); offset += 4;
// 下个地址开始到文件尾总字节数,即文件大小-8
data.setUint32(offset, 36 + dataLength, true); offset += 4;
// WAV文件标志
writeString('WAVE'); offset += 4;
// 波形格式标志
writeString('fmt '); offset += 4;
// 过滤字节,一般为 0x10 = 16
data.setUint32(offset, 16, true); offset += 4;
// 格式类别 (PCM形式采样数据)
data.setUint16(offset, 1, true); offset += 2;
// 通道数
data.setUint16(offset, channelCount, true); offset += 2;
// 采样率,每秒样本数,表示每个通道的播放速度
data.setUint32(offset, sampleRate, true); offset += 4;
// 波形数据传输率 (每秒平均字节数) 单声道×每秒数据位数×每样本数据位/8
data.setUint32(offset, channelCount * sampleRate * (sampleBits / 8), true); offset += 4;
// 快数据调整数 采样一次占用字节数 单声道×每样本的数据位数/8
data.setUint16(offset, channelCount * (sampleBits / 8), true); offset += 2;
// 每样本数据位数
data.setUint16(offset, sampleBits, true); offset += 2;
// 数据标识符
writeString('data'); offset += 4;
// 采样数据总数,即数据总大小-44
data.setUint32(offset, dataLength, true); offset += 4;
// 写入采样数据
if (sampleBits === 8) {
for (var i = 0; i < bytes.length; i++, offset++) {
var s = Math.max(-1, Math.min(1, bytes[i]));
var val = s < 0 ? s * 0x8000 : s * 0x7FFF;
val = parseInt(255 / (65535 / (val + 32768)));
data.setInt8(offset, val, true);
}
} else {
for (var i2 = 0; i2 < bytes.length; i2++, offset += 2) {
var s2 = Math.max(-1, Math.min(1, bytes[i2]));
data.setInt16(offset, s2 < 0 ? s2 * 0x8000 : s2 * 0x7FFF, true);
}
}
return new Blob([data], { type: 'audio/wav' });
}
};
//开始录音
this.start = function () {
audioInput.connect(recorder);
recorder.connect(context.destination);
}
//停止
this.stop = function () {
recorder.disconnect();
}
//获取音频文件
this.getBlob = function (clearBuff) {
clearBuff = clearBuff || false;
// this.stop();
return audioData.encodeWAV(clearBuff);
}
//回放
this.play = function (audio) {
// var blob=this.getBlob();
// saveAs(blob, "F:/3.wav");
audio.src = window.URL.createObjectURL(this.getBlob());
}
//上传
this.upload = function () {
return this.getBlob()
}
//音频采集
recorder.onaudioprocess = function (e) {
audioData.input(e.inputBuffer.getChannelData(0));
//record(e.inputBuffer.getChannelData(0));
}
}
调用的地方：文件名为HelloWorld.vue
<template>
<div>
<button @click="btnClick">开始</button>
</div>
</template>
<script>
import { HZRecorder} from '../HZRecorder.js';
export default {
name: 'HelloWorld',
props: {
msg: String
},
data() {
return {
index:0,
timeCount:0,
recorder :HZRecorder,
audio_context:AudioContext,
};
},
methods:{
btnClick: function () {
this.recorder.start()
setInterval(() => {
var blob = this.recorder.getBlob();
const file = new File([blob], 'audio.wav');
console.log("-----------"+ file.size)
var blob2 = blob.slice(this.index,blob.size -1);
console.log("----blob2-------" + blob2.size)
this.index = blob.size -1;
// this.recorder.start()
this.timeCount = this.timeCount +1
if (this.timeCount === 50){
const blobUrl = window.URL.createObjectURL(blob)
const a = document.createElement('a')
a.style.display = 'none'
a.download = new Date().getTime() +'.wav'
a.href = blobUrl
a.click()
}
}, 100)
}
},
mounted() {
var that = this
this.$nextTick(() => {
try {
// <!-- 检查是否能够调用麦克风 -->
window.AudioContext = window.AudioContext || window.webkitAudioContext;
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia;
window.URL = window.URL || window.webkitURL;
that.audio_context = new AudioContext;
console.log('navigator.getUserMedia ' + (navigator.getUserMedia ? 'available.' : 'not present!'));
} catch (e) {
alert('No web audio support in this browser!');
}
navigator.getUserMedia({audio: true}, function (stream) {
that.recorder = new HZRecorder(stream,{
sampleBits: 16,
sampleRate: 8000
})
console.log('初始化完成');
}, function(e) {
console.log('No live audio input: ' + e);
});
})
},
}
</script>
<!-- Add "scoped" attribute to limit CSS to this component only -->
<style scoped>
h3 {
margin: 40px 0 0;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
a {
color: #42b983;
}
</style>
Copy the codeClick start on the page,
F12 then opens the console and sees the output. Each frame is 1681 in length, completing the requirement
conclusion
Did the back-end pretty long front end suddenly contact, although want to learn but always feel a bit difficult, a little psychological unintelligent, although the final did speculation, also calculate to solve the problem, but still need to study in the field of strange state of mind to do a checking, to empty their own, can not use the original historical knowledge to reject the new contact of knowledge, absorbed to understand first