Recently, the company needs users to upload large files. A file may be larger than 1GB. If the upload is interrupted due to network fluctuation or user’s illegal operation, the company must start uploading again. As the front end, after consulting with the back end, I looked at some mature implementation schemes, and finally used breakpoint continuation to optimize the upload logic.
Full code – Github
What is breakpoint continuation
If the upload is terminated due to some reasons (refresh or network interruption), the next time you upload the same file, you should continue to upload the same file at the same place half of the last time to save the upload time.
Implementation approach
The main idea is to divide large files into several slices. Rather than transferring a file in its entirety to the server at once, the sliced pieces are passed to the server in pieces. The server will temporarily save the uploaded slices. When all the slices are uploaded successfully, the slices will be merged into a whole file.
Doing so can take advantage of the slice using the breakpoint continuation feature. The specific logic is that if the section is broken during uploading, the next time the same file is uploaded, a request can be sent to the server to obtain the uploaded sections and compare with the whole section before uploading the remaining unuploaded slices.
The back-end processing blades and merging of a detail point is: the big files to slice, the back-end need to each section of the file named ${original file HASH value} _ ${section number}. ${file} ‘f07ec272dbb0b883eed4b2f415625a90_2. Mp4’ for example. And name the temporary folder where the slices are stored as hash values. Finally, when all slices are uploaded, the merge interface is called, and the back-end merges slices in all temporary folders.
Apis provided by the back end are required
The following apis require back-end development
Gets the uploaded slice
/upload_already method:GET params: HASH: HASH name of the file return: Application /json code:0 succeeded 1 failed codeText: status description, fileList:[...]Copy the codeThis method is used to get the slice names of all slices of uploaded files, for example returning:
{
fileList: ['f07ec272dbb0b883eed4b2f415625a90_1.mp4'.'f07ec272dbb0b883eed4b2f415625a90_2.mp4'.'f07ec272dbb0b883eed4b2f415625a90_3.mp4']}Copy the codeMean for the HASH value for ‘f07ec272dbb0b883eed4b2f415625a9 files have been uploaded three slices, then you just need to start from the fourth section to upload. If the array is empty, the file is uploaded for the first time.
Upload section
Url :/upload_chunk method:POST params:multipart/form-data file: slice data filename: slice name HASH_ slice number generated by the file. Suffix Return :application/json code:0 succeeded 1 failed, codeText: status description, originalFilename: originalFilename, servicePath: file server addressCopy the codeUpload the slice file in the format of File object, and pass the slice name to the back end in the format of ${HASH value of the original file}_${slice number}.${file suffix}. The back end uses the HASH value to temporarily store the slice in a temporary folder.
Merge slice
Url :/upload_merge method:POST Params :application/x-www-form-urlencoded HASH: HASH name of the file Count: number of slices Return :application/json CodeText: status description, originalFilename: originalFilename, servicePath: file server addressCopy the codeWhen all the slices are uploaded, the merge interface is called, and the back end will merge the slices and delete the temporary folder storing the slices.
Implementation of front-end code details
Method for obtaining the HASH value
- use
FileReaderObject converts the selected file object tobuffer - document-based
bufferUse the MD5 library to generate the HASH value of the file.
Encapsulate as a function:
/** * Pass in the file object, return the HASH value generated by the file, suffix,buffer, new file name with HASH name *@param file
* @returns {Promise<unknown>}* /
const changeBuffer = file= > {
return new Promise(resolve= > {
let fileReader = new FileReader();
fileReader.readAsArrayBuffer(file);
fileReader.onload = ev= > {
let buffer = ev.target.result,
spark = new SparkMD5.ArrayBuffer(),
HASH,
suffix;
spark.append(buffer);
HASH = spark.end();
suffix = /.([a-zA-Z0-9]+)$/.exec(file.name)[1];
resolve({
buffer,
HASH,
suffix,
filename: `${HASH}.${suffix}`
});
};
});
};
Copy the codeslice
Files can be sliced using the slice function in Blob objects. Files can be cut to a custom size and number.
Blob. Slice documentation reference
For example, file.slice(0,1024) slices 0-1024 bytes of file data and returns a new object.
Overall HTML structure
<div class="container">
<div class="item">
<h3>Large file upload</h3>
<section class="upload_box" id="upload7">
<input type="file" class="upload_inp">
<div class="upload_button_box">
<button class="upload_button select">Upload a file</button>
</div>
<div class="upload_progress">
<div class="value"></div>
</div>
</section>
</div>
</div>
Copy the code
Use AXIos to send the request
/* Extract the public information from the request sent by AXIos */
// Create a single instance, without resolving project global or other AXIos conflicts
let instance = axios.create();
instance.defaults.baseURL = 'http://127.0.0.1:8888';
// The default format is multipart/form-data
instance.defaults.headers['Content-Type'] = 'multipart/form-data';
instance.defaults.transformRequest = (data, headers) = > {
// Compatible with X-www-form-urlencoded format requests sent
const contentType = headers['Content-Type'];
if (contentType === "application/x-www-form-urlencoded") return Qs.stringify(data);
return data;
};
// Unified result processing
instance.interceptors.response.use(response= > {
return response.data;
},reason= >{
// Process the unification failure
return Promise.reject(reason)
});
Copy the codeOverall logic and code
Detailed logic in the comments, written in more detail
Full code – Github
(function () {
let upload = document.querySelector('#upload7'),
upload_inp = upload.querySelector('.upload_inp'),
upload_button_select = upload.querySelector('.upload_button.select'),
upload_progress = upload.querySelector('.upload_progress'),
upload_progress_value = upload_progress.querySelector('.value');
const checkIsDisable = element= > {
let classList = element.classList;
return classList.contains('disable') || classList.contains('loading');
};
/** * Pass in the file object, return the HASH value generated by the file, suffix,buffer, new file name with HASH name *@param file
* @returns {Promise<unknown>}* /
const changeBuffer = file= > {
return new Promise(resolve= > {
let fileReader = new FileReader();
fileReader.readAsArrayBuffer(file);
fileReader.onload = ev= > {
let buffer = ev.target.result,
spark = new SparkMD5.ArrayBuffer(),
HASH,
suffix;
spark.append(buffer);
HASH = spark.end();
suffix = /.([a-zA-Z0-9]+)$/.exec(file.name)[1];
resolve({
buffer,
HASH,
suffix,
filename: `${HASH}.${suffix}`
});
};
});
};
upload_inp.addEventListener('change'.async function () {
//get native file object
let file = upload_inp.files[0];
if(! file)return;
//button add loading
upload_button_select.classList.add('loading');
//show progress
upload_progress.style.display = 'block';
// Get the HASH of the file
let already = [],// The name of the slice that has been uploaded
data = null,
{
HASH,
suffix
} = await changeBuffer(file);// Get the hash and suffix of the original file
// Get the uploaded slice information
try {
data = await instance.get('/upload_already', {
params: {
HASH
}
});
if (+data.code === 0) { already = data.fileList; }}catch (err) {}
// Implement file slice processing "fixed number & fixed size"
let max = 1024 * 100.// Set the slice size to 100KB
count = Math.ceil(file.size / max),// Get the slices that should be uploaded
index = 0.// use it when storing sliced arrays
chunks = [];// To store section values
if (count > 100) {// If the number of slices exceeds 100, only 100 will be cut, because too many slices will also affect the speed of calling the interface
max = file.size / 100;
count = 100;
}
while (index < count) {// Loop to generate slices
//index 0 => 0~max
//index 1 => max~max*2
//index*max ~(index+1)*max
chunks.push({
file: file.slice(index * max, (index + 1) * max),
filename: `${HASH}_${index+1}.${suffix}`
});
index++;
}
index = 0;
const clear = () = > {// When the upload is complete, return the status
upload_button_select.classList.remove('loading');
upload_progress.style.display = 'none';
upload_progress_value.style.width = '0%';
};
// Upload a slice successfully every time [progress control & slice Merge]
const complate = async() = > {// Control progress bar: Increase the length of progress bar a bit after uploading a slice
index++;
upload_progress_value.style.width = `${index/count*100}% `;
if (index < count) return;
// When all slices have been uploaded successfully, the slices are merged
upload_progress_value.style.width = ` 100% `;
try {
// Call the merge slice method
data = await instance.post('/upload_merge', {
HASH,
count
}, {
headers: {
'Content-Type': 'application/x-www-form-urlencoded'}});if (+data.code === 0) {
alert('Congratulations, the file uploaded successfully, you can based on${data.servicePath}Access the file ~~ ');
clear();
return;
}
throw data.codeText;
} catch (err) {
alert('Slice merge failed, please try again later ~~'); clear(); }};// Upload each slice loop
chunks.forEach(chunk= > {
// No need to upload already uploaded
// The format of already returned is [' hash_1.png ',' hash_2.png '], which is the slice name of the uploaded file
if (already.length > 0 && already.includes(chunk.filename)) {
// There is no need to call the interface to upload slices that have already been uploaded
complate();// Move the progress bar or merge all slices
return;
}
let fm = new FormData;
fm.append('file', chunk.file);
fm.append('filename', chunk.filename);
instance.post('/upload_chunk', fm).then(data= > {// Upload slices using form data format
if (+data.code === 0) {
complate();//// move the progress bar or merge all slices
return;
}
return Promise.reject(data.codeText);
}).catch(() = > {
alert('The current slice upload failed, please try again later ~~');
clear();
});
});
});
// Triggers the native upload file box
upload_button_select.addEventListener('click'.function () {
if (checkIsDisable(this)) return; upload_inp.click(); }); }) ();Copy the codeImplementation effect
When uploading for the first time, call /upload_already and /upload_chunk methods respectively to get the uploaded slices (empty array), and then split the slices and upload them one by one.
When refreshing the page at this time, terminal slice upload behavior. At this point, let’s look at the temporary data on the back end
You see a temporary folder named Hash to and 24 slices have been uploaded
If we select the same file upload again, the progress bar will immediately return to the same position as the last upload, already returning an array of the names of the 24 slices that have been uploaded.
Once the remaining slices have been uploaded, the Merge interface is called. Finished uploading
At this point, the temporary folder on the back end is deleted and merged into a single file. End of the upload
