Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.
preface
If a file larger than 100 MB is encountered in a project, fragment uploading is required to improve the uploading success rate through resumable upload and retry.
There are three reasons for upload failure.
- The server set a single file size threshold.
- The client sets the request timeout period.
- The network fails.
In project practice, can rely on Ali Cloud OSS to achieve (Ali cloud upload files). You should be able to get started quickly with documentation, but this article won’t cover that much. 🤞
The main body of the article is to achieve large file fragmentation upload, the purpose is also to go from shallow to deep, involved in the pit will be detailed.
The code is a combination of online groupies and their own bad ideas.
A lot of knowledge points, it is recommended to collect step by step after knocking, the end of the article will integrate all the code, there is a direct copy comparison.
It is to want to put an online case originally, be afraid you pass to me what not decent video 😘
The problem
Before we officially talk about large file sharding upload, there are a few small problems, you can pay attention to.
-
What is the content-Type of the shard transport request?
-
What is the Ajax request upload progress function? Axios wrapped, okay?
Shard to upload
The basic page
Same old same old! Again, take Vue3. Start with an ugly but tidy page 🤦♂️.
<template>
<div class="file-upload-fragment">
<div class="file-upload-fragment-container">
<el-upload class="fufc-upload"
action=""
:on-change="handleFileChange"
:auto-upload="false"
:show-file-list="false"
>
<template #trigger>
<el-button class="fufc-upload-file" size="small" type="primary">Select the file</el-button>
</template>
<el-button
class="fufc-upload-server"
size="small"
type="success"
@click="handleUploadFile"
>Uploading to the server</el-button>
<el-button
class="fufc-upload-stop"
size="small"
type="primary"
@click="stopUpload"
>pause</el-button>
<el-button
class="fufc-upload-continue"
size="small"
type="success"
@click="continueUpload"
></el-button ></el-upload>
<el-progress :percentage="percentage" color="#409eff" />
</div>
</div>
</template>
<script setup>
import { ref } from 'vue'
let percentage = ref(0)
/ * * *@description: File upload Change event *@param {*}
* @return {*}* /
const handleFileChange = async (file) => {
}
/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async () => {
}
/ * * *@descriptionPause upload Click event *@param {*}
* @return {*}* /
const stopUpload = () = >{}/ * * *@description: Continue uploading the Click event *@param {*}
* @return {*}* /
const continueUpload = () = >{}</script>
<style scoped lang="scss">
.file-upload-fragment {
width: 100%;
height: 100%;
padding: 10px;
&-container {
position: relative;
margin: 0 auto;
width: 600px;
height: 300px;
top: calc(50% - 150px);
min-width: 400px;
.fufc-upload {
display: flex;
justify-content: space-between;
align-items: center;
}
.el-progress {
margin-top: 10px; : :v-deep(.el-progress__text) {
min-width: 0px; }}}}</style>
Copy the codeSelect the file
Declare the variable currentFile to complete the handleFileChange event.
let currentFile = ref(null)
/ * * *@description: File upload Change event *@param {*}
* @return {*}* /
const handleFileChange = async (file) => {
if(! file)return
currentFile.value = file
}
Copy the codeCreate a shard
Slice a File in file. slice mode. File is based on blobs, which are binary objects. File inherits the functionality of Blob and extends it to support files on the user’s system.
const chunkSize = 5 * 1024 * 1024
/ * * *@description: Create file sharding *@param {*}
* @return {*}* /
const createChunkList = (file, chunkSize) = > {
const fileChunkList = []
let cur = 0
while (cur < file.size) {
fileChunkList.push(file.slice(cur, cur + chunkSize))
cur += chunkSize
}
return fileChunkList
}
Copy the codeIdentity documents
Slice files using the Spark-MD5 library to generate Hash values, which are used to identify files and perform operations such as second transmission based on the Hash values. Read file slices with FileReader.
import SparkMD5 from 'spark-md5'
/ * * *@description: Generates file Hash *@param {*}
* @return {*}* /
const createMD5 = (fileChunkList) = > {
return new Promise((resolve, reject) = > {
const slice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice
const chunks = fileChunkList.length
let currentChunk = 0
let spark = new SparkMD5.ArrayBuffer()
let fileReader = new FileReader()
// Read file slices
fileReader.onload = function (e) {
spark.append(e.target.result)
currentChunk++
if (currentChunk < chunks) {
loadChunk()
} else {
// Reads the slice and returns the Hash value of the final file
resolve(spark.end())
}
}
fileReader.onerror = function (e) {
reject(e)
}
function loadChunk() {
fileReader.readAsArrayBuffer(fileChunkList[currentChunk])
}
loadChunk()
})
}
Copy the codeSection to upload
Slicing and file Hash are ready. Next, we need to do two things.
- Mark our slices. The purpose is to determine which slices have not been uploaded successfully and ensure the retransmission is effective.
- Upload all slices concurrently, transfer data in formData format, see our request from Chrome’s Network panel
content-typeismultipart/form-data, details 🙌, may be asked oh.
Let’s improve the handleUploadFile function
import { postUploadFile } from '@/api/api.js'
import { ElMessage } from 'element-plus'
let chunkFormData = ref([])
let fileHash = ref(null)
/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async() = > {if(! currentFile) { ElMessage.warning('Please select file')
return
}
// File fragmentation
let fileChunkList = createChunkList(currentFile.value.raw, chunkSize)
fileHash.value = await createMD5(currentFile.value.raw, chunkSize)
let chunkList = fileChunkList.map((file, index) = > {
return {
file: file,
chunkHash: fileHash.value + The '-' + index,
fileHash: fileHash.value,
}
})
chunkFormData.value = chunkList.map((chunk) = > {
let formData = new FormData()
formData.append('chunk', chunk.file)
formData.append('chunkHash', chunk.chunkHash)
formData.append('fileHash', chunk.fileHash)
return {
formData: formData,
}
})
Promise.all(
chunkFormData.value.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(data.formData)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
)
}
Copy the codeFile header added interface postUploadFile, let’s use Koa framework to write the interface. Use file Hash as the name of the folder where the slices are stored.
const fsExtra = require('fs-extra')
const path = require('path')
const UPLOAD_DIR = path.resolve(__dirname, '.. '.'files')
class FileController {
static async uploadFile(ctx) {
// Slices are retrieved from the files field, not from the body
const file = ctx.request.files.chunk
// Get the Hash and slice number of the file
const body = ctx.request.body
const fileHash = body.fileHash
const chunkHash = body.chunkHash
const chunkDir = `${UPLOAD_DIR}/${fileHash}`
const chunkIndex = chunkHash.split(The '-') [1]
const chunkPath = `${UPLOAD_DIR}/${fileHash}/${chunkIndex}`
// If no directory exists, create a new directory
if(! fsExtra.existsSync(chunkDir)) {await fsExtra.mkdirs(chunkDir)
}
// File. path is the temporary address for uploading slices
await fsExtra.move(file.path, path.resolve(chunkDir, chunkHash.split(The '-') [1]))
ctx.success('received file chunk')}}Copy the codeI uploaded a 105M file with a base of 5M and the server accepted slices successfully, as shown below.
Upload progress
Uploads of 100M May be ok, but when the scale rises to G, the elegant progress bar can greatly improve the user experience. Native Ajax is onProgress is the event. I’m using Axios, which wraps the onUploadProgress function on top of Ajax.
The Percentage field is added to verify whether fragments are uploaded. Total file upload progress = Uploaded fragments/Total number of fragments. The Percentage field bound to the progress bar responds using computed.
import {
ref,
+ computed
} from 'vue'
let percentage = computed(() = > {
if(! chunkFormData.value.length)return 0
let uploaded = chunkFormData.value.filter((item) = > item.percentage).length
return Number(((uploaded / chunkFormData.value.length) * 100).toFixed(2))})/ * * *@description: shard returns call *@param {*}
* @return {*}* /
const uploadProgress = (item) = > {
return (e) = > {
item.percentage = parseInt(String((e.loaded / e.total) * 100))}}/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async () => {
...
chunkFormData.value = chunkList.map((chunk) = > {
let formData = new FormData()
formData.append('chunk', chunk.file)
formData.append('chunkHash', chunk.chunkHash)
formData.append('fileHash', chunk.fileHash)
return {
formData: formData,
+ percentage: 0}})Promise.all(
chunkFormData.value.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(
data.formData,
+ uploadProgress(data)
)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
)
}
Copy the codeMerge files
After all fragment files are successfully uploaded, the front-end requests to merge the fragment interface. The server finds the corresponding folder based on the Hash value of the transmitted file and sorts all fragments according to the fragment number to merge the fragments.
The backend mergeUploadFile interface is called after promise.all returns successfully
import {
postUploadFile
+ mergeUploadFile
} from '@/api/api.js'
/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async () => {
...
Promise.all(
chunkFormData.value.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(
data.formData,
uploadProgress(data)
)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
+ ).then(() = > {
+ mergeUploadFile({
+ fileName: currentFile.value.name,
+ fileHash: fileHash.value,
+ chunkSize: chunkSize
+ })
+ })
}
Copy the codeBack-end authoring interface.
static async mergeUploadFile(ctx) {
const params = ctx.request.query
const fileHash = params.fileHash
const chunkSize = params.chunkSize
const fileName = params.fileName
const chunkDir = path.resolve(UPLOAD_DIR, fileHash)
// Read all shards under the folder
const chunkPaths = await fsExtra.readdir(chunkDir)
const chunkNumber = chunkPaths.length
let count = 0
// Slice sort to prevent out of order
chunkPaths.sort((a, b) = > a - b)
chunkPaths.forEach((chunk, index) = > {
const chunkPath = path.resolve(UPLOAD_DIR, fileHash, chunk)
// Create a writable stream
const writeStream = fsExtra.createWriteStream(fileHash + fileName, {
start: index * chunkSize,
end: (index + 1) * chunkSize
})
// Create a readable stream
const readStream = fsExtra.createReadStream(chunkPath)
readStream.on('end'.() = > {
// Delete the slice file
fsExtra.unlinkSync(chunkPath)
count++
// Delete the slice folder
if (count === chunkNumber) {
fsExtra.rmdirSync(chunkDir)
let uploadedFilePath = path.resolve(__dirname, '.. ', fileHash + fileName)
fsExtra.move(uploadedFilePath, UPLOAD_DIR + '/' + fileHash + fileName)
}
})
readStream.pipe(writeStream)
})
ctx.success('file merged')}Copy the codeAt this point, our large file upload is basically completed.
Check out my video has been merged successfully ~ if follow this step down, you can too. ✌
File for heavy
Upload file function added file judge, file judge according to the file name + file hash judge.
import {
postUploadFile
mergeUploadFile
+ verifyUpload
} from '@/api/api.js'
/ * * *@description: File upload *@param {*}
* @return {*}* /
const handleUploadFile = async() = > {if(! currentFile) { ElMessage.warning('Please select file')
return
}
// File fragmentation
let fileChunkList = createChunkList(currentFile.value.raw, chunkSize)
fileHash.value = await createMD5(fileChunkList, chunkSize)
// Check whether the file exists
+ let { isUploaded } = await verifyUpload({
+ fileHash: fileHash.value,
+ fileName: currentFile.value.name
+ })
+ if (isUploaded) {
+ ElMessage.warning('File already exists')
+ return+}let chunkList = fileChunkList.map((file, index) = > {
return {
file: file,
chunkHash: fileHash.value + The '-' + index,
fileHash: fileHash.value
}
})
...
}
Copy the codeAdd the verifyUpload function on the backend
static async verifyUpload(ctx) {
const params = ctx.request.params
const fileHash = params.fileHash
const fileName = params.fileName
const filePath = path.resolve(
__dirname,
'.. '.`files/${fileHash + fileName}`
)
if (fsExtra.existsSync(filePath)) {
ctx.success(
{
isUploaded: true
},
'file is uploaded')}else {
ctx.success(
{
isUploaded: false
},
'file need upload ')}}Copy the codepause
In real life, there should be less need to pause uploads and more need to simulate abnormal network conditions. Here we use the CancelToken function in Axios. Add the cancelToken field to each shard.
import axios from 'axios'
const cancelToken = axios.CancelToken
/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async () => {
...
chunkFormData.value = chunkList.map((chunk) = > {
let formData = new FormData()
formData.append('chunk', chunk.file)
formData.append('chunkHash', chunk.chunkHash)
formData.append('fileHash', chunk.fileHash)
return {
formData: formData,
percentage: 0,
+ cancelToken: cancelToken.source()
}
})
...
}
Copy the codeComplete the stopUpload function.
/ * * *@description: Pause upload *@param {*}
* @return {*}* /
const stopUpload = () = > {
chunkFormData.value.forEach((data) = > {
data.cancelToken.cancel('Cancel upload')
// Ensure continuation
data.cancelToken = cancelToken.source()
})
}
Copy the codeRestrictions are added to merge files so that they can be merged only after all fragments are uploaded successfully.
/ * * *@description: file upload Click event *@param {*}
* @return {*}* /
const handleUploadFile = async () => {
...
Promise.all(
chunkFormData.value.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(
data.formData,
uploadProgress(data),
data.cancelToken.token
)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
).then((data) = >{+if(! data.includes(undefined)) {
mergeUploadFile({
fileName: currentFile.value.name,
fileHash: fileHash.value,
chunkSize: chunkSize
})
}
+ })
}
Copy the codeBreakpoint continuingly
The front-end filter only uploads fragments that have not been uploaded before, and the back-end filter also has restrictions.
ContinueUpload function is improved, in fact, the previous promise.all part of the package.
/ * * *@description: Resumable from breakpoint *@param {*}
* @return {*}* /
const continueUpload = () = > {
let notUploaded = chunkFormData.value.filter((item) = >! item.percentage)Promise.all(
notUploaded.value.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(
data.formData,
uploadProgress(data),
data.cancelToken.token
)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
).then((data) = > {
if(! data.includes(undefined)) {
mergeUploadFile({
fileName: currentFile.value.name,
fileHash: fileHash.value,
chunkSize: chunkSize
})
}
})
}
Copy the codeThere is a limit to whether the server can add slices.
static async uploadFile(ctx) {
const file = ctx.request.files.chunk
const body = ctx.request.body
const fileHash = body.fileHash
const chunkHash = body.chunkHash
const chunkDir = `${UPLOAD_DIR}/${fileHash}`
const chunkIndex = chunkHash.split(The '-') [1]
const chunkPath = `${UPLOAD_DIR}/${fileHash}/${chunkIndex}`
// If no directory exists, create a new directory
if(! fsExtra.existsSync(chunkDir)) {await fsExtra.mkdirs(chunkDir)
}
// Check whether the slice exists, the non-existent moving slice
+ if(! fsExtra.existsSync(chunkPath)) {await fsExtra.move(
file.path,
path.resolve(chunkDir, chunkHash.split(The '-') [1])
)
+ }
ctx.success('received file chunk')}Copy the codeAt this point, it’s finally done.
Complete source code
The front end
<template>
<div class="file-upload-fragment">
<div class="file-upload-fragment-container">
<el-upload
class="fufc-upload"
action=""
:on-change="handleFileChange"
:auto-upload="false"
:show-file-list="false"
>
<template #trigger>
<el-button class="fufc-upload-file" size="small" type="primary">Select the file</el-button>
</template>
<el-button
class="fufc-upload-server"
size="small"
type="success"
@click="handleUploadFile"
>Uploading to the server</el-button>
<el-button
class="fufc-upload-stop"
size="small"
type="primary"
@click="stopUpload"
>pause</el-button>
<el-button
class="fufc-upload-continue"
size="small"
type="success"
@click="continueUpload"
></el-button ></el-upload>
<el-progress :percentage="percentage" color="#409eff" />
</div>
</div>
</template>
<script setup>
import { ref, computed } from 'vue'
import { postUploadFile, mergeUploadFile, verifyUpload } from '@/api/api.js'
import { ElMessage } from 'element-plus'
import axios from 'axios'
import SparkMD5 from 'spark-md5'
const cancelToken = axios.CancelToken
const chunkSize = 5 * 1024 * 1024
/ * * *@description: Generates file hash *@param {*}
* @return {*}* /
const createMD5 = (fileChunkList) = > {
return new Promise((resolve, reject) = > {
const slice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice
const chunks = fileChunkList.length
let currentChunk = 0
let spark = new SparkMD5.ArrayBuffer()
let fileReader = new FileReader()
fileReader.onload = function (e) {
spark.append(e.target.result)
currentChunk++
if (currentChunk < chunks) {
loadChunk()
} else {
resolve(spark.end())
}
}
fileReader.onerror = function (e) {
reject(e)
}
function loadChunk() {
fileReader.readAsArrayBuffer(fileChunkList[currentChunk])
}
loadChunk()
})
}
let currentFile = ref(null)
let chunkFormData = ref([])
let fileHash = ref(null)
let percentage = computed(() = > {
if(! chunkFormData.value.length)return 0
let uploaded = chunkFormData.value.filter((item) = > item.percentage).length
return Number(((uploaded / chunkFormData.value.length) * 100).toFixed(2))})/ * * *@description: Create file sharding *@param {*}
* @return {*}* /
const createChunkList = (file, chunkSize) = > {
const fileChunkList = []
let cur = 0
while (cur < file.size) {
fileChunkList.push(file.slice(cur, cur + chunkSize))
cur += chunkSize
}
return fileChunkList
}
/ * * *@description: Select file event *@param {*}
* @return {*}* /
const handleFileChange = async (file) => {
if(! file)return
currentFile.value = file
}
/ * * *@description: shard returns call *@param {*}
* @return {*}* /
const uploadProgress = (item) = > {
return (e) = > {
item.percentage = parseInt(String((e.loaded / e.total) * 100))}}/ * * *@description: Pause upload *@param {*}
* @return {*}* /
const stopUpload = () = > {
chunkFormData.value.forEach((data) = > {
data.cancelToken.cancel('Cancel upload')
data.cancelToken = cancelToken.source()
})
}
/ * * *@description: Resumable from breakpoint *@param {*}
* @return {*}* /
const continueUpload = () = > {
let notUploaded = chunkFormData.value.filter((item) = >! item.percentage)Promise.all(
notUploaded.map((data) = > {
return new Promise((resolve, reject) = > {
postUploadFile(
data.formData,
uploadProgress(data),
data.cancelToken.token
)
.then((data) = > {
resolve(data)
})
.catch((err) = > {
reject(err)
})
})
})
).then((data) = > {
if(! data.includes(undefined)) {
mergeUploadFile({
fileName: currentFile.value.name,
fileHash: fileHash.value,
chunkSize: chunkSize
})
}
})
}
/ * * *@description: File upload *@param {*}
* @return {*}* /
const handleUploadFile = async() = > {if(! currentFile) { ElMessage.warning('Please select file')
return
}
// File fragmentation
let fileChunkList = createChunkList(currentFile.value.raw, chunkSize)
/ / file hash
// let fileHash = await MultiThreadCreateMD5(currentFile.value.raw, chunkSize)
fileHash.value = await createMD5(fileChunkList, chunkSize)
// Check whether the file exists
let { isUploaded } = await verifyUpload({
fileHash: fileHash.value,
fileName: currentFile.value.name
})
if (isUploaded) {
ElMessage.warning('File already exists')
return
}
let chunkList = fileChunkList.map((file, index) = > {
return {
file: file,
chunkHash: fileHash.value + The '-' + index,
fileHash: fileHash.value
}
})
chunkFormData.value = chunkList.map((chunk) = > {
let formData = new FormData()
formData.append('chunk', chunk.file)
formData.append('chunkHash', chunk.chunkHash)
formData.append('fileHash', chunk.fileHash)
return {
formData: formData,
percentage: 0.cancelToken: cancelToken.source()
}
})
continueUpload()
}
</script>
<style scoped lang="scss">
.file-upload-fragment {
width: 100%;
height: 100%;
padding: 10px;
&-container {
position: relative;
margin: 0 auto;
width: 600px;
height: 300px;
top: calc(50% - 150px);
min-width: 400px;
.fufc-upload {
display: flex;
justify-content: space-between;
align-items: center;
}
.el-progress {
margin-top: 10px; : :v-deep(.el-progress__text) {
min-width: 0px; }}}}</style>
Copy the codeThe back-end
const fsExtra = require('fs-extra')
const path = require('path')
const UPLOAD_DIR = path.resolve(__dirname, '.. '.'files')
class FileController {
static async uploadFile(ctx) {
const file = ctx.request.files.chunk
const body = ctx.request.body
const fileHash = body.fileHash
const chunkHash = body.chunkHash
const chunkDir = `${UPLOAD_DIR}/${fileHash}`
const chunkIndex = chunkHash.split(The '-') [1]
const chunkPath = `${UPLOAD_DIR}/${fileHash}/${chunkIndex}`
// If no directory exists, create a new directory
if(! fsExtra.existsSync(chunkDir)) {await fsExtra.mkdirs(chunkDir)
}
// Check whether the slice exists, the non-existent moving slice
if(! fsExtra.existsSync(chunkPath)) {await fsExtra.move(
file.path,
path.resolve(chunkDir, chunkHash.split(The '-') [1])
)
}
ctx.success('received file chunk')}static async mergeUploadFile(ctx) {
const params = ctx.request.query
const fileHash = params.fileHash
const chunkSize = params.chunkSize
const fileName = params.fileName
const chunkDir = path.resolve(UPLOAD_DIR, fileHash)
const chunkPaths = await fsExtra.readdir(chunkDir)
const chunkNumber = chunkPaths.length
let count = 0
// Slice sort to prevent out of order
chunkPaths.sort((a, b) = > a - b)
chunkPaths.forEach((chunk, index) = > {
const chunkPath = path.resolve(UPLOAD_DIR, fileHash, chunk)
// Create a writable stream
const writeStream = fsExtra.createWriteStream(fileHash + fileName, {
start: index * chunkSize,
end: (index + 1) * chunkSize
})
const readStream = fsExtra.createReadStream(chunkPath)
readStream.on('end'.() = > {
// Delete the slice file
fsExtra.unlinkSync(chunkPath)
count++
// Delete the folder
if (count === chunkNumber) {
fsExtra.rmdirSync(chunkDir)
let uploadedFilePath = path.resolve(
__dirname,
'.. ',
fileHash + fileName
)
fsExtra.move(uploadedFilePath, UPLOAD_DIR + '/' + fileHash + fileName)
}
})
readStream.pipe(writeStream)
})
ctx.success('file merged')}static async verifyUpload(ctx) {
const params = ctx.request.params
const fileHash = params.fileHash
const fileName = params.fileName
const filePath = path.resolve(
__dirname,
'.. '.`files/${fileHash + fileName}`
)
if (fsExtra.existsSync(filePath)) {
ctx.success(
{
isUploaded: true
},
'file is uploaded')}else {
ctx.success(
{
isUploaded: false
},
'file need upload ')}}}module.exports = FileController
Copy the codeconclusion
The general uploading process of large file fragments is as follows:
- use
blob.sliceSlice files. - Use by slice
spark-md5The filehashValue that uniquely identifies the file. - Multiple slices are concurrently requested. After all slices are uploaded successfully, files are merged.
- The use of axios
onUploadProgressMonitor file upload and obtain the file upload progress. - If the slices are not fully uploaded due to network errors, resumable upload is performed.
Pick out the details, don’t forget my two small questions 😁.
In real projects, there are more details, which can be expanded based on this article.