background

When we do the file import function, if the imported file is too large, it may take a long time and need to upload again after failure. We need to combine the front and back ends to solve this problem

Train of thought

We need to do a few things:

Slicing a file, that is, splitting a request into multiple requests, reduces the time for each request, and if a request fails, you only need to re-send the request instead of starting from scratch
Notify the server to merge slices. After uploading slices, the front end notifies the server to merge slices
Controls the concurrency of multiple requests to prevent browser memory overflow and page congestion caused by multiple requests being sent at the same time
When multiple requests fail to be sent, such as network failure, page shutdown, etc., we have to process the failed requests and let them be sent repeatedly

implementation

The front end

Sample code repository

The warehouse address

Step 1- Slice and merge slices

In JavaScript, the FIle object is a subclass of the Blob object, which contains an important method, slice, through which we can split binary files as follows:

<! DOCTYPEhtml>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="Width = s, initial - scale = 1.0">
    <title>Document</title>
    <script src="https://cdn.bootcdn.net/ajax/libs/axios/0.24.0/axios.min.js"></script>
</head>
<body>
    <input type="file" id="fileInput">
    <button id="uploadBtn">upload</button>
</body>
<script>
// Request the base address
axios.defaults.baseURL = 'http://localhost:3000'
// The selected file
var file = null
// Select the file
document.getElementById('fileInput').onchange = function({target: {files}}){
    file = files[0]}// Start uploading
document.getElementById('uploadBtn').onclick = async function(){
    if(! file)return
    // Create a slice
    // let size = 1024 * 1024 * 10
    let size = 1024 * 50  //50KB slice size
    let fileChunks = []
    let index = 0 // Section number
    for(let cur = 0; cur < file.size; cur += size){
        fileChunks.push({
            hash: index++,
            chunk: file.slice(cur, cur + size)
        })
    }
    // Upload slices
    const uploadList = fileChunks.map((item, index) = > {
        let formData = new FormData()
        formData.append('filename', file.name)
        formData.append('hash', item.hash)
        formData.append('chunk', item.chunk)
        return axios({
            method: 'post'.url: '/upload'.data: formData
        })
    })
    await Promise.all(uploadList)
    // merge slices
    await axios({
        method: 'get'.url: '/merge'.params: {
            filename: file.name
        }
    });
    console.log('Upload completed')}</script>
</html>
Copy the code

Step 2- Concurrency control

Combined with promise.race and asynchronous function implementation, the number of concurrent requests to prevent overflow of browser memory, the code is as follows:

<! DOCTYPEhtml>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="Width = s, initial - scale = 1.0">
    <title>Document</title>
    <script src="https://cdn.bootcdn.net/ajax/libs/axios/0.24.0/axios.min.js"></script>
</head>
<body>
    <input type="file" id="fileInput">
    <button id="uploadBtn">upload</button>
</body>
<script>
// Request the base address
axios.defaults.baseURL = 'http://localhost:3000'
// The selected file
var file = null
// Select the file
document.getElementById('fileInput').onchange = function({target: {files}}){
    file = files[0]}// Start uploading
document.getElementById('uploadBtn').onclick = async function(){
    if(! file)return
    // Create a slice
    // let size = 1024 * 1024 * 10; //10MB slice size
    let size = 1024 * 50 //50KB slice size
    let fileChunks = []
    let index = 0 // Section number
    for(let cur = 0; cur < file.size; cur += size){
        fileChunks.push({
            hash: index++,
            chunk: file.slice(cur, cur + size)
        });
    }
    // Control concurrency
    let pool = []/ / concurrent pool
    let max = 3 // Maximum concurrency
    for(let i=0; i<fileChunks.length; i++){let item = fileChunks[i]
        let formData = new FormData()
        formData.append('filename', file.name)
        formData.append('hash', item.hash)
        formData.append('chunk', item.chunk)
        // Upload slices
        let task = axios({
            method: 'post'.url: '/upload'.data: formData
        })
        task.then((data) = >{
            // Remove the Promise task from the concurrency pool after the request ends
            let index = pool.findIndex(t= > t===task)
            pool.splice(index)
        })
        pool.push(task)
        if(pool.length === max){
            // Each time the concurrent pool finishes running a task, another task is inserted
            await Promise.race(pool)
        }
    }
    // All tasks completed, merge slices
    await axios({
        method: 'get'.url: '/merge'.params: {
            filename: file.name
        }
    });
    console.log('Upload completed')}</script>
</html>
Copy the code

Step 3- Resumable

After a single request fails, when the catch method is triggered, the current request is put into the failure list. After the completion of this round of requests, the failed request is processed repeatedly, with the specific code as follows:

<! DOCTYPEhtml>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="Width = s, initial - scale = 1.0">
    <title>Document</title>
    <script src="https://cdn.bootcdn.net/ajax/libs/axios/0.24.0/axios.min.js"></script>
</head>
<body>
    <input type="file" id="fileInput">
    <button id="uploadBtn">upload</button>
</body>
<script>
// Request the base address
axios.defaults.baseURL = 'http://localhost:3000'
// The selected file
var file = null
// Select the file
document.getElementById('fileInput').onchange = function({target: {files}}){
    file = files[0]}// Start uploading
document.getElementById('uploadBtn').onclick = function(){
    if(! file)return;
    // Create a slice
    // let size = 1024 * 1024 * 10; //10MB slice size
    let size = 1024 * 50; //50KB slice size
    let fileChunks = [];
    let index = 0 // Section number
    for(let cur = 0; cur < file.size; cur += size){
        fileChunks.push({
            hash: index++,
            chunk: file.slice(cur, cur + size)
        })
    }
    // Control concurrency and breakpoint continuation
    const uploadFileChunks = async function(list){
        if(list.length === 0) {// All tasks completed, merge slices
            await axios({
                method: 'get'.url: '/merge'.params: {
                    filename: file.name
                }
            });
            console.log('Upload completed')
            return
        }
        let pool = []/ / concurrent pool
        let max = 3 // Maximum concurrency
        let finish = 0// The number of completions
        let failList = []// List of failures
        for(let i=0; i<list.length; i++){let item = list[i]
            let formData = new FormData()
            formData.append('filename', file.name)
            formData.append('hash', item.hash)
            formData.append('chunk', item.chunk)
            // Upload slices
            let task = axios({
                method: 'post'.url: '/upload'.data: formData
            })
            task.then((data) = >{
                // Remove the Promise task from the concurrency pool after the request ends
                let index = pool.findIndex(t= > t===task)
                pool.splice(index)
            }).catch(() = >{
                failList.push(item)
            }).finally(() = >{
                finish++
                // All requests are completed
                if(finish===list.length){
                    uploadFileChunks(failList)
                }
            })
            pool.push(task)
            if(pool.length === max){
                // Each time the concurrent pool finishes running a task, another task is inserted
                await Promise.race(pool)
            }
        }
    }
    uploadFileChunks(fileChunks)

}
</script>
</html>
Copy the code

The back-end

Step 1. Install dependencies

NPM I [email protected] NPM I [email protected]Copy the code

Step 2. Interface implementation

const express = require('express')
const multiparty = require('multiparty')
const fs = require('fs')
const path = require('path')
const { Buffer } = require('buffer')
// Upload the final path of the file
const STATIC_FILES = path.join(__dirname, './static/files')
// Upload a temporary file path
const STATIC_TEMPORARY = path.join(__dirname, './static/temporary')
const server = express()
// Static file hosting
server.use(express.static(path.join(__dirname, './dist')))
// Interface for uploading slices
server.post('/upload'.(req, res) = > {
    const form = new multiparty.Form();
    form.parse(req, function(err, fields, files) {
        let filename = fields.filename[0]
        let hash = fields.hash[0]
        let chunk = files.chunk[0]
        let dir = `${STATIC_TEMPORARY}/${filename}`
        // console.log(filename, hash, chunk)
        try {
            if(! fs.existsSync(dir)) fs.mkdirSync(dir)const buffer = fs.readFileSync(chunk.path)
            const ws = fs.createWriteStream(`${dir}/${hash}`)
            ws.write(buffer)
            ws.close()
            res.send(`${filename}-${hash}Section uploaded successfully)}catch (error) {
            console.error(error)
            res.status(500).send(`${filename}-${hash}Section upload failed ')}})})// Merge slicing interfaces
server.get('/merge'.async (req, res) => {
    const { filename } = req.query
    try {
        let len = 0
        const bufferList = fs.readdirSync(`${STATIC_TEMPORARY}/${filename}`).map(hash= > {
            const buffer = fs.readFileSync(`${STATIC_TEMPORARY}/${filename}/${hash}`)
            len += buffer.length
            return buffer
        });
        // Merge files
        const buffer = Buffer.concat(bufferList, len);
        const ws = fs.createWriteStream(`${STATIC_FILES}/${filename}`)
        ws.write(buffer);
        ws.close();
        res.send('Slice merge completed');
    } catch (error) {
        console.error(error);
    }
})

server.listen(3000._= > {
    console.log('http://localhost:3000/')})Copy the code

Other implementations

If you use Tencent Cloud or Ali Cloud file upload services, they provide NPM libraries, such as Tencent Cloud coS-Js-SDK-V5, which provides its own slicing related configurations

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

How to handle large files uploaded from the front end

background

Train of thought

implementation

The front end

Sample code repository

Step 1- Slice and merge slices

Step 2- Concurrency control

Step 3- Resumable

The back-end

Step 1. Install dependencies

Step 2. Interface implementation

Other implementations

How to handle large files uploaded from the front end

background

Train of thought

implementation

The front end

Sample code repository

Step 1- Slice and merge slices

Step 2- Concurrency control

Step 3- Resumable

The back-end

Step 1. Install dependencies

Step 2. Interface implementation

Other implementations

Related Posts

Getting the best of it, performance tuning and custom Hooks

Differences between Vue versions

Interview Summary (2)