preface

At the beginning of the article in Zhihu, later feel zhihu platform is not suitable for me, so I came to dig gold.

Basic file uploading process

The front-end obtains the file selected by the user through input, puts it into FormData, sets the content-Type to multipart/form-data and sends it to the server. Server through cookie/token/… And so on, and then file/database processing operations.

I don’t think there’s much to talk about. Let’s just go to the code.

<input type="file" onChange={handleFileChange} ref={inputRef} multiple />

constfiles = inputRef.current? .files;// Get the array file, process the array, and get the file

const formData = new FormData();
formData.append('file', file);
formData.append('xx', xx);
// Take some information with you

const { code } = awaituploadFiles(formData); On the server SIDE I used Node Express and multer to handle multipart/form-data form data. Multer portalimport multer from 'multer';

const storage = multer.diskStorage({
  destination: (req, file, cb) = > {
    cb(null.'./dist')},filename: (req, file, cb) = > {
    cb(null.`The ${(+new Date()}${file.originalname}`)}});const upload = multer({ storage });
app.use(upload.single('file'));
// Req.file; If you want multiple files, that's array, for req.files

router.post('/upload'.async (req, res) => {
    const { file } = req;
    // to do sth.
    Fs.renamesync /fs.rename can be used to rename a directory
});
Copy the code

Drag and drop to upload

Drag-and-drop uploads use onDrop and onDragOver to block browser default events and then get the corresponding files.

<div onDrop={onDrop} onDragOver={onDragOver} ></div>

const onDragOver = (e: DragEvent | React.DragEvent<HTMLDivElement>) = > {
    e.preventDefault();
};
const onDrop = (e: DragEvent | React.DragEvent<HTMLDivElement>) = >{ e.preventDefault(); handleFileChange(e.dataTransfer? .files); };Copy the code

Large file upload

Large file upload may occur, upload time is too long, the interface limits the file size. Therefore, direct upload of large files is also very unfriendly, generally adopt the way of fragment upload to upload. Blob provides the slice method, and file inheriting bloB can also use slice to slice processing.

Processing process:

  • The front-end shards large files, and the fragment name is fileHash will talk about that lateraddThe subscript
  • To prevent total occupancytcpResources, I’m using 4, 4 uploads
  • At the end, send another merge request
  • Upon request, the server merges the previous shards into a whole file and deletes the shards
const handleUpload = () = > {
    chunks = [];

    const file: File = files.current[dataIndex];
    // Get the corresponding file file
    let start = 0, end = BIG_FILE_SIZE;
    
    // const BIG_FILE_SIZE = 1024 * 1024 * 2;
    while(true) {
        const part: Blob = file.slice(start, end);
        if (part.size) {
            chunks.push(part);
        } else {
            break;
        }
        start += BIG_FILE_SIZE;
        end += BIG_FILE_SIZE;
    };
    
    // worker! .postMessage({ chunkList: chunks });
    // Use webworker to get hash, more on that later
    return;
};

const partUpload = async (start: number) = > {const uploadArr = [];
    let restReq = MAX_FILE_TCP;
    // MAX_FILE_TCP = 4; Initiate four connections simultaneously
    let index = start;
    while (restReq) {
        if (index >= chunkCount) {
        // chunkCount is the length of the chunk, i.e. how many fragments
            break;
        };
        
        // const blobPart = `${hash}-${index}`;
        // if (hashData[hash] && hashData[hash][blobPart]) 
        / / {
        // index++;
        // continue;
        // };
        // The comment part is, the breakpoint continues part of the code, can skip
        
        const formData = new FormData();
        formData.append('file', chunks[index]);
        formData.append('xx', xx);
        
        const uploadFunc = () = > {
            return new Promise(async (res) => {
                const { code } = await uploadPart(formData);
                res(code);
            });
        };
        
        uploadArr.push(uploadFunc);
        index++;
        restReq--;
    };
    
    const result = await Promise.all(uploadArr.map(v= > v()));
    
    result.map((v) = > {
      if (v === 0) {
        console.log('Upload clip successful');
      } else {
        throw new Error('Upload failed');
      }
      return null;
    });
    
    if (index < chunkCount) {
      partUpload(index);
    } else {
        const params = {
            // sth.
        };
        const {code} = await partMerge(params);
        // Send the merge request
        // todo code sth.}};Copy the code

On the server side, I name the file according to the corresponding hash and subscript, namely static/hash/hash-i. Rename to modify the file & path and merge the file with pipe.

router.post('/upload_part'.(req, res) = > {
    try {
        const { hash, index } = req.body;
        const { file } = req;
        const sourcePath = path.join(__dirname, file.path);
        const destPath = path.join(__dirname, `xxxx/${hash}/${hash}-${index}`);
        fs.renameSync(sourcePath, destPath);
        return res.json({code: 0});
    } catch(e) {
        return res.json({code: 1.msg: e}); }}); router.post('/merge_part'.(req, res) = > {
    try {
        const destPath = 'xxx/yyy/zzz/a.png';
        const writePath = fs.createWriteStream(destPath);
        // Where is the final merge result stored
        const fileMerge = (i: number) = > {
            const blobPath = `xxx/part/${hash}/${hash}-${i}`;
            const blobFile = fs.createReadStream(blobPath);
            blobFile.on("end".async (err) => {
                if (err) {
                    return res.json({code: 1.msg: err});
                };
                fs.unlink(blobPath);
                // Delete the fragment
                if (i + 1 < chunkCount) {
                    fileMerge(i + 1);
                } else {
                    fs.rmdirSync(`xxx/part/${hash}`);
                    // Delete the folder
                    // Database todo
                    return res.json({ code: 0}); }}); blobFile.pipe(writeStream, {end: false });
        };
        fileMerge(0);
    } catch(e) {
        return res.json({code: 1.msg: e}); }});// omit unnecessary content
Copy the code

Breakpoint continuingly

If the client abnormally interrupts the upload process, the next time you upload a large file in fragments, it is more friendly to skip the uploaded fragments. So the question is, how do I skip those files? ${hash}-${I}-${hash}-${I}-${hash}-${I}-${hash} For this type of data store, there are generally two methods:

  • The front store
  • Server storage

Front-end storage, generally using localStorage, not recommended. Because if the user clears the cache, or logs in on another device, it won’t work.

On the server side, the corresponding successfully uploaded fragment name is returned. Because the file hash-i corresponding to the userId is unique. Node uses readdirSync/readdir to read the file name.

So this is the front end that I mentioned earlier.

const blobPart = `${hash}-${index}`;
if (hashData[hash] && hashData[hash][blobPart]) 
{
    // hashData is the data returned by the server. If the fragment exists, skip it
    // For details, see the previous section
    index++;
    continue;
};
Copy the code

WebWoker get hash

All that remains is to get the hash of the file. You are advised to use spark-MD5 to generate the hash of the file. Since the process of generating hash is time-consuming, I used webworker to calculate hash.

The basics of Webwoker portal

Create webWoker with Blob & url.createObjecturl. Create webWoker with Blob & url.createObjecturl.

<script id="worker" type="app/worker">
self.importScripts("https://cdn.bootcss.com/spark-md5/3.0.0/spark-md5.min.js");

self.onmessage = function(e) {
    var spark = new self.SparkMD5.ArrayBuffer();
    var chunkList = e.data.chunkList;
    var count = 0;
    var next = function(index) {
        var fileReader = new FileReader();
        fileReader.readAsArrayBuffer(chunkList[index]);
        fileReader.onload = function(e) {
            count ++;
            spark.append(e.target.result);
            if (count === chunkList.length) {
              self.postMessage({
                hash: spark.end()
              });
            } else{ loadNext(count); }}; }; next(0);
};
</script>
Copy the code

Then call createObjectURL in the hook function to generate the DOMString as the Worker argument.

const newWebWorker = () = > {
  const textContent = document.getElementById('worker')! .textContent;const blob = new Blob([textContent as BlobPart]);
  createURL = window.URL.createObjectURL(blob);
  worker = new Worker(createURL);
  worker.onmessage = (e) = > {
    hash = e.data.hash;
    // Get hash here
    // todo sth.
    partUpload(0);
    // Upload the file fragment
  };
};
Copy the code

RevokeObjectURL revokeObjectURL revokeObjectURL revokeObjectURL

File download

There are many ways to download files: form forms, open, direct A tags, blob, etc. I use BLOB to download files, mainly considering that authentication can be carried out & can download various types of files and so on.

Process is

  • Server returnBlob
  • The front end goes through herecreateObjectURLgenerateDOMString
  • Set up theaOf the labelhrefIt then executes its click event
{
    url
      ? <a ref={aRef} style={{ display: 'none'}}href={url} download={data.file_name} />
      : null
}
// Download is the name of the file you downloaded from

const onDownload = async() = > {const blob = await getFileBlob({ id: data.id, type: data.type });
    const url = window.URL.createObjectURL(blob);
    setUrl(url);
};

useEffect(() = > {
    if(url) { aRef.current? .click();window.URL.revokeObjectURL(url);
        setUrl(' ');
    }
}, [url]);

export const getFileBlob = async (params: GetFileBlob) => {
  const { data } = await api.get<Data>('/get_file_blob', {
    params,
    responseType: 'blob'
    // responseType needs to set blob
  });
  return data;
};
Copy the code

Node here reads the file with createReadStream and sets content-Type and Content-disposition.

router.get('/get_file_blob'.async (req, res) => {
    // todo sth.
    const destPath = 'xxx/xxx';
    const fileStream = fs.createReadStream(destPath);
    res.setHeader('Content-Type'.'application/octet-stream');
    res.setHeader('Content-Disposition'.`attachment; filename=whatareudoing`);
    // The previous a tag has been set to download, filename here does not affect
    fileStream.on("data".(chunk) = > res.write(chunk, "binary"));
    fileStream.on("end".() = > res.end());
});
Copy the code

The progress bar

The ProgressEvent bar is called ProgressEvent. If it is a breakpoint continuation, the section is first iterated to see if any have been uploaded, and then an initial progress is obtained.

const onUploadProgress = (event: ProgressEvent) = > {
    const progress = Math.floor((event.loaded / event.total) * 100);
    // todo sth.
};

export const uploadFiles = async ({ formData, onUploadProgress }: UploadFiles) => {
  const { data } = await api.post<Data>('/upload', formData, { onUploadProgress });
  / /...
};
Copy the code

Get the thumbnail of the video file

If it is a video file, you will need to obtain the thumbnail of the video file in many cases. There are two ways. One is to get it in the front end, using canvas. The other is for the server to fetch.

I’m using Node and I’m using FFmPEG to grab the first frame of the video and save it as a thumbnail.

const getVideoSceenshots = (videoPath: string, outPutPath: string, outPutName: string) = > {
  const process = new ffmpeg(videoPath);
  process.then((video) = > {
    video.fnExtractFrameToJPG(outPutPath, {
      frame_rate: 1.number: 1.file_name: outPutName
    }, error= > {
      if (error) {
        console.log('error: ' + error)
      }
    })
  }, err= > {
    console.log('Error: ' + err)
  })
};

// videoPath video file path, outPutPath thumbnail outPutPath, outPutName outPutName
Copy the code