Manage large file uploads and resumable uploads based on JS

preface

File upload is a common problem in the development process. You may be able to achieve the relevant functions, but after the completion of the code implementation is not a little “overwhelmed”? Do you really know anything about file uploads? How to upload large files and power off the continuation of it, the common format of the front and back end communication, file upload progress control, how to achieve the server? Next, let’s start the hand touch series of learning!! If there are shortcomings, hope not stingy advice, next according to the figure for study and discussion

All set, let’s go!!

The front-end structure

• The page display

• Project depend on

Back-end architecture (Node + Express)

• The directory structure

  

• Simple encapsulation of Axios

  let instance = axios.create();
  instance.defaults.baseURL = 'http://127.0.0.1:8888';
  instance.defaults.headers['Content-Type'] = 'multipart/form-data';
  instance.defaults.transformRequest = (data, headers) = > {
      const contentType = headers['Content-Type'];
      if (contentType === "application/x-www-form-urlencoded") return Qs.stringify(data);
      return data;
  };
  instance.interceptors.response.use(response= > {
      return response.data;
  });
  Copy the code

File uploads are generally done in two ways,FormDataAs well asBase64

Implement file upload based on FormData

 // Front-end code
    // Show the core code for upload based on ForData
    upload_button_upload.addEventListener('click'.function () {
            if (upload_button_upload.classList.contains('disable') || upload_button_upload.classList.contains('loading')) return;
            if(! _file) { alert('Please select the file to upload first ~~');
                return;
            }
            changeDisable(true);
            // Pass the file to the server: FormData
            let formData = new FormData();
            // Add fields according to background requirements
            formData.append('file', _file);
            formData.append('filename', _file.name);
            instance.post('/upload_single', formData).then(data= > {
                if (+data.code === 0) {
                    alert('The file has been uploaded successfully ~~, you can base on${data.servicePath}Access this resource ~~ ');
                    return;
                }
                return Promise.reject(data.codeText);
            }).catch(reason= > {
                alert('File upload failed, please try again later ~~');
            }).finally(() = > {
                clearHandle();
                changeDisable(false);
            });
        });
Copy the code

File upload based on BASE64

BASE64 Method

• The first step is to stream the file to BASE64, where you can encapsulate a method

  export changeBASE64(file) => {
     return new Promise(resolve= > {
      let fileReader = new FileReader();
      fileReader.readAsDataURL(file);
      fileReader.onload = ev= > {
          resolve(ev.target.result);
      };
    });
  };
  Copy the code

• The specific implementation

  upload_inp.addEventListener('change'.async function () {
          let file = upload_inp.files[0],
              BASE64,
              data;
          if(! file)return;
          if (file.size > 2 * 1024 * 1024) {
              alert('File uploaded cannot exceed 2MB~~');
              return;
          }
          upload_button_select.classList.add('loading');
          / / get Base64
          BASE64 = await changeBASE64(file);
          try {
              data = await instance.post('/upload_single_base64', {
              // encodeURIComponent(BASE64) to prevent the transmission process of special characters garbled, at the same time the back end needs to use decodeURIComponent to decode
                  file: encodeURIComponent(BASE64),
                  filename: file.name
              }, {
                  headers: {
                      'Content-Type': 'application/x-www-form-urlencoded'}});if (+data.code === 0) {
                  alert('Congratulations, the file uploaded successfully, you can based on${data.servicePath}Address to access ~~ ');
                  return;
              }
              throw data.codeText;
          } catch (err) {
              alert('Unfortunately, file upload failed, please try again later ~~');
          } finally {
              upload_button_select.classList.remove('loading'); * *}}); 支那Copy the code

In the example above, the back end receives the file from the front end and generates a random name for it and saves it, but some companies do this in the front end and generates the name and sends it to the back end. Let’s implement this function

The front-end generates the file name and passes it to the back-end

Here we need to use the plug-in SparkMD5 mentioned above. I will not elaborate on how to use it. Please refer to the document

• Encapsulates a method for reading a file stream

  const changeBuffer = file= > {
      return new Promise(resolve= > {
          let fileReader = new FileReader();
          fileReader.readAsArrayBuffer(file);
          fileReader.onload = ev= > {
              let buffer = ev.target.result,
                  spark = new SparkMD5.ArrayBuffer(),
                  HASH,
                  suffix;
              spark.append(buffer);
              // Get the file name
              HASH = spark.end();
              // Get the suffix name
              suffix = /\.([a-zA-Z0-9]+)$/.exec(file.name)[1];
              resolve({
                  buffer,
                  HASH,
                  suffix,
                  filename: `${HASH}.${suffix}`
              });
          };
      });
    };
  Copy the code

• Upload the server code

  upload_button_upload.addEventListener('click'.async function () {
          if (checkIsDisable(this)) return;
          if(! _file) { alert('Please select the file to upload first ~~');
              return;
          }
          changeDisable(true);
          // Generate the HASH name of the file
          let {
              filename
          } = await changeBuffer(_file);
          let formData = new FormData();
          formData.append('file', _file);
          formData.append('filename', filename);
          instance.post('/upload_single_name', formData).then(data= > {
              if (+data.code === 0) {
                  alert('The file has been uploaded successfully ~~, you can base on${data.servicePath}Access this resource ~~ ');
                  return;
              }
              return Promise.reject(data.codeText);
          }).catch(reason= > {
              alert('File upload failed, please try again later ~~');
          }).finally(() = > {
              changeDisable(false);
              upload_abbre.style.display = 'none';
              upload_abbre_img.src = ' ';
              _file = null;
          });
      });
  Copy the code

Upload Progress Control

This function is relatively simple. The request library used in this paper is AXIOS, and the progress control is mainly realized based on the onUploadProgress function provided by AXIOS. Here is the realization principle of this function

• Listening to the XHR. Upload. Onprogress

  

• The object obtained after the file is uploaded

  

• The specific implementation

  (function () {
      let upload = document.querySelector('#upload4'),
          upload_inp = upload.querySelector('.upload_inp'),
          upload_button_select = upload.querySelector('.upload_button.select'),
          upload_progress = upload.querySelector('.upload_progress'),
          upload_progress_value = upload_progress.querySelector('.value');
  
      // Verify that the state is operable
      const checkIsDisable = element= > {
          let classList = element.classList;
          return classList.contains('disable') || classList.contains('loading');
      };
  
      upload_inp.addEventListener('change'.async function () {
          let file = upload_inp.files[0],
              data;
          if(! file)return;
          upload_button_select.classList.add('loading');
          try {
              let formData = new FormData();
              formData.append('file', file);
              formData.append('filename', file.name);
              data = await instance.post('/upload_single', formData, {
                  // The file upload callback function xhr.upload.onprogress
                  onUploadProgress(ev) {
                      let {
                          loaded,
                          total
                      } = ev;
                      upload_progress.style.display = 'block';
                      upload_progress_value.style.width = `${loaded/total*100}% `; }});if (+data.code === 0) {
                  upload_progress_value.style.width = ` 100% `;
                  alert('Congratulations, the file uploaded successfully, you can based on${data.servicePath}Access the file ~~ ');
                  return;
              }
              throw data.codeText;
          } catch (err) {
              alert('Unfortunately, file upload failed, please try again later ~~');
          } finally {
              upload_button_select.classList.remove('loading');
              upload_progress.style.display = 'none';
              upload_progress_value.style.width = ` 0% `; }}); upload_button_select.addEventListener('click'.function () {
          if (checkIsDisable(this)) return; upload_inp.click(); }); }) ();Copy the code

Large file upload

Large file upload generally adopt the way of slicing to upload, so we can improve the speed of file upload, front end to get the document flow after slicing, and then transfer of communication with the back-end, general will combine a breakpoint after the transfer, then the back-end generally provide three interfaces, the first interface to get already uploaded slice information, the second interface will front slice file transmission, The third interface is to tell the back end to merge files after uploading all slices

• Slicing can be divided into fixed quantity and fixed size. Here we combine the two

  // Implement file slice processing "fixed number & fixed size"
  let max = 1024 * 100,
      count = Math.ceil(file.size / max),
      index = 0,
      chunks = [];
  if (count > 100) {
      max = file.size / 100;
      count = 100;
  }
  while (index < count) {
      chunks.push({
      // The file file itself has the slice method, as shown below
          file: file.slice(index * max, (index + 1) * max),
          filename: `${HASH}_${index+1}.${suffix}`
      });
      index++;
  }
  Copy the code

• Send to the server

  chunks.forEach(chunk= > {
      let fm = new FormData;
      fm.append('file', chunk.file);
      fm.append('filename', chunk.filename);
      instance.post('/upload_chunk', fm).then(data= > {
          if (+data.code === 0) {
              complate();
              return;
          }
          return Promise.reject(data.codeText);
      }).catch(() = > {
          alert('The current slice upload failed, please try again later ~~');
          clear();
      });
     });
  Copy the code

• File upload + power-off continuation + progress control

      upload_inp.addEventListener('change'.async function () {
          let file = upload_inp.files[0];
          if(! file)return;
          upload_button_select.classList.add('loading');
          upload_progress.style.display = 'block';
  
          // Get the HASH of the file
          let already = [],
              data = null,
              {
                  HASH,
                  suffix
              } = await changeBuffer(file);
  
          // Get the uploaded slice information
          try {
              data = await instance.get('/upload_already', {
                  params: {
                      HASH
                  }
              });
              if (+data.code === 0) { already = data.fileList; }}catch (err) {}
  
          // Implement file slice processing "fixed number & fixed size"
          let max = 1024 * 100,
              count = Math.ceil(file.size / max),
              index = 0,
              chunks = [];
          if (count > 100) {
              max = file.size / 100;
              count = 100;
          }
          while (index < count) {
              chunks.push({
                  file: file.slice(index * max, (index + 1) * max),
                  filename: `${HASH}_${index+1}.${suffix}`
              });
              index++;
          }
  
          // Upload successfully processed
          index = 0;
          const clear = () = > {
              upload_button_select.classList.remove('loading');
              upload_progress.style.display = 'none';
              upload_progress_value.style.width = '0%';
          };
          const complate = async() = > {// Control progress bar
              index++;
              upload_progress_value.style.width = `${index/count*100}% `;
  
              // When all slices have been uploaded successfully, we merge the slices
              if (index < count) return;
              upload_progress_value.style.width = ` 100% `;
              try {
                  data = await instance.post('/upload_merge', {
                      HASH,
                      count
                  }, {
                      headers: {
                          'Content-Type': 'application/x-www-form-urlencoded'}});if (+data.code === 0) {
                      alert('Congratulations, the file uploaded successfully, you can based on${data.servicePath}Access the file ~~ ');
                      clear();
                      return;
                  }
                  throw data.codeText;
              } catch (err) {
                  alert('Slice merge failed, please try again later ~~'); clear(); }};// Upload each slice to the server
          chunks.forEach(chunk= > {
              // No need to upload already uploaded
              if (already.length > 0 && already.includes(chunk.filename)) {
                  complate();
                  return;
              }
              let fm = new FormData;
              fm.append('file', chunk.file);
              fm.append('filename', chunk.filename);
              instance.post('/upload_chunk', fm).then(data= > {
                  if (+data.code === 0) {
                      complate();
                      return;
                  }
                  return Promise.reject(data.codeText);
              }).catch(() = > {
                  alert('The current slice upload failed, please try again later ~~');
                  clear();
              });
          });
      });
  Copy the code

Server code (large file upload + resumable breakpoint)

 // Upload large file slices & merge slices
    const merge = function merge(HASH, count) {
        return new Promise(async (resolve, reject) => {
            let path = `${uploadDir}/${HASH}`,
                fileList = [],
                suffix,
                isExists;
            isExists = await exists(path);
            if(! isExists) { reject('HASH path is not found! ');
                return;
            }
            fileList = fs.readdirSync(path);
            if (fileList.length < count) {
                reject('the slice has not been uploaded! ');
                return;
            }
            fileList.sort((a, b) = > {
                let reg = /_(\d+)/;
                return reg.exec(a)[1] - reg.exec(b)[1];
            }).forEach(item= > {
                !suffix ? suffix = /\.([0-9a-zA-Z]+)$/.exec(item)[1] : null;
                fs.appendFileSync(`${uploadDir}/${HASH}.${suffix}`, fs.readFileSync(`${path}/${item}`));
                fs.unlinkSync(`${path}/${item}`);
            });
            fs.rmdirSync(path);
            resolve({
                path: `${uploadDir}/${HASH}.${suffix}`.filename: `${HASH}.${suffix}`
            });
        });
    };
    app.post('/upload_chunk'.async (req, res) => {
        try {
            let {
                fields,
                files
            } = await multiparty_upload(req);
            let file = (files.file && files.file[0]) || {},
                filename = (fields.filename && fields.filename[0) | |"",
                path = ' ',
                isExists = false;
            // Create a temporary directory for slices
            let [, HASH] = /^([^_]+)_(\d+)/.exec(filename);
            path = `${uploadDir}/${HASH}`; ! fs.existsSync(path) ? fs.mkdirSync(path) :null;
            // Store slices in a temporary directory
            path = `${uploadDir}/${HASH}/${filename}`;
            isExists = await exists(path);
            if (isExists) {
                res.send({
                    code: 0.codeText: 'file is exists'.originalFilename: filename,
                    servicePath: path.replace(__dirname, HOSTNAME)
                });
                return;
            }
            writeFile(res, path, file, filename, true);
        } catch (err) {
            res.send({
                code: 1.codeText: err }); }}); app.post('/upload_merge'.async (req, res) => {
        let {
            HASH,
            count
        } = req.body;
        try {
            let {
                filename,
                path
            } = await merge(HASH, count);
            res.send({
                code: 0.codeText: 'merge success'.originalFilename: filename,
                servicePath: path.replace(__dirname, HOSTNAME)
            });
        } catch (err) {
            res.send({
                code: 1.codeText: err }); }}); app.get('/upload_already'.async (req, res) => {
        let {
            HASH
        } = req.query;
        let path = `${uploadDir}/${HASH}`,
            fileList = [];
        try {
            fileList = fs.readdirSync(path);
            fileList = fileList.sort((a, b) = > {
                let reg = /_(\d+)/;
                return reg.exec(a)[1] - reg.exec(b)[1];
            });
            res.send({
                code: 0.codeText: ' '.fileList: fileList
            });
        } catch (err) {
            res.send({
                code: 0.codeText: ' '.fileList: fileList }); }});Copy the code

conclusion

To sum up, this is my summary of file uploading. My ability is limited. If there are any mistakes, please kindly advise me.

Husband learning must be static also, only to learn also, not learn beyond wide only, not beyond into learning. Slow sex can not stimulate the essence, risk impetuous can not cure sex. Year and time chi, meaning and day go, then withered