Hello everyone, I am Han Cao 🌿, a grass code ape 🐒 who has been working for one and a half years. If you like my article, you can follow ➕ and give a thumbs up. Please grow up with me

“This is the 18th day of my participation in the First Challenge 2022. For details: First Challenge 2022.”

background

Hello everyone, I am hancao 🌿, long time no see, at this time you may ask:

Clearly recently see you crazy hair “neat structure of the way” article, how long time no see?

Although it is true to say that, I really haven’t written some articles in line with my own style or design ideas for a long time, and the topic of this article is also related to the requirement of uploading large files that I developed last week, and the content is as follows:

File size limit 2G, 100, batch upload, maximum concurrent 5, and support the cancellation of upload, as well as display upload progress.

This requirement may seem trivial, mainly because of the large size and number of files, but it is a relatively basic requirement. However, I want to propose a more powerful and more general file upload capability, namely:

More than the number of
Large volume of
Multiple concurrent
Supports breakpoint continuation

General file upload capability ~

And the whole process is certainly not overnight, IT is my first time to do this, encountered a lot of strange questions, some I still need to go to the W3C document, you can also participate in the discussion, is also I leave readers to think:

Locally uploadedFileThe object of thewebkitRelativePathWhy there are no propertiespathorURLinformation

This might start with privacy and security and file specifications

After the file was uploadedFileIn the objectuidProperties whereas thisuid 在 MDNAs well asW3C 的 FileIt’s not even in the documentation, so thisuidIs how to come, in each browser compatibility?

If you are familiar with the above questions, please refer to ☀️. Here are also some reference links for you:

fs.spec.whatwg.org/
w3c.github.io/FileAPI/

I will publish at least three articles on file uploads, in order:

Implement the basic requirements mentioned above
Principle and optimization of breakpoint continuation
Implementation of follow-on capabilities

This article is about how I implement the basic requirements mentioned above, and I need to ensure their scalability and versatility in order to further expand capabilities.

If there is a problem with my implementation, please also give advice, exchange and discuss together to better progress ~

The early stage of the design

The design here is the general direction, and the design of the code should also belong to the preliminary design, but for the readability of the article, I have broken it down to the following section of coding implementation. If there is a problem, please peaceful discussion, after all, I am the first time to do ~

To my shame, I started out with a big plan to accomplish all my goals, so I sketched this:

The figure above is a simple list of the capabilities I want to implement and how the business layer communicates with the logical layer.

Tip: Actually, my description is not accurate enough. View refers to the consumer of upload capability, which belongs to the business layer. The logical layer refers to instances of file upload objects.

Next, I will continue to talk from the two directions of communication mode and general ability

In the design and implementation process reference:

Bytedance Interviewer: Please implement a large file upload and resumable breakpoint
Bytedance interviewer, I also implemented large file upload and resumable breakpoint

Communication mode

Now THAT I have completed the basic functions, the overall communication mode is basically the same as the one in the above picture:

Consumer -> Capability provider: The consumer instantiates the file upload capability through the constructor parameters
Capability provider -> Consumer: The provider of file upload capability invokes consumer-injected methods at key nodes

You might wonder, what do I define as a critical node?

Upload Progress Changes
File uploaded successfully
Failed to upload files.
All files are uploaded

Namely PROGRESS, SUCCESS, FAIL and END, the method of consumer injection is also used to process the information generated by these key nodes.

Ability to take apart

File fragmentation

Leaving aside the principle of file sharding, which we’ll talk about next time, one of the things I want to add here is the ability to smartly shard large files based on file size, number of concurrent files, number of files, etc.
The hash computation
To generate hash values based on file contents, there are two methods for large files:
- Full file content calculationhashThe value is too longweb-workerorrequestIdleCallbackDon’t blockThe UI threadAffecting User operations
- Sample the file contents and generatehashValue, which can greatly improve the efficiency, but will affect the accuracy of the judgment of the existence of the resumable file
Upload progress

If large file fragments are used, the upload progress of a single file must be calculated based on the hash calculation
Concurrency control

Limit the number of concurrent requests, avoid the situation of massive requests blasting the interface, and schedule requests intelligently
Breakpoint continuingly

Support file upload pause, recovery
There must be more to it than I thought…

Without further ado, the next chapter will focus on the thinking and details of code implementation.

coded

The birth of the class

My first idea is to separate it from the view so that its capabilities can be used under different “skins”, as shown below:

So the first step is to create a new class:

class ZetaUploader {
  constructor(){}}Copy the code

Then we went on to think that we had written so many capabilities in our pre-design-capability Disassembly section, and did it in a common module:

Determine whether the consumer file is fragmented
Determine whether the consumer wants to enable breakpoint continuation
.

The logic for supporting and not supporting file sharding is completely inconsistent. Encapsulating so much judgment logic in a class with the same name but different implementation logic is extremely painful and will greatly increase the use and maintenance cost, so I decided to break it down:

We use the long life of nine children, different traditional concept, decided to ZetaUploader as a base class, provide basic capabilities and internal variables, to produce his different subclasses, and this time, I want to work with you to achieve is BasicZetaUploader, provide the ability to upload large files concurrently:

Large file upload
Concurrency control
Upload status response (i.e. PROGRESS, SUCCESS, FAIL, END)

ZetaUploader is a base class that requires all subclasses. Here are some of the variables and methods I want to put in the base class:

Variables:

fileState

Save the number of files that were successfully uploaded, failed uploaded, and canceled uploaded during the whole upload process
progressEvent

As mentioned above, consumer-injected methods are used to notify consumers of changes in processing upload progress at key points
concurrency

The number of concurrent uploads
xhrMap

An uploaded XMLHttpRequest instance that can be used to abort the XHR request.
unUploadFileList

List of files that have not been uploaded and are waiting to be uploaded
xhrOptions

Basic information to initiate a request, including URL, Method, header, and getXhrDataByFile. GetXhrDataByFile is the method that generates the request data from file

Methods:

isRequestSuccess

This method is used to determine whether the request was successful, and since it does not rely on the ZetaUploader instance, I define it as static

static isRequestSuccess(progressEvent) {
  return String(progressEvent.target.status).startsWith('2');
}
Copy the code

The syntax for onload is progressEvent (progressEvent).

XMLHttpRequest.onload = callback;
Copy the code

Where callback is the function to be executed when the request completes successfully. It receives a ProgressEvent object as its first argument, and the value (context) of this is the same as the XMLHttpRequest for this callback.

The complete base class code is as follows:

class ZetaUploader {
  fileState = {
    failCount: 0.successCount: 0.abortCount: 0
  }
  /* progressEvent arguments * @params {Object} [params] * @params {ENUM} [params?.state] FAIL PROGRESS SUCCESS END * @params {Info} [params?.info] * @params {File} [info?.file] * @params {Number} [info?.progress] * @params {FileState} [info?.fileState] */
  progressEvent = () = >{}; concurrency =1;
  xhrMap = null;
  unUploadFileList = [];
  /* * @params {Object} [xhrOptions] * @params {String} [params.url] * @params {String} [params.method] * @params {Object}  [params.headers] * @params {Function} [params.getXhrDataByFile] */
  xhrOptions = null; 

  static isRequestSuccess(progressEvent) {
    return String(progressEvent.target.status).startsWith('2');
  }

  constructor(progressEvent, fileList, concurrency, xhrOptions) {
    const basicXhrOptions = {
      url: ' '.method: 'post'.headers: [].getXhrDataByFile: file= > {
        const formData = new FormData();
        formData.append('file', file);
        returnformData; }};// BASIC ATTRS
    this.progressEvent = progressEvent;
    this.concurrency = concurrency;
    this.xhrOptions = Object.assign(basicXhrOptions, xhrOptions);
    // COMPUTED ATTRS
    this.unUploadFileList = fileList;
    this.xhrMap = new Map();
  }
}
Copy the code

The factory method

At this point, our subclass Basiczea Uploader is about to start writing, and now it looks like this:

class BasicZetaUploader extends ZetaUploader {
  constructor(progressEvent, fileList, concurrency, xhrOptions) {
    super(progressEvent, fileList, concurrency, xhrOptions); }}Copy the code

We need a method factory for batch output methods, parameters are File, xhrOptions, onProgress, and parameters are as follows:

The file object
Requested configuration information
Handling methods when progress changes

To create a request method for each file. Since the factory that generates the XHR request relies on onProgress, I’ll cover this first

OnProgress factory

The use of onProgress is as follows:

XMLHttpRequest.onprogress = function (event) {
  event.loaded;
  event.total;
};
Copy the code

Parameters as follows:

event.loadedThe amount of data transferred
event.totalThe total amount of data

I write a progressFactory method, which is also a factory method, that generates onProgress via file:

progressFactory(file) {
  return e= > {
    this.progressEvent('PROGRESS', {
      file,
      progress: parseInt(String((e.loaded / e.total) * 100))}); }; }Copy the code

Request the factory

Now that we have the onProgress factory, we can start a serious requestFactory that initializes an XMLHttpRequest and Promise, but here’s the trick:

Promise’s Reject and resolve are controlled by XHR’s callback

There is little else to explain, just a process of creating a new XHR and adding callbacks to it. The overall code is as follows:

requestFactory(file, xhrOptions, onProgress) {
  const { url, method, headers, getXhrDataByFile } = xhrOptions;
  const xhr = new XMLHttpRequest();
  xhr.open(method, url);
  Object.keys(headers).forEach(key= > xhr.setRequestHeader(key, headers[key]));
  let _resolve = () = >{};let _reject = () = >{}; xhr.onprogress = onProgress; xhr.onload =e= > {
    // Need to add response judgment
    if (ZetaUploader.isRequestSuccess(e)) {
      this.fileState.successCount++;
      this.progressEvent('SUCCESS', {
        file
      });
      _resolve({
        data: e.target.response
      });
    } else {
      this.fileState.failCount++;
      this.progressEvent('FAIL', { file }); _reject(); }}; xhr.onerror =() = > {
    this.fileState.failCount++;
    this.progressEvent('FAIL', {
      file
    });
    _reject();
  };
  xhr.onabort = () = > {
    this.fileState.abortCount++;
    _resolve({
      data: null
    });
  };
  const request = new Promise((resolve, reject) = > {
    _resolve = resolve;
    _reject = reject;
    xhr.send(getXhrDataByFile(file));
  });
  return {
    xhr,
    request
  };
}
Copy the code

Concurrency control

Here the previous unUploadFileList and xhrMap come in handy:

Whether a request can be made:

Files to be uploaded and the number of requests being uploaded in xhrMap is less than the maximum number of concurrent requests

unUploadFileList.length > 0 && xhrMap.size < concurrency
Copy the code

Whether to end uploading:

There are no files to upload and no ongoing requests in xhrMap

isUploaded() {
  return this.unUploadFileList.length === 0 && this.xhrMap.size === 0;
}
Copy the code

Complete code:

upload() {
  const { concurrency, xhrMap, unUploadFileList, xhrOptions, fileState } = this;
  return new Promise(resolve= > {
    const run = async() = > {while (unUploadFileList.length > 0 && xhrMap.size < concurrency) {
        const file = unUploadFileList.shift();
        const { xhr, request } = this.requestFactory(file, xhrOptions, this.progressFactory(file));
        xhrMap.set(file, xhr);
        request.finally(() = > {
          xhrMap.delete(file);
          if (this.isUploaded()) {
            resolve();
            this.progressEvent('END', {
              fileState
            });
          } else{ run(); }}); }}; run(); }); }Copy the code

Cancel the upload

Abort uploads abort single file uploads abort incomplete file uploads abort single file uploads abort incomplete files abort

xhrInstance.abort();
Copy the code

If the request has already been issued, the xmlHttprequest.abort () method terminates the request. When a request is terminated, its readyState is set to xmlHttprequest.unsent (0) and the status of the request is set to 0.

The logic for canceling the upload is abort if the request is in xhrMap, and remove it from the array if it is in unUploadFileList.

abort(file) {
  if (this.xhrMap.has(file)) {
    this.xhrMap.get(file)? .abort(); }else {
    const fileIndex = this.unUploadFileList.indexOf(file);
    this.unUploadFileList.splice(fileIndex, 1); }}abortAll() {
  this.xhrMap.forEach(xhr= >xhr? .abort());this.unUploadFileList = [];
}
Copy the code

Upload again

If the file fails to upload, we can upload it again, which is very simple, just add the file to the unUploadFileList. Of course, when you upload it again, the upload may have already ended, so it will trigger the upload again.

redoFile(file) {
    this.fileState.failCount--;
    this.unUploadFileList.push(file);
    if (this.isUploaded()) {
      this.upload(); }}Copy the code

The complete code

class ZetaUploader {
  fileState = {
    failCount: 0.successCount: 0.abortCount: 0
  }
  /* progressEvent arguments * @params {Object} [params] * @params {ENUM} [params?.state] FAIL PROGRESS SUCCESS END * @params {Info} [params?.info] * @params {File} [info?.file] * @params {Number} [info?.progress] * @params {FileState} [info?.fileState] */
  progressEvent = () = >{}; concurrency =1;
  xhrMap = null;
  unUploadFileList = [];
  /* * @params {Object} [xhrOptions] * @params {String} [params.url] * @params {String} [params.method] * @params {Object}  [params.headers] * @params {Function} [params.getXhrDataByFile] */
  xhrOptions = null; 

  static isRequestSuccess(progressEvent) {
    return String(progressEvent.target.status).startsWith('2');
  }

  constructor(progressEvent, fileList, concurrency, xhrOptions) {
    const basicXhrOptions = {
      url: ' '.method: 'post'.headers: [].getXhrDataByFile: file= > {
        const formData = new FormData();
        formData.append('file', file);
        returnformData; }};// BASIC ATTRS
    this.progressEvent = progressEvent;
    this.concurrency = concurrency;
    this.xhrOptions = Object.assign(basicXhrOptions, xhrOptions);
    // COMPUTED ATTRS
    this.unUploadFileList = fileList;
    this.xhrMap = new Map();
  }
}

class BasicZetaUploader extends ZetaUploader {
  constructor(progressEvent, fileList, concurrency, xhrOptions) {
    super(progressEvent, fileList, concurrency, xhrOptions);
  }

  progressFactory(file) {
    return e= > {
      this.progressEvent('PROGRESS', {
        file,
        progress: parseInt(String((e.loaded / e.total) * 100))}); }; }/* * @params {File} [file] * @params {xhrOptions} [xhrOptions] * @params {Function} [onProgress] */
  requestFactory(file, xhrOptions, onProgress) {
    const { url, method, headers, getXhrDataByFile } = xhrOptions;
    const xhr = new XMLHttpRequest();
    xhr.open(method, url);
    Object.keys(headers).forEach(key= > xhr.setRequestHeader(key, headers[key]));
    let _resolve = () = >{};let _reject = () = >{}; xhr.onprogress = onProgress; xhr.onload =e= > {
      // Need to add response judgment
      if (ZetaUploader.isRequestSuccess(e)) {
        this.fileState.successCount++;
        this.progressEvent('SUCCESS', {
          file
        });
        _resolve({
          data: e.target.response
        });
      } else {
        this.fileState.failCount++;
        this.progressEvent('FAIL', { file }); _reject(); }}; xhr.onerror =() = > {
      this.fileState.failCount++;
      this.progressEvent('FAIL', {
        file
      });
      _reject();
    };
    xhr.onabort = () = > {
      this.fileState.abortCount++;
      _resolve({
        data: null
      });
    };
    const request = new Promise((resolve, reject) = > {
      _resolve = resolve;
      _reject = reject;
      xhr.send(getXhrDataByFile(file));
    });
    return {
      xhr,
      request
    };
  }

  upload() {
    const { concurrency, xhrMap, unUploadFileList, xhrOptions, fileState } = this;
    return new Promise(resolve= > {
      const run = async() = > {while (unUploadFileList.length > 0 && xhrMap.size < concurrency) {
          const file = unUploadFileList.shift();
          const { xhr, request } = this.requestFactory(file, xhrOptions, this.progressFactory(file));
          xhrMap.set(file, xhr);
          request.finally(() = > {
            xhrMap.delete(file);
            if (this.isUploaded()) {
              resolve();
              this.progressEvent('END', {
                fileState
              });
            } else{ run(); }}); }}; run(); }); }isUploaded() {
    return this.unUploadFileList.length === 0 && this.xhrMap.size === 0;
  }

  abort(file) {
    if (this.xhrMap.has(file)) {
      this.xhrMap.get(file)? .abort(); }else {
      const fileIndex = this.unUploadFileList.indexOf(file);
      this.unUploadFileList.splice(fileIndex, 1); }}abortAll() {
    this.xhrMap.forEach(xhr= >xhr? .abort());this.unUploadFileList = [];
  }

  redoFile(file) {
    this.fileState.failCount--;
    this.unUploadFileList.push(file);
    if (this.isUploaded()) {
      this.upload(); }}}export {
  BasicZetaUploader
};

Copy the code

conclusion

This article ends here, and the next article in this series will, unsurprisingly, take you through the principles and implementation of file sharding and breakpoint continuation.

Finally, see you in the next article

✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨

I was born in the sky, longer than the sun;

I fly on the wind, never far away.

✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨

Your support “like ➕ attention” is my continuous power, you can add my wechat: HancAO97, invite you to join the group, learn and communicate together, become a better front-end engineer ~

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

“Big file upload from scratch” how am I still writing about this in 2022…

background

The early stage of the design

Communication mode

Ability to take apart

coded

The birth of the class

The factory method

OnProgress factory

Request the factory

Concurrency control

Cancel the upload

Upload again

The complete code

conclusion

“Big file upload from scratch” how am I still writing about this in 2022…

background

The early stage of the design

Communication mode

Ability to take apart

coded

The birth of the class

The factory method

OnProgress factory

Request the factory

Concurrency control

Cancel the upload

Upload again

The complete code

conclusion

Related Posts

图 图 React source code – Fiber tree structure (contrast update) – one of the most central interpretation of fiber in the whole network

Event bubbling and capturing

[luffy]_leetcode-409- longest palindrome string

图图 React source code – Fiber tree structure (contrast update) – one of the most central interpretation of fiber in the whole network