Talk about front-end large file sharding upload

Front-end file uploading mode

Form + iframe

How it works: To submit a file through a form, set the encType of the form tag to multipart/form-data, and set method to POST.

Enctype attribute:

• Application/X-www-form-urlencoded all characters before sending (default)
• Multipart /form-data does not have character encoding and must be used when uploading files

• Text /plain Spaces are converted to “+” plus signs, but no special character encoding.

<form action="/formUpload" enctype="multipart/form-data" method="post" target="upload"> <input type="file" name="file" /> < form> <iframe name='upload' id='uploader' style="display: none;" ></iframe>Copy the code

Ajax + formData

FormData is used to carry file contents, and Ajax asynchronously uploads files.

const formData = new FormData() formData.append("file", file); $. Ajax ({url:'/formUpload', type:'POST', data:formData, processData:false, // Tell jquery not to process sent data contentType:false, / / tell jquery don't set the content-type request header succuss: () = > {}, error: () = > {}})Copy the code

Blob

The Blob **** object represents an immutable, raw data-like file object. Its data can be read in text or binary format, or converted to ReadableStream for data manipulation.

The File interface is based on Blob, inheriting the functionality of Blob and extending it to support files on the user’s system.

Blob.size

The size (in bytes) of the data contained in the Blob object.

Blob.type

A string that indicates the MIME type (image/ PNG, text/ HTML, and so on) of the data that the Blob object contains. If the type is unknown, the value is an empty string.

Blob.slice()

Returns a new Blob object containing the data in the specified range in the source Blob object. Large file slicing is similar to the slice method of arrays, which intercepts the contents of arrays.

See a Blob

Worker

Worker interface is a part of Web Workers API, which refers to a background task that can be created by script and can send and receive information to its creator during task execution. To create a Worker, simply call the Worker(URL) constructor, whose parameter URL is the script specified to handle some very time-consuming tasks.

In addition to worker, there are other Workers, such as Shared Workers, Service Workers, audio Workers and so on.

Worker()

Create a dedicated Web worker that executes only the script specified by the URL. It is also possible to use a Blob URL as a parameter.

const w = new Worker('./test-worker.js')
Copy the code

Worker.onmessage()

Receives data from the nearest outer object.

w.onmessage = event => {
  console.log(event.data)
}
Copy the code

Worker.postMessage()

Sends a message to the nearest outer object, which can be composed of any JavaScript object.

self.postMessage('666')
Copy the code

Self. ImportScripts ()

Other threads import JS

Details see the Worker

Webworker tutorial

File section

• Why slice it up?

If a file is too large, it will be very slow to upload, and it may fail in the middle of the process. Reuploading files is very bad for the user experience. Once files are sliced, they can be uploaded concurrently, much faster than before. And if you fail, you don’t have to upload what you’ve already uploaded.

• How do I slice a file?

Slice method can be used to slice the file. The slice size should not be too large, usually up to 50M.

• The relevant implementation is as follows

Const maxChunkSize = 52428800 // MaxChunkSum = math.ceil (file? .size / maxChunkSize) export const createFileChunks = async ({file, chunkSum, setProgress}) => { const fileChunkList = []; const chunkSize = Math.ceil(file? .size / chunkSum); let start = 0; for (let i = 0; i < chunkSum; i++) { const end = start + chunkSize; fileChunkList.push({ index: i, filename: file? .name, file: file.slice(start, end) }); start = end; } const result = await getFileHash({chunks: fileChunkList, setProgress}); fileChunkList.map((item, index) => { item.key = result; }); return fileChunkList; };Copy the code

Compute file hash

• Why compute a file hash?

The hash of the file is obtained by encrypting the file content. The file content corresponds to the hash one by one. When we modify the contents of the file, the hash changes. We hash to see if it’s the same file. By discriminating file hash, you can know which files or file slices have been uploaded.

• How do I hash a file?

First of all, we uploaded a large file, so it was time-consuming to hash the file. We all know that JS is single-threaded, and it is definitely inappropriate to use it to compute such a time-consuming task. There are two solutions, as follows

1. Use Worker to simulate JS multithreading to handle time-consuming tasks.

Self. ImportScripts (" https://cdn.bootcdn.net/ajax/libs/spark-md5/3.0.0/spark-md5.min.js "); Self. onmessage = event => {console.log('worker receives data: ',event.data) const chunks = event.data const spark = new self.SparkMD5.ArrayBuffer(); const appendToSpark = async file => new Promise(resolve => { const reader = new FileReader(); reader.readAsArrayBuffer(file); reader.onload = e => { console.log(e.target.result) spark.append(e.target.result); resolve(); }; }); let count = 0; const workLoop = async () => { while (count < chunks.length) { await appendToSpark(chunks[count].file); count++; if (count >= chunks.length){ self.postMessage(spark.end()) } } }; workLoop() }Copy the code

2. Compute the file hash during the browser’s idle time.

• The concrete implementation is as follows

export const getFileHash = async ({chunks, setProgress}) => new Promise(resolve => { const spark = new sparkMD5.ArrayBuffer(); const appendToSpark = async file => new Promise(resolve => { const reader = new FileReader(); reader.readAsArrayBuffer(file); reader.onload = e => { spark.append(e.target.result); resolve(); }; }); let count = 0; Const workLoop = async deadline => {const workLoop = async deadline => { While (count < chunks.length && deadline. TimeRemaining () > 1) {await appendToSpark(chunks[count].file); count++; setProgress(Number(((100 * count) / chunks.length).toFixed(2))); if (count >= chunks.length) resolve(spark.end()); } window.requestIdleCallback(workLoop); }; window.requestIdleCallback(workLoop); });Copy the code

• How can I quickly compute a file hash?

The purpose of calculating the MD5 value of a file is to determine whether the file exists or not. We can consider designing a sampling hash to sacrifice some hit ratio and improve efficiency. The design idea is as follows

1. The file was cut into 2M slices
2. The first and last slices all the content, the other slices take the first, middle and last three places 2 bytes each

3. The merged content computes MD5, which is called the shard Hash
4. The result of this hash, there is a small probability of miscalculation. If the samples sampled are all the same, but the unsampled content is different, it will cause misjudgment.

function createFileChunks({ file,setProgress }){ const fileChunkList = []; const chunkSum = Math.ceil(file? .size / maxChunkSize) const chunkSize = Math.ceil(file? .size / chunkSum); let start = 0; for (let i = 0; i < chunkSum; i++) { const end = start + chunkSize; if(i == 0 || i == chunkSum - 1) { fileChunkList.push({ index: i, filename: file? .name, file: file.slice(start, end), }); } else { fileChunkList.push({ index: i, filename: file? .name, file:sample(file,start,end), }); } start = end; } return fileChunkList; }; Function sample(file,start,end) {const pre = file.slice(start, const pre = file. start + 1024 * 2) const after = file.slice(end - 1024 * 2, end) const merge = new Blob([pre, after]) return merge }Copy the code

requestIdleCallback

Window. RequestIdleCallback () method inserts a function, this function will be called the browser idle period.

The page is drawn frame by frame, and the page is smooth when the number of frames drawn per second (FPS) reaches 60, below which the user feels stuck. 1S is 60 frames, so the time allotted to each frame is 1000/60 ≈ 16 ms.

• What does the browser do in one frame?

1. Handle user interactions
2. JS parsing execution
3. Start frame. Window size change, page roll away, etc
4. requestAnimationFrame(rAF)
5. layout
6. draw

That is, requestIdleCallback() is executed at the end of a 16ms frame, waiting until the page is drawn to see if there is any free time to execute. If so, execute; otherwise, do not execute. Do not manipulate the DOM in the requestIdleCallback since the DOM has already been drawn. This can cause layout and view redrawing to be recalculated. DOM manipulation is recommended in requestAnimationFrame.

var handle = window.requestIdleCallback(callback[, options])
const cb = ({didTimeout, timeRemaining()}) => {}
const op = {timeout: 3000}
Copy the code

• DidTimeout Whether the task times out
• TimeRemaining () The remaining time of the current frame

• Timeout This parameter is mandatory

Detailed see requestIdleCallback

requestAnimationFrame

Tell the browser window. RequestAnimationFrame () – you want to perform an animation, and required the browser until the next redraw calls the specified callback function to update the animation. This method takes as an argument a callback function that is executed before the browser’s next redraw.

RequestAnimationFrame differs from setTimeout or setInterval in that it is up to the system to determine when the callback should be executed, asking the browser to execute the callback before the next rendering. Regardless of the refresh rate of the device, the requestAnimationFrame interval follows the time it takes to refresh the screen once;

SetTimeout animation can be slow, because JS is single threaded. If the previous task is blocked, setTimeout will wait for the previous task to complete, causing a lag.

Detailed see requestAnimationFrame

Concurrent post

With promise.all (), all slices can be uploaded concurrently. Uploading all slices concurrently may result in the network request being completely occupied, and thus the success rate of the upload is greatly compromised. Control the concurrent number of slices uploaded to solve this problem.

1. Use mapLimit to control concurrency

See link for details

npm install --save async import mapLimit from 'async/mapLimit'; export const uploadChunks = ({ chunks, url, chunkSum, limit, setProgress }) => new Promise((resolve, reject) => { mapLimit( chunks, limit, ({index, key, filename, file}, cb) => { const formData = new FormData(); formData.append('slice', index); formData.append('guid', key); formData.append('slices', chunkSum); formData.append('filename', filename); formData.append('file', file); request({ url, data: formData, onProgress: e => { setProgress(parseInt(String((e.loaded / e.total) * 100)), index); }, cb }); }, (err, result) => {if (err) {message. error(' error ') reject(err) return} resolve(result)})});Copy the code

Section Upload Progress

The upload progress of slices is monitored through the XMLHttpRequest upload.onprogress event.

export const request = ({
  url,
  method = 'post',
  data,
  headers = {},
  cb = e => e,
  onProgress = e => e
}) => {
    const xhr = new XMLHttpRequest();
    xhr.withCredentials = true
    xhr.timeout = 1000 * 60 * 60
    xhr.upload.onprogress = onProgress;
    xhr.open(method, url);
    headers = Object.assign({}, headers, {'EDR-Token': Cookie.get('auth')})
    Object.keys(headers).forEach(key => xhr.setRequestHeader(key, headers[key])
    );
    xhr.send(data);
    xhr.onload = e => {
      cb(null, {
        data: JSON.parse(e.target.response)
      });
    };
    xhr.onerror = error => {
      cb(error)
    }
  }
Copy the code

Slice merge and resumable

When all slices are uploaded successfully, we will initiate a request to merge slices from the back end.

What if there is a fault or no network when the slice is halfway uploaded? Do you want to re-upload all the slices?

No, we’ll just upload the slices we haven’t uploaded yet. Record the uploaded information of slices through localStorage or ask the background whether the slices have been uploaded before uploading each time. The specific process is as follows

conclusion

1. The hash function is used to determine the uniqueness of the file and whether the file has been uploaded successfully.
2. File slices are sliced through blob.prototype. slice. Whether the slices are uploaded successfully can be judged in service processing or the uploaded slice information can be kept in the front end.

3. Computing file hash is a very time-consuming task, which can be handled through worker simulation of multi-threading, or requestIdleCallback in the browser’s idle time to handle this time-consuming task. In order to quickly compute the hash of the file, you can also use sampling.
4. The real-time upload progress of files can be handled through the upload.onprogress of XMLHttpRequest or monitored in real time through websocket.

5. When all slices are uploaded successfully, a merge request is initiated to merge all slices into one large file.