Large file uploads are often required when OSS is used. However, large file uploads often fail due to network reasons. As a result, you have to upload again after a long time of hard work. Ali Cloud OSS provides Multipart Upload and resumable Upload functions, which can divide the uploaded file into multiple data blocks (also known as parts in OSS) to Upload separately. After the Upload is completed, the INTERFACE of OSS can be called to combine these parts into an Object to achieve the effect of resumable Upload.

This application example ali Cloud has sorted it into the demo upload git: https://github.com/ali-sdk/ali-oss.

However, in this demo, normal uploads are only implemented, stopping, during the upload process, and restarting uploads if the page is refreshed. So let me show you how to solve the problem of refreshing and continuing the breakpoint.

The basic idea

Let’s start with the basic implementation idea. The principle of breakpoint upload is to split the file into multiple data blocks and upload them separately, then combine them into a single file. After uploading a part of the file, you can see that the fragments increase in the FILE management of OSS, and these fragments are the parts that have been uploaded successfully.

The first thing we need to do is get the accessKeyId, accessKeySecret, stsToken, Region, bucket, and endpoint information from the background. If your site is HTTP, you don’t need to write this parameter. But if your site is HTTPS, you need to fill in this parameter. Otherwise, uploading fails, because the default upload is HTTP. When your site is HTTPS, it will also add HTTPS in front of it, but aliyuncs.com has an extra port 80. To solve the access problem, you need to add the endpoint parameter and specify HTTPS.

With this information in hand, it is time to build OSS objects

Const client = new OSS({// here or using new oss.wrapper ({})) creds.AccessKeyId, accessKeySecret: creds.AccessKeySecret, stsToken: creds.SecurityToken, bucket, endpoint:"https://"+region+".aliyuncs.com" })Copy the code

There is a listUploads interface in the constructed object, and the status in the upload list can be obtained through “prefix”, which is actually the key we upload, for example, we want to upload to “oecom/cn/123.jpg”, The prefix is “oecom/cn/123.jpg”. So how do we know which file is going to be uploaded, here involves a key to refresh the breakpoint upload, that is to save the file upload progress to localstorage.

We use the file name and file size for MD5 encryption to store the upload progress as a key. As long as this key is found in the LocslStorage, it indicates that the upload has not been completed before

The specific implementation

Let’s start with the global variables we need

let OSS = OSS.Wrapper; / / oss object let uploadFileClient currentCheckpoint; // Let retryCount = 0; Const retryCountMax = 3; // The current number of retries. The OSS re-initiates the upload request after the upload fails due to network reasons. Var upload_oss = {fileKey: "", fileUrl: ""}Copy the code

Then we set up the OSS connection, which is our main method:

var applyTokenDo = function (func, inputFile) { var filename = inputFile[0].files[0].name; var fileSize = inputFile[0].files[0].size; var fileInfo = JSON.parse(localStorage.getItem(ossMd5(filename+fileSize))) var fileKey = ""; if(fileInfo){ var pos = fileInfo.name.lastIndexOf("."); var suffix = (pos >= 0) ? fileInfo.name.substring(0,pos) : fileInfo.name; fileKey = suffix if(fileKey){ fileKey = "? fileKey="+fileKey; }else{ fileKey = ""; } } var url = '/api/checkUploadFileInfo' + fileKey; $.get(url, function (cr) {upload_oss.filekey = cr.filekey; upload_oss.fileUrl = cr.fileUrl; Endpont cr.endpoint = "https://"+cr.region+".aliyuncs.com" var client = new OSS(cr); func(client, inputFile); }); };Copy the code

In this method, two parameters need to be passed, one is the function to upload the file and the other is the input object of the upload file domain. We can obtain the upload situation of the uploaded file from the localStorage here. If part of the uploaded file was uploaded before, the path of the uploaded file does not need to be generated again. In order to avoid file overwriting problems caused by duplicate file names, the background operation strategy is to use 30 random strings as file names. Therefore, if the file path stored in localStorage already exists, the file path is extracted and sent to the background. If the background finds this parameter, the parameter is directly returned. We’ll talk more about what’s stored in this object.

The preparatory work in front is basically done, the following is the main upload and upload detection:

var uploadFile = function (client, inputFile) { var file = inputFile[0].files[0]; if (! uploadFileClient || Object.keys(uploadFileClient).length === 0) { uploadFileClient = client; } var fileName = file.name; var fileSize = file.size; var pos = fileName.lastIndexOf("."); var suffix = (pos >= 0) ? fileName.substring(pos) : ""; PartSize: 500 * 1024,// Fragment size timeout: 60000}; upload_oss.fileKey += suffix; upload_oss.fileUrl += suffix; uploadFileClient.listUploads({"prefix":upload_oss.fileKey}).then(res=> { if(res.nextUploadIdMarker){ let theCheckPoint =  JSON.parse(localStorage.getItem(ossMd5(fileName+fileSize))); currentCheckpoint = theCheckPoint; currentCheckpoint.file = file; } if (currentCheckpoint) { options.checkpoint = currentCheckpoint; } return uploadFileClient.multipartUpload( upload_oss.fileKey, file,options).then(function (uploadRes) { if(currentCheckpoint){ localStorage.removeItem(ossMd5(currentCheckpoint.file.name+currentCheckpoint.file.size)) } currentCheckpoint = null; uploadFileClient = null; client_callback.pushUrl(upload_oss, inputFile, uploadRes); }).catch((err) => { if (uploadFileClient && uploadFileClient.isCancel()) { console.log('stop-upload! '); } else { console.error(err); console.log(`err.name : ${err.name}`); console.log(`err.message : ${err.message}`); if (err.name.toLowerCase().indexOf('connectiontimeout') ! == -1) { // timeout retry if (retryCount < retryCountMax) { retryCount++; console.error(`retryCount : ${retryCount}`); uploadFile(''); }}}}); }); };Copy the code

So this uploadFile method is the first parameter that applyTokenDo passes in, so I’m not going to go over all of this, but the listUploads method, which passes in a path parameter, gets information about an unfinished file, so let’s take a look at the image below, and this is what it returns



When the file is really not completed, nextUploadIdMarker is its uploadID. If the file has not been uploaded or has been uploaded successfully, the returned result is empty. We judge whether nextUploadIdMarker exists to indicate whether part of the file has been uploaded. Check the upload progress from localStorage and assign the current file object to it. The file object will be lost after checkpoint serialization. After the upload is successful, remove the localstorage key. The client_callback method is a callback function added to the business code that has nothing to do with resumable breakpoints and can be ignored

Take a look at the progress bar below:

Var progress = function (p,checkpoint) {if(checkpoint){var checkOnlyKey = ossMd5(checkpoint.file.name+checkpoint.fileSize); localStorage.setItem(checkOnlyKey,JSON.stringify(checkpoint)) } console.log(checkpoint) currentCheckpoint = checkpoint; Return function (done) {var bar = $("#uploadProcess"); var showText = Math.floor(p * 100) + '%'; //bar.style.width = showText; bar.html(showText); done(); }};Copy the code

The method of updating the progress bar will not be described. Check out the structure of checkpoint:



Name is the path to upload the file, other parameters are can be seen on behalf of what is not said.

The entire OSS large file supports refresh breakpoint continuations and that’s basically it.