SpringBoot implementation of large file resumable breakpoint (front-end based on WebUploader implementation, 1G file upload only 5s)

In recent work has involved file upload function, limit the demand requirement documents for 1 g, if directly to upload file on the front end, there will be a long time to wait, if the server memory, memory directly, at this time we can be resolved through the breakpoint continuingly, front we achieved by WebUploader and upload file segmentation, React is the language of the backend, we use SpringBoot to implement file receiving and assembly functions, below I list the main functions of the backend code (see the download address at the end of the article for complete code).

First, front-end code

Since WebUploader relies on Jquery, we should have introduced Jquery in the beginning. The front-end scaffolding we used was Ant Design, so I introduced it in SRC /pages/document.ejs file with the following code:

< script type = "text/javascript" SRC = "< % = context. The config. The manifest. BasePath + 'jquery - 1.11.1. Min. Js' % >" > < / script >Copy the code

After introduction, the illustration is as follows:

Then we register three events in the front-end code, namely before-send-file, before-send and after-send-file. These three hooks are respectively executed before sending the file (before uploading the file, triggering once) and before sending the request (before uploading the file into blocks). Trigger multiple times), after file upload (execute after block upload is complete, trigger once)

JSX: @/pages/device/index.jsx

<Form.Item
  label="Select version pack"
  name="file"
>
  <div name="file">
    <label
      id="uploadWrapper"
      style={{
        display: 'flex',
        alignItems: 'center',
        justifyContent: 'center',
        flexDirection: 'column',
        textAlign: 'center',
        border: '1px dashed #E5E5E5',
        cursor: inputDisabled ? 'default' : 'pointer'}}htmlFor={'softUpload'}
      onDrop={(e)= >{ document.getElementById('uploadWrapper').style.border = '1px dashed #E5E5E5'; Document.getelementbyid ('uploadOption').innerhtml = 'Click or drag the file here to upload '; if (! inputDisabled) { inputOnChange(e); } }} onDragOver={(e) => { e.preventDefault(); document.getElementById('uploadWrapper').style.border = '1px dashed #1890FF'; Document.getelementbyid ('uploadOption').innerhtml = 'release mouse '; }} ><input
        disabled={inputDisabled}
        type="file"
        title=""
        id={'softUpload'}
        multiple={false}
        name="file"
        style={{ opacity: 0 }}
        onChange={(e)= > inputOnChange(e)}
      />
      <label
        htmlFor={'softUpload'}
        style={{ cursor: inputDisabled ? 'default' : 'pointer'}} >
        <p style={{ marginBottom: '10px' }}>
          <span
            style={{
              display: 'block',
              width: '102px',
              height: '30px',
              lineHeight: '30px',
              margin: '0 auto',
              color: '#1890FF',
              backgroundColor: '#E7F3FF',
              border: '1px solid #E7F3FF'}} >
            <UploadOutlined />upload</span>
        </p>
        <div>
          <p id="uploadOption" className="ant-upload-text">Click or drag the file here to upload</p>
          <p className="ant-upload-hint">Supported extensions:<span style={{ color: 'red' }}>{' Version package size does not exceed 1GB'}</span>
          </p>
        </div>
      </label>
    </label>
    <div
      style={{
        maxWidth: '100% ',whiteSpace: 'nowrap',
        textOverflow: 'ellipsis',
        overflow: 'hidden',
        lineHeight: 1.padding: '3px 0',
        marginTop: '13px'}} >
      <span>{task && task.file && task.file.name}</span>
    </div>
    <div style={{ padding: '0 24px 8px 0', width: '100'}} % >
      <Progress
        showInfo
        style={{ display:` ${inputDisabled ? 'block' : 'none` '}}}strokeColor={{
          from: '#108ee9',
          to: '#108ee9'}}percent={task.progress}
        size="small"
      />
    </div>
  </div>
</Form.Item>
Copy the code

The core code of file input click event in @/pages/ Device /index.jsx is as follows:

const inputOnChange = async (e) => {
  e.preventDefault();
  e.stopPropagation();
  const files = e.target.files || e.dataTransfer.files;
  if (files && files[0]) {
    const isLt1G = files[0].size / 1024 / 1024 < 1024;
    if(! isLt1G) { message.error('Version package size cannot exceed 1GB! ');
      return;
    }
    addToTaskList(files[0]); }};// Update a single task
const updateTask = (task, newProps) = > {
  const newTask = Object.assign({}, task, newProps);
  setTask(newTask, function (data) {
    uploadNext(data);
  });
};

// Call the upload interface
const startUpload = (task) = > {
  // Initialization state
  const uploader = new Uploader({
    file: task.file,
    onSuccess: (props) = > {
      updateTask(task, { progress: 100.status: 'success' });
      setCode(props.code);
      setMd5(props.md5);
      setInputDisabled(false);
      message.success(`${props.fileName}File uploaded successfully);
    },
    onError: ({ msg }) = > {
      setInputDisabled(false);
      updateTask(task, { progress: 0.status: 'error' });
      message.error(msg);
    },
    onProgress: ({ file, percentage }) = > {
      const progress = (parseInt(percentage.toFixed(4) * 10000.10) - 1) / 100;
      updateTask(task, { progress, status: 'uploading'}); }}); updateTask(task, {progress: 0.status: 'uploading', uploader });
  setInputDisabled(true);
  uploader.start();
};

// Start the next upload task
const uploadNext = (task) = > {
  // Only one upload task is allowed at a time
  if (task.status === 'uploading') return;
  if (task.status === 'init') { startUpload(task); }};// Add a task
const addToTaskList = (file) = > {
  setTask({ id: new Date().getTime(), file, progress: 0.status: 'init' }, function (data) {
    uploadNext(data);
  });
};
Copy the code

@/utils/upload.js encapsulates the breakpoint continuation core code as follows:

import request from '@/utils/request';
import WebUploader from '.. /.. /public/webuploader.min';
import { TP_TOKE, BPR_BASE_URL } from '@/utils/constant';

** * uploader = new uploader ({* file: targetFile, * onSuccess: ({ fileName, resourceId, filePath }) => { * }, * onError: ({ msg }) => { * }, * onProgress: ({ data, percentage }) => { * }, * }); * * uploader.start(); * ` ` ` *@class Uploader* /
class Uploader {
  constructor({ file, onSuccess, onError, onProgress }) {
    // const files = e.target.files || e.dataTransfer.files;
    // Convert to an internal WebUploader file object
    this.file = new WebUploader.File(new WebUploader.Lib.File(WebUploader.guid('rt_'), file));
    this.onSuccess = props= > {
      this.clean();
      if (onSuccess) onSuccess(props);
    };
    this.onError = props= > {
      this.clean();
      if (onError) onError(props);
    };
    this.onProgress = onProgress;
    this.uploader = null;
  }

  init = () = > {
    WebUploader.Uploader.register({
      name: 'webUploaderHookCommand'.'before-send-file': 'beforeSendFile'.'before-send': 'beforeSend'.'after-send-file': 'afterSendFile'}, {beforeSendFile: file= > {
        const task = new WebUploader.Deferred();
        this.fileName = file.name;
        this.fileSize = file.size;
        this.mimetype = file.type;
        this.fileExt = file.ext;
        (new WebUploader.Uploader())
          .md5File(file, 0.10 * 1024 * 1024 * 1024 * 1024).progress(percentage= > { })
          .then(val= > {
            this.fileMd5 = val;
            const url = `${BPR_BASE_URL}/register`;
            const data = {
              fileMd5: this.fileMd5,
              fileName: file.name,
              fileSize: file.size,
              mimetype: file.type,
              fileExt: file.ext,
            };
            request(url, {
              method: 'post',
              data,
            }).then(res= > {
              console.log('register', res);
              if (res && res.status === 1) {
                task.resolve();
              } else if (res && res.data && res.code === 103404) {
                // The file is uploaded
                this.onSuccess({
                  fileName: this.fileName,
                  resourceId: res.data.resId,
                  filePath: res.data.filePath,
                });
                task.reject();
              } else{ file.statusText = res && res.message; task.reject(); }}); });return task.promise();
      },
      beforeSend: block= > {
        console.log('beforeSend');
        const task = new WebUploader.Deferred();
        const url = `${BPR_BASE_URL}/checkChunk`;
        const data = {
          fileMd5: this.fileMd5,
          chunk: block.chunk,
          chunkSize: block.end - block.start,
        };
        request(url, {
          method: 'post',
          data,
        }).then(res= > {
          console.log('checkChunk', res);
          if (res && res.data === true) {
            task.reject(); // Upload is skipped if fragments exist
          } else{ task.resolve(); }});this.uploader.options.formData.fileMd5 = this.fileMd5;
        this.uploader.options.formData.chunk = block.chunk;
        return task.promise();
      },
      afterSendFile: () = > {
        console.log('start afterSendFile');
        const task = new WebUploader.Deferred();
        const url = `${BPR_BASE_URL}/mergeChunks`;
        const data = {
          fileMd5: this.fileMd5,
          fileName: this.fileName,
          fileSize: this.fileSize,
          mimetype: this.mimetype,
          fileExt: this.fileExt,
        };
        request(url, {
          method: 'post',
          data,
        }).then(res= > {
          console.log('mergeChunks', res);
          if (res && res.status === 1 && res.data && res.data.resId) {
            task.resolve();
            this.onSuccess({
              fileName: this.fileName,
              resourceId: res.data.resId,
              filePath: res.data.filePath,
            });
          } else {
            task.reject();
            this.onError({ msg: 'Failed to merge file'}); }}); }}); } clean =() = > {
    if (this.uploader) {
      WebUploader.Uploader.unRegister('webUploaderHookCommand');
    }
  }

  start = () = > {
    if (!this.uploader) {
      this.init();
    }
    / / instantiate
    this.uploader = WebUploader.create({
      server: BPR_BASE_URL,
      chunked: true.chunkSize: 1024 * 1024 * 5.chunkRetry: 1.threads: 3.duplicate: true.formData: { // Upload the data carried with the SHARded HTTP request
        appid: '1'.token: localStorage.getItem(TP_TOKE),
        methodname: 'breakpointRenewal',}});// This method is called when a fragment is successfully uploaded
    this.uploader.on('uploadProgress'.(data, percentage) = > {
      console.log('uploadProgress');
      this.onProgress({ data, percentage });
    });

    this.uploader.on('error'.err= > {
      this.onError({ msg: 'Error uploading, please try again' });
    });

    this.uploader.addFiles(this.file);
    this.uploader.upload();
  }

  cancel = () = > {
    console.log('call cancel');
    this.uploader.stop(true);
    this.uploader.destroy();
    console.log('getStats'.this.uploader.getStats()); }}export default Uploader;
Copy the code

The constant used in the above code is defined in the @/utils/constant file as follows:

const constants = {
  BPR_BASE_URL: '/ v1.0 / sys/admin/files/breakpointRenewal'};module.exports = constants;
Copy the code

That’s the end of the front-end code.

Second, back-end code

1, ResponseResult code

/ * * *@Classname: com.openailab.oascloud.common.model.ResponseResult
 * @Description: Returns a unified result *@Author: zxzhang
 * @Date: 2019/6/26 * /
@Component
public class ResponseResult implements Serializable {

    private static final long serialVersionUID = 5836869421731990598L;
    /** * State description */
    @Getter
    private String message;
    /** * returns data */
    @Getter
    private Object data;
    /**
     * 响应码
     */
    @Getter
    private int code;
    /** * Status (0: failed, 1: succeeded) */
    @Getter
    private int status;
    /** * total number of entries */
    @Getter
    private Integer total;

    public ResponseResult(a) {}public ResponseResult(int status, Object data) {
        this.status = status;
        this.data = data;
    }

    public ResponseResult(int status, String message, Object data) {
        this.status = status;
        this.message = message;
        this.data = data;
    }

    public ResponseResult(String message, Object data, int code, int status) {
        this.message = message;
        this.data = data;
        this.code = code;
        this.status = status;
    }

    public ResponseResult(String message, Object data, int code, int status, Integer total) {
        this.message = message;
        this.data = data;
        this.code = code;
        this.status = status;
        this.total = total;
    }

    public static ResponseResult fail(String msg) {
        if (StringUtils.isEmpty(msg)) {
            return new ResponseResult(ResponseEnum.RESPONSE_CODE_FAIL.getMsg(), null, ResponseEnum.RESPONSE_CODE_FAIL.getCode(), CommonConst.RESPONSE_FAIL);
        } else {
            return new ResponseResult(msg, null, ResponseEnum.RESPONSE_CODE_FAIL.getCode(), CommonConst.RESPONSE_FAIL); }}public static ResponseResult fail(int code, String msg) {
        return new ResponseResult(msg, null, code, CommonConst.RESPONSE_FAIL);
    }

    public static ResponseResult fail(ResponseEnum responseEnum, Object obj) {
        return new ResponseResult(responseEnum.getMsg(), obj, responseEnum.getCode(), 0);
    }

    public static ResponseResult success(Object data) {
        return new ResponseResult(ResponseEnum.RESPONSE_CODE_SUCCESS.getMsg(), data, ResponseEnum.RESPONSE_CODE_SUCCESS.getCode(), CommonConst.RESPONSE_SUCCESS);
    }

    public static ResponseResult success(Object data, int code, String message) {
        return new ResponseResult(message, data, code, CommonConst.RESPONSE_SUCCESS);
    }

    public ResponseResult setMessage(String message) {
        this.message = message;
        return this;
    }

    public ResponseResult setData(Object data) {
        this.data = data;
        return this;
    }

    public ResponseResult setStatus(int status) {
        this.status = status;
        return this;
    }

    public ResponseResult setCode(int code) {
        this.code = code;
        return this;
    }

    public ResponseResult setTotal(Integer total) {
        this.total = total;
        return this; }}Copy the code

2. FileController code

/ * * *@description: File management -Controller *@author: zhangzhixiang
 * @createDate: 2019/12/9
 * @version: 1.0 * /
@RestController
@ RequestMapping ("/v1.0 / sys/admin/files "),
public class FileController {

    private static Logger LOG = LoggerFactory.getLogger(FileController.class);
    @Autowired
    private IFileService fileService;
    @Autowired
    private IUserService userService;

    /** ** *@param file
     * @param fileMd5
     * @param chunk
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/13 * /
    @PostMapping(value = "/breakpointRenewal", produces = {MediaType.APPLICATION_JSON_UTF8_VALUE}, consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
    public ResponseResult breakpointRenewal(@RequestPart("file") MultipartFile file,
                                            @RequestParam("fileMd5") String fileMd5,
                                            @RequestParam("chunk") Integer chunk) {
        try {
            return fileService.breakpointRenewal(file, fileMd5, chunk);
        } catch (Exception e) {
            LOG.error("********FileController->breakpointRenewal throw Exception.fileMd5:{},chunk:{}********", fileMd5, chunk, e);
        }
        return ResponseResult.fail(null);
    }

    /** * register **@param fileMd5
     * @param fileName
     * @param fileSize
     * @param mimetype
     * @param fileExt
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/13 * /
    @PostMapping(value = "/breakpointRenewal/register")
    public ResponseResult breakpointRegister(@RequestParam("fileMd5") String fileMd5,
                                             @RequestParam("fileName") String fileName,
                                             @RequestParam("fileSize") Long fileSize,
                                             @RequestParam("mimetype") String mimetype,
                                             @RequestParam("fileExt") String fileExt) {
        try {
            return fileService.breakpointRegister(fileMd5, fileName, fileSize, mimetype, fileExt);
        } catch (Exception e) {
            LOG.error("********FileController->breakpointRegister throw Exception.fileMd5:{},fileName:{}********", fileMd5, fileName, e);
        }
        return ResponseResult.fail(null);
    }

    /** * check whether the partition exists **@param fileMd5
     * @param chunk
     * @param chunkSize
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/10 * /
    @PostMapping(value = "/breakpointRenewal/checkChunk")
    public ResponseResult checkChunk(@RequestParam("fileMd5") String fileMd5,
                                     @RequestParam("chunk") Integer chunk,
                                     @RequestParam("chunkSize") Integer chunkSize) {
        try {
            return fileService.checkChunk(fileMd5, chunk, chunkSize);
        } catch (Exception e) {
            LOG.error("********FileController->breakpointRenewal throw Exception.fileMd5:{},chunk:{}********", fileMd5, chunk, e);
        }
        return ResponseResult.fail(null);
    }

    /** * Merge file blocks **@param fileMd5
     * @param fileName
     * @param fileSize
     * @param mimetype
     * @param fileExt
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/11 * /
    @PostMapping(value = "/breakpointRenewal/mergeChunks")
    public ResponseResult mergeChunks(@RequestParam("fileMd5") String fileMd5,
                                      @RequestParam("fileName") String fileName,
                                      @RequestParam("fileSize") Long fileSize,
                                      @RequestParam("mimetype") String mimetype,
                                      @RequestParam("fileExt") String fileExt,
                                      @RequestParam("token") String token) {
        try {
            LoginUserInfo user = userService.getLoginUser(token);
            return fileService.mergeChunks(fileMd5, fileName, fileSize, mimetype, fileExt, user);
        } catch (Exception e) {
            LOG.error("********FileController->breakpointRenewal throw Exception.fileMd5:{},fileName:{}********", fileMd5, fileName, e);
        }
        return ResponseResult.fail(null); }}Copy the code

3. IFileService code

/ * * *@description: file management -Interface *@author: zhangzhixiang
 * @createDate: 2019/12/9
 * @version: 1.0 * /
public interface IFileService {
    /** * register **@param fileMd5
     * @param fileName
     * @param fileSize
     * @param mimetype
     * @param fileExt
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/10 * /
    ResponseResult breakpointRegister(String fileMd5, String fileName, Long fileSize, String mimetype, String fileExt);

    /** ** *@param file
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2019/12/9 * /
    ResponseResult breakpointRenewal(MultipartFile file, String fileMd5, Integer chunk);

    /** * check whether the partition exists **@param fileMd5
     * @param chunk
     * @param chunkSize
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/10 * /
    ResponseResult checkChunk(String fileMd5, Integer chunk, Integer chunkSize);

    /** * Merge file blocks **@param fileMd5
     * @param fileName
     * @param fileSize
     * @param mimetype
     * @param fileExt
     * @return com.openailab.oascloud.common.model.ResponseResult
     * @author zxzhang
     * @date2020/1/11 * /
    ResponseResult mergeChunks(String fileMd5, String fileName, Long fileSize, String mimetype, String fileExt, LoginUserInfo user);
}
Copy the code

FileServiceImpl code

/ * * *@description: file management -service *@author: zhangzhixiang
 * @createDate: 2019/12/9
 * @version: 1.0 * /
@Service
public class FileServiceImpl implements IFileService {

    private final static Logger LOG = LoggerFactory.getLogger(FileServiceImpl.class);
    private static final SimpleDateFormat format = new SimpleDateFormat("yyyyMMdd");
    @Autowired
    private FileDao fileDao;
    @Autowired
    private BootstrapConfig bootstrapConfig;
    @Autowired
    private FileManagementHelper fileManagementHelper;
    @Autowired
    private PageObjUtils pageObjUtils;
    @Autowired
    private RedisDao redisDao;

    private String getUploadPath(a) {
        return bootstrapConfig.getFileRoot() + bootstrapConfig.getUploadDir() + "/";
    }

    private String getFileFolderPath(String fileMd5) {
        return getUploadPath() + fileMd5.substring(0.1) + "/" + fileMd5.substring(1.2) + "/";
    }

    private String getFilePath(String fileMd5, String fileExt) {
        return getFileFolderPath(fileMd5) + fileMd5 + "." + fileExt;
    }

    private String getFileRelativePath(String fileMd5, String fileExt) {
        return bootstrapConfig.getUploadDir() + "/" + fileMd5.substring(0.1) + "/" + fileMd5.substring(1.2) + "/" + fileMd5 + "." + fileExt;
    }

    private String getChunkFileFolderPath(String fileMd5) {
        return bootstrapConfig.getFileRoot() + bootstrapConfig.getBreakpointDir() + "/" + fileMd5 + "/";
    }

    @Override
    public ResponseResult breakpointRegister(String fileMd5, String fileName, Long fileSize, String mimetype, String fileExt) {
        Map<String, String> ret = Maps.newHashMap();
        // Check whether the file exists on disk
        String fileFolderPath = this.getFileFolderPath(fileMd5);
        String filePath = this.getFilePath(fileMd5, fileExt);
        File file = new File(filePath);
        boolean exists = file.exists();

        // Check whether the file exists in PostgreSQL (file with unique identifier fileMd5)
        ResourceBO resourceBO = new ResourceBO();
        resourceBO.setFileMd5(fileMd5);
        resourceBO.setIsDelete(0);
        List<ResourceBO> resourceBOList = fileDao.selectResourceByCondition(resourceBO);
        if (exists && resourceBOList.size() > 0) {
            ResId, filePath, resId, filePath, resId, filePath
            resourceBO = resourceBOList.get(0);
            ret.put("filePath", resourceBO.getFilePath());
            ret.put("resId", String.valueOf(resourceBO.getResourceId()));
            return ResponseResult.fail(ResponseEnum.RESPONSE_CODE_BREAKPOINT_RENEVAL_REGISTRATION_ERROR, ret);
        }

        // If it exists on disk but not in database, generate resource record and save it to redis
        if (resourceBOList.size() == 0) {
            // The first breakpoint file needs to create a new resource record and return redId to be stored in Redis
            resourceBO.setType(fileManagementHelper.judgeDocumentType(fileExt));
            resourceBO.setStatus(TranscodingStateEnum.UPLOAD_NOT_COMPLETED.getCode());
            resourceBO.setFileSize(fileSize);
            resourceBO.setFileMd5(fileMd5);
            resourceBO.setFileName(fileName);
            resourceBO.setCreateDate(new Date());
            resourceBO.setIsDelete(0);
            final Integer resourceId = fileDao.addResource(resourceBO);
            resourceBO.setResourceId(resourceId);
            redisDao.set(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5, JSONObject.toJSONString(resourceBO), RedisPrefixConst.EXPIRE);
        }

        // If it does not exist in redis but exists in the database, store it in redis
        String breakpoint = redisDao.get(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5);
        if (StringUtils.isEmpty(breakpoint) && resourceBOList.size() > 0) {
            resourceBO = resourceBOList.get(0);
            redisDao.set(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5, JSONObject.toJSONString(resourceBO), RedisPrefixConst.EXPIRE);
        }

        // If the file does not exist, check whether the file directory exists
        File fileFolder = new File(fileFolderPath);
        if(! fileFolder.exists()) {// The directory was not created (the directory was created based on the MD5 value from the front end)
            fileFolder.mkdirs();
        }
        return ResponseResult.success(null);
    }

    @Override
    public ResponseResult breakpointRenewal(MultipartFile file, String fileMd5, Integer chunk) {
        Map<String, String> ret = Maps.newHashMap();
        // Check whether the block directory exists
        String chunkFileFolderPath = this.getChunkFileFolderPath(fileMd5);
        File chunkFileFolder = new File(chunkFileFolderPath);
        if(! chunkFileFolder.exists()) { chunkFileFolder.mkdirs(); }// Upload file input stream
        File chunkFile = new File(chunkFileFolderPath + chunk);
        try (InputStream inputStream = file.getInputStream(); FileOutputStream outputStream = new FileOutputStream(chunkFile)) {
            IOUtils.copy(inputStream, outputStream);
            // Check whether there are fileMd5 block records in Redis (resId)
            String breakpoint = redisDao.get(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5);
            ResourceBO resourceBO = new ResourceBO();
            if(! StringUtils.isEmpty(breakpoint)) {// The existence of block records indicates that the resource is being uploaded. The resId corresponding to fileMd5 is returned directly, and the resource records will not be created repeatedly
                resourceBO = JSONObject.parseObject(breakpoint, ResourceBO.class);
                ret.put("resId", String.valueOf(resourceBO.getResourceId())); }}catch (IOException e) {
            e.printStackTrace();
        }
        return ResponseResult.success(ret);
    }

    @Override
    public ResponseResult checkChunk(String fileMd5, Integer chunk, Integer chunkSize) {
        // Check whether the partitioned file exists
        String chunkFileFolderPath = this.getChunkFileFolderPath(fileMd5);
        // The path where the partition is located + the index of the partition can locate the specific partition
        File chunkFile = new File(chunkFileFolderPath + chunk);
        if (chunkFile.exists() && chunkFile.length() == chunkSize) {
            return ResponseResult.success(true);
        }
        return ResponseResult.success(false);
    }

    @Override
    public ResponseResult mergeChunks(String fileMd5, String fileName, Long fileSize, String mimetype, String fileExt, LoginUserInfo user) {
        FileClient fileClient = ClientFactory.createClientByType(bootstrapConfig.getFileClientType());
        String chunkFileFolderPath = this.getChunkFileFolderPath(fileMd5);
        File chunkFileFolder = new File(chunkFileFolderPath);
        File[] files = chunkFileFolder.listFiles();
        final String filePath = this.getFilePath(fileMd5, fileExt);
        File mergeFile = new File(filePath);
        List<File> fileList = Arrays.asList(files);

        // 1. Merge blocks
        mergeFile = this.mergeFile(fileList, mergeFile);
        if (mergeFile == null) {
            return ResponseResult.fail(ResponseEnum.RESPONSE_CODE_MERGE_FILE_ERROR, null);
        }
        // 2. Check whether the MD5 file is consistent with the incoming one
        boolean checkResult = this.checkFileMd5(mergeFile, fileMd5);
        if(! checkResult) {return ResponseResult.fail(ResponseEnum.RESPONSE_CODE_VERIFY_FILE_ERROR, null);
        }

        // 3. Delete all blocks of the file
        FileUtil.deleteDir(chunkFileFolderPath);
        // Get file block records in Redis
        String breakpoint = redisDao.get(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5);
        if (StringUtils.isEmpty(breakpoint)) {
            return ResponseResult.fail("File block does not exist");
        }
        ResourceBO resourceBO = JSONObject.parseObject(breakpoint, ResourceBO.class);
        // Delete redis blocks
        redisDao.del(RedisPrefixConst.BREAKPOINT_PREFIX + fileMd5);

        // 6
        ret.put("filePath", getFileRelativePath(fileMd5, fileExt));
        ret.put("resId", String.valueOf(resourceBO.getResourceId()));
        return ResponseResult.success(ret);
    }

    /** * merge files **@param chunkFileList
     * @param mergeFile
     * @return java.io.File
     * @author zxzhang
     * @date2020/1/11 * /
    private File mergeFile(List<File> chunkFileList, File mergeFile) {
        try {
            // Yes Delete no create
            if (mergeFile.exists()) {
                mergeFile.delete();
            } else {
                mergeFile.createNewFile();
            }
            / / sorting
            Collections.sort(chunkFileList, (o1, o2) -> {
                if (Integer.parseInt(o1.getName()) > Integer.parseInt(o2.getName())) {
                    return 1;
                }
                return -1;
            });

            byte[] b = new byte[1024];
            RandomAccessFile writeFile = new RandomAccessFile(mergeFile, "rw");
            for (File chunkFile : chunkFileList) {
                RandomAccessFile readFile = new RandomAccessFile(chunkFile, "r");
                int len = -1;
                while((len = readFile.read(b)) ! = -1) {
                    writeFile.write(b, 0, len);
                }
                readFile.close();
            }
            writeFile.close();
            return mergeFile;
        } catch (IOException e) {
            e.printStackTrace();
            return null; }}/** * Verify file MD5 **@param mergeFile
     * @param md5
     * @return boolean
     * @author zxzhang
     * @date2020/1/11 * /
    private boolean checkFileMd5(File mergeFile, String md5) {
        try {
            // get the MD5 file
            FileInputStream inputStream = new FileInputStream(mergeFile);
            String md5Hex = DigestUtils.md5Hex(inputStream);

            if (StringUtils.equalsIgnoreCase(md5, md5Hex)) {
                return true; }}catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }

    /** * obtain the file suffix **@param fileName
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getExt(String fileName) {
        return fileName.substring(fileName.lastIndexOf(".") + 1);
    }

    /** * Get the file directory **@param filePath
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getFileDir(String filePath) {
        return filePath.substring(0, filePath.lastIndexOf(BootstrapConst.PATH_SEPARATOR));
    }

    /** * get the file name **@param filePath
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getFileName(String filePath) {
        return filePath.substring(filePath.lastIndexOf(BootstrapConst.PATH_SEPARATOR) + 1, filePath.lastIndexOf(".")); }}Copy the code

5. FileUtil code

/ * * *@description:
 * @author: zhangzhixiang
 * @createDate: 2020/1/7
 * @version: 1.0 * /
public class FileUtil {

    private static final Logger LOG = LoggerFactory.getLogger(FileUtil.class);

    /** * Clear all files in the folder **@param path
     * @return boolean
     * @author zxzhang
     * @date2020/1/7 * /
    public static boolean deleteDir(String path) {
        File file = new File(path);
        if(! file.exists()) {// Check whether the directory to be deleted exists
            return false;
        }
        String[] content = file.list();// Get all files and folders in the current directory
        for (String name : content) {
            File temp = new File(path, name);
            if (temp.isDirectory()) {// Check if it is a directory
                deleteDir(temp.getAbsolutePath());// Delete the contents of the directory recursively
                temp.delete();// Delete an empty directory
            } else {
                if(! temp.delete()) {// Delete the file directly
                    LOG.error("******** file deletion failed, file: {}********"+ name); }}}return true;
    }

    /** * Copy a single file **@paramOldPath String Path of the original file, for example, c:/ FQF *@paramNewPath String Path after replication, for example, f:/ FQF /ff *@return boolean
     */
    public static void copyFile(String oldPath, String newPath) {
        try {
            int bytesum = 0;
            int byteread = 0;
            File oldfile = new File(oldPath);
            if (oldfile.exists()) { // If the file exists
                InputStream inStream = new FileInputStream(oldPath); // Read the original file
                // The newFilePath folder does not exist
                File newParentFile = new File(newPath).getParentFile();
                if(! newParentFile.exists()) { newParentFile.mkdirs(); } FileOutputStream fs =new FileOutputStream(newPath);
                byte[] buffer = new byte[1444];
                int length;
                while((byteread = inStream.read(buffer)) ! = -1) {
                    bytesum += byteread; // Number of bytes File size
                    System.out.println(bytesum);
                    fs.write(buffer, 0, byteread); } inStream.close(); }}catch (Exception e) {
            LOG.error("******** Error copying single file ********"); e.printStackTrace(); }}/** * copy the entire folder **@paramOldPath String Path of the original file, for example, c:/ FQF *@paramNewPath String Path after replication, for example, f:/ FQF /ff *@return boolean
     */
    public static void copyFolder(String oldPath, String newPath) {
        try {
            // The newFilePath folder does not exist
            File newParentFile = new File(newPath).getParentFile();
            if(! newParentFile.exists()) { newParentFile.mkdirs(); } File a =new File(oldPath);
            String[] file = a.list();
            File temp = null;
            for (int i = 0; i < file.length; i++) {
                if (oldPath.endsWith(File.separator)) {
                    temp = new File(oldPath + file[i]);
                } else {
                    temp = new File(oldPath + File.separator + file[i]);
                }
                
                if (temp.isFile()) {
                    FileInputStream input = new FileInputStream(temp);
                    FileOutputStream output = new FileOutputStream(newPath + "/" +
                            (temp.getName()).toString());
                    byte[] b = new byte[1024 * 5];
                    int len;
                    while((len = input.read(b)) ! = -1) {
                        output.write(b, 0, len);
                    }
                    output.flush();
                    output.close();
                    input.close();
                }
                if (temp.isDirectory()) {// If it is a subfolder
                    copyFolder(oldPath + "/" + file[i], newPath + "/"+ file[i]); }}}catch (Exception e) {
            LOG.error("******** Error copying contents of entire folder ********"); e.printStackTrace(); }}/** * Get the MD5 value of a file **@param file
     * @return java.lang.String
     * @author zxzhang
     * @date2020/3/23 * /
    public static String getMD5(MultipartFile file) {
        InputStream fileInputStream = null;
        try {
            MessageDigest MD5 = MessageDigest.getInstance("MD5");
            fileInputStream = file.getInputStream();
            byte[] buffer = new byte[8192];
            int length;
            while((length = fileInputStream.read(buffer)) ! = -1) {
                MD5.update(buffer, 0, length);
            }
            return new String(Hex.encodeHex(MD5.digest()));
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        } finally {
            try {
                if(fileInputStream ! =null) { fileInputStream.close(); }}catch(IOException e) { e.printStackTrace(); }}}/** * Get the MD5 value of a file **@param file
     * @return java.lang.String
     * @author zxzhang
     * @date2020/3/23 * /
    public static String getMD5(File file) {
        try(FileInputStream fileInputStream = new FileInputStream(file)) {
            MessageDigest MD5 = MessageDigest.getInstance("MD5");
            byte[] buffer = new byte[8192];
            int length;
            while((length = fileInputStream.read(buffer)) ! = -1) {
                MD5.update(buffer, 0, length);
            }
            return new String(Hex.encodeHex(MD5.digest()));
        } catch (Exception e) {
            e.printStackTrace();
            return null; }}/** * find the MD5 value of a string **@param target
     * @return java.lang.String
     * @author zxzhang
     * @date2020/3/23 * /
    public static String MD5(String target) {
        returnDigestUtils.md5Hex(target); }}Copy the code

6. FileManagementHelper code

/ * * *@description:
 * @author: zhangzhixiang
 * @createDate: 2019/12/11
 * @version: 1.0 * /
@Component
public class FileManagementHelper {

    private static final Logger LOG = LoggerFactory.getLogger(FileManagementHelper.class);
    @Autowired
    private BootstrapConfig bootstrapConfig;

    /** * Determine the file type based on the file suffix **@param ext
     * @return java.lang.Integer
     * @author zxzhang
     * @date2019/12/10 * /
    public Integer judgeDocumentType(String ext) {
        / / video class
        if(VedioEnum.containKey(ext) ! =null) {
            return ResourceTypeEnum.VIDEO.getCode();
        }
        / / picture
        if(ImageEnum.containKey(ext) ! =null) {
            return ResourceTypeEnum.IMAGE.getCode();
        }
        / / document class
        if(DocumentEnum.containKey(ext) ! =null) {
            return ResourceTypeEnum.FILE.getCode();
        }
        / / unknown
        return ResourceTypeEnum.OTHER.getCode();
    }

    /** * Generates a random file name **@param ext
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public static String createFileName(String ext) {
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
        return simpleDateFormat.format(new Date()) + (int) (Math.random() * 900 + 100) + ext;
    }

    /** * obtain the file suffix **@param fileName
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getExt(String fileName) {
        return fileName.substring(fileName.lastIndexOf(".") + 1);
    }

    /** * Get the file directory **@param filePath
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getFileDir(String filePath) {
        return filePath.substring(0, filePath.lastIndexOf(BootstrapConst.PATH_SEPARATOR));
    }

    /** * get the file name **@param filePath
     * @return java.lang.String
     * @author zxzhang
     * @date2019/12/10 * /
    public String getFileName(String filePath) {
        return filePath.substring(filePath.lastIndexOf(BootstrapConst.PATH_SEPARATOR) + 1, filePath.lastIndexOf(".")); }}Copy the code

7, PageObjUtils code

/ * * *@Classname: com.openailab.oascloud.um.util.PageObjUtils
 * @Description: paging object utility class *@Author: ChenLiang
 * @Date: 2019/7/17 * /
@Component
public class PageObjUtils<T> {

    public PageVO getPageList(PageInfo<T> personPageInfo) {
        PageVO result = new PageVO();
        if(personPageInfo ! =null) {
            if (!personPageInfo.getList().isEmpty()) {
                result.setPageNo(personPageInfo.getPageNum());
                result.setPageSize(personPageInfo.getPageSize());
                result.setTotal(Integer.valueOf(String.valueOf(personPageInfo.getTotal())));
                result.setItems(personPageInfo.getList());
            }
        }
        returnresult; }}Copy the code

8. RedisDao code

/ * * *@Classname: com.openailab.oascloud.datacenter.api.IRedisApi
 * @Description: Redis API
 * @Author: zxzhang
 * @Date: 2019/7/1 * /
@FeignClient(ServiceNameConst.OPENAILAB_DATA_CENTER_SERVICE)
public interface RedisDao {
    / * * *@api{POST} /redis/set Normal cache is put and set expiration time *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam* {String} key key@apiParam* {String} the value value@apiParam{long} expire Expire time */
    @PostMapping("/redis/set")
    ResponseResult set(@RequestParam("key") String key, @RequestParam("value") String value, @RequestParam("expire") long expire);

    / * * *@api{POST} /redis/get Common cache fetch *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam* {String} key key@apiSuccess{String} the value value * /
    @PostMapping("/redis/get")
    String get(@RequestParam("key") String key);

    / * * *@api{POST} /redis/del Normal cache delete *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam{String} key key * /
    @PostMapping("/redis/del")
    ResponseResult del(@RequestParam("key") String key);

    / * * *@api{POST} /redis/hset stores Hash value and sets expiration time *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam* {String} key key@apiParam{String} * item item@apiParam* {String} the value value@apiParam{long} expire Expire time */
    @PostMapping("/redis/hset")
    ResponseResult hset(@RequestParam("key") String key, @RequestParam("item") String item, @RequestParam("value") String value, @RequestParam("expire") long expire);

    / * * *@api{POST} /redis/hget gets the Hash value *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam* {String} key key@apiParam{String} * item item@apiSuccess* {String} the value value@apiSuccessExample{json} Successful Example * {"name":" Zhang SAN ","age":30} */
    @PostMapping("/redis/hget")
    Object hget(@RequestParam("key") String key, @RequestParam("item") String item);

    / * * *@api{POST} /redis/hdel Deletes the Hash value SaasAppKeyDao *@apiGroup Redis
     * @apiVersion0.1.0 from *@apiParam* {String} key key@apiParam{String} * / item item
    @PostMapping("/redis/hdel")
    ResponseResult hdel(@RequestParam("key") String key, @RequestParam("item") String item);
}
Copy the code

9. BootstrapConfig code

/ * * *@Classname: com.openailab.oascloud.security.common.config.BootstrapConsts
 * @Description: Bootstrap *@Author: zxzhang
 * @Date: 2019/10/8 * /
@Data
@Configuration
public class BootstrapConfig {

    @Value("${file.client.type}")
    private String fileClientType;

    @Value("${file.root}")
    private String fileRoot;

    @Value("${file.biz.file.upload}")
    private String uploadDir;

    @Value("${file.biz.file.download}")
    private String downloadDir;

    @Value("${file.biz.file.backup}")
    private String backupDir;

    @Value("${file.biz.file.tmp}")
    private String tmpDir;

    @Value("${file.biz.file.breakpoint}")
    private String breakpointDir;
}
Copy the code

10. And application properties

eureka.instance.instance-id=${spring.application.name}:${server.port} eureka.instance.prefer-ip-address=true Eureka. Client. ServiceUrl. DefaultZone = http://127.0.0.1:32001/eureka/ server port = 32018 spring.application.name=openailab-file-management #file file.client.type = ceph file.root = /usr/local/oas/file file.biz.file.upload = /upload file.biz.file.download = /download file.biz.file.backup = /backup file.biz.file.tmp = /tmp file.biz.file.breakpoint = /breakpoint #ribbon ribbon.ReadTimeout=600000 ribbon.ConnectTimeout=600000 #base The info. The description = file management service [email protected] spring. Servlet. Multipart. Enabled = true spring.servlet.multipart.max-file-size=5120MB spring.servlet.multipart.max-request-size=5120MBCopy the code

Table structure

File breakpoint continued to upload here to introduce the end, the complete front and back end code here, portal:

WebUploader

Resumable Back-end Code Download (Java)