Recently, due to the business of the company, we need to do the file upload function. When talking about this, many people will think that it is not simple, a file input control will do it.
Things, of course, are not that simple, because in addition to basic file uploads, you need to support sliced uploads, breakpoint uploads, breakpoint uploads, cross-browser breakpoint uploads, and so on.
Because the technology stack of the project is VUE, I found a plug-in vue-simple-Uploader which realized file slice uploading and file queue management based on VUE.
The front-end function can be solved by the plug-in, and when writing the file upload function, because the back-end interface is not provided, so on a whim, I use nodeJS to load a upload service vue-uploader for debugging, here to share:
Upload process
- The user selects the file to be uploaded
- Send a request to verify the upload status of the file
- Uploading file Slices
- Merge file slices
The logic of user selection and upload interaction is handled on the front side, so here are a few aspects:
Verify file upload status
The main operation of this interface is to determine whether the file to be uploaded exists on the server. If so, an identifier is returned to inform the user that the upload operation can be skipped and the upload is successful. The main steps and some codes are as follows:
- Generate a file access path based on the unique file id and file name
const router = require('express').Router()
const sparkMD5 = require('spark-md5')
// Encapsulate the file operation code
const { genFilePath } = require('.. /utils/file')
// GET /upload interface internal code
router.get('/', (req, res) => {
const { identifier, filename } = req.query
const name = decodeURIComponent(filename)
const fileHash = sparkMD5.hash(name + identifier)
const filePath = genFilePath(fileHash)
// ... other code
})
Copy the code- Check whether the file already exists. If yes, return the available transmission and file download path
const router = require('express').Router()
const sparkMD5 = require('spark-md5')
// The product hash is used to generate file signatures
const proHash = sparkMD5.hash('This is your flag')
// File signatures generate merge symbols between fields
const hashSeparator = '!!!!!! '
// Encapsulates the method that generates the response object
const { success } = require('.. /utils/response')
// GET /upload interface internal code
router.get('/', (req, res) => {
/ /... Generate file access path code based on the unique file id and file name
const result = {
isRapidUpload: false.url: ' '.uploadedChunks: []}if(fs.existsSync(filePath)) {
// File signature
const sig = sparkMD5.hash([proHash, name, fileHash].join(hashSeparator))
return res.json(success(Object.assign(result, {
isRapidUpload: true.url: [
'http://localhost:3000/upload'.'file', fileHash, filename, sig
].join('/')}),'Can transmit in seconds'))}// ... other code
})
Copy the code- If the full file does not exist, the list of uploaded slices of the file is returned
const router = require('express').Router()
const sparkMD5 = require('spark-md5')
// Slice name prefix
const chunkNamePre = 'chunk'
// Encapsulate the file operation code
const { genChunkDir } = require('.. /utils/file')
// Encapsulates the method that generates the response object
const { success } = require('.. /utils/response')
// GET /upload interface internal code
router.get('/', (req, res) => {
/ /... Generate file access path code based on the unique file id and file name
/ /... Check whether the file exists. If yes, return the code of the file download path
const chunkDir = genChunkDir(identifier)
const existsChunks = fs.readdirSync(chunkDir)
// There are no uploaded slices
if(! existsChunks.length)return res.json(success(result, 'File does not exist'))
// Upload some slices
const uploadedChunks = existsChunks.map(chunk= > parseInt(chunk.slice(chunkNamePre.length)))
res.json(success(Object.assign(result, { uploadedChunks }), 'Partially uploaded slices'))})Copy the codeThe above is the logic for verifying file status before uploading. The front-end can decide whether to upload files and which slices need to be uploaded based on the status.
Uploading file Slices
If the file to be uploaded does not exist on the server, the front-end will invoke the slice upload interface, and its logic is as follows:
- Save the slices uploaded from the front end to the corresponding temporary directory
const router = require('express').Router()
const multer = require('multer')
const upload = multer({ dest: path.join(__dirname, '.. /tpl/')})// Slice name prefix
const chunkNamePre = 'chunk'
// Encapsulate the file operation code
const { genChunkDir } = require('.. /utils/file')
router.post('/', upload.single(fileParamName), async (req, res) => {
const { identifier, chunkNumber, totalSize, chunkSize } = req.body
// Generate a temporary storage directory for slices
const chunksDir = genChunkDir(identifier)
// Generate a temporary file name for each stored block based on the block index and unique file identifier
const chunkPath = chunksDir + '/' + chunkNamePre + chunkNumber
// Rename the uploaded block file to the file name generated above
fs.renameSync(req.file.path, chunkPath)
// Polling verifies that each block exists. If each block exists, the response is uploaded
let currChunkNumber = 1
const totalChunks = Math.max(Math.floor(totalSize / chunkSize), 1)
const chunkPathList = []
while(currChunkNumber <= totalChunks) {
const currChunkPath = chunksDir + '/' + chunkNamePre + currChunkNumber
chunkPathList.push(currChunkPath)
currChunkNumber++
}
Promise.all(chunkPathList.map(chunkPath= > testChunkExist(chunkPath)))
.then(resultList= > {
res.json(success({
isComplete: resultList.every(result= > result)
}))
})
})
Copy the codeMulter is a nodejs plugin that handles file uploads
- Verify whether file slices are uploaded completely
const router = require('express').Router()
const multer = require('multer')
const upload = multer({ dest: path.join(__dirname, '.. /tpl/')})// Slice name prefix
const chunkNamePre = 'chunk'
// Encapsulate the file operation code
const { testChunkExist } = require('.. /utils/file')
router.post('/', upload.single(fileParamName), async (req, res) => {
const { identifier, chunkNumber, totalSize, chunkSize } = req.body
// Check whether each slice exists. If each block exists, the response upload is complete
let currChunkNumber = 1
const totalChunks = Math.max(Math.floor(totalSize / chunkSize), 1)
const chunkPathList = []
while(currChunkNumber <= totalChunks) {
const currChunkPath = chunksDir + '/' + chunkNamePre + currChunkNumber
chunkPathList.push(currChunkPath)
currChunkNumber++
}
// Asynchronous batch check whether file slices exist
Promise.all(chunkPathList.map(chunkPath= > testChunkExist(chunkPath)))
.then(resultList= > {
res.json(success({
isComplete: resultList.every(result= > result)
}))
})
})
Copy the codeIf isComplete is true, the file slice is uploaded completely and can be merged
Merge file slices
If the file slices are uploaded completely, this interface merges all the file slices and returns the file download path
const router = require('express').Router()
const sparkMD5 = require('spark-md5')
const childProcess = require('child_process')
// The product hash is used to generate file signatures
const proHash = sparkMD5.hash('This is your flag')
// File signatures generate merge symbols between fields
const hashSeparator = '!!!!!! '
// Encapsulate the file operation code
const { testChunkExist, genFilePath, writeChunks } = require('.. /utils/file')
const { success, fail } = require('.. /utils/response')
router.post('/merge', (req, res) => {
const { identifier, fileName } = req.body
const chunkDir = genChunkDir(identifier)
// File name + suffix
const name = decodeURIComponent(fileName)
// Generate the actual storage name of the file
const fileHash = sparkMD5.hash(name + identifier)
// Actual file storage path
const filePath = genFilePath(fileHash)
// File signature
const sig = sparkMD5.hash([proHash, name, fileHash].join(hashSeparator))
// Get all the slice paths in the slice directory and sort
try {
// Write each slice to the final path
const writeSuccess = writeChunks(chunkDir, filePath, chunkNamePre)
if(! writeSuccess)return res.json(fail(101.void 0.'This file fragment does not exist'))
// Delete the cache slice directory
childProcess.exec(`rm -rf ${chunkDir}`)
// Returns the file access URL
res.json(success({
url: [
'http://localhost:3000/upload'.'file', fileHash, fileName, sig
].join('/')},'Section merge successful'))}catch(err) {
childProcess.exec(`rm -rf ${chunkDir}`)
res.json(fail(102.void 0.'Slice merge failed'))}})Copy the codeDelete the temporary slice directory after the file slice merge is completed. Since this operation is not necessarily related to the interface response, the sub-process is enabled to perform the operation to avoid affecting the response time
conclusion
The above is a simple interface to support file slicing upload. Because it is a demo used by myself, it is relatively simple and only used to provide ideas for communication. Have any good ideas and opinions, look forward to your exchange.
The above is all the content of this article, if you have any questions, please correct; If need to reprint, hope to indicate the source