I. Functional and non-functional requirements
Easy to operate, select multiple files and folders to upload at a time; Support PC operating system, Windows,Linux,Mac
Support file and folder batch download, resumable breakpoint. Refresh the page and continue transferring. Save the progress information after closing the browser.
Supports batch upload and download of folders, and retains the folder hierarchy on the server, which is the same as that on the local server.
Large files can be uploaded in batches (20G) and downloaded, and users’ computers should not be jammed during the upload. Supports folder upload, the number of files in the folder reaches more than 10,000, and contains a hierarchical structure.
Support breakpoint continuation, close the browser or refresh the browser can still keep the progress.
Support folder structure management, support new folder, support folder directory navigation
Friendly interaction, can timely feedback upload progress;
The security of the server side does not affect the use of other functions due to the overflow of JVM memory caused by the upload file function;
Maximize the use of uplink bandwidth to improve the upload speed.
Second, design analysis
For the processing of large files, whether client or server, it is not advisable to read, send and receive at one time, which may easily lead to memory problems. Therefore, for large files to upload, the use of block upload
From the perspective of upload efficiency, the maximum efficiency can be achieved by using multi-thread concurrent upload.
Iii. Solutions:
The front end of the file upload page can choose to use some more useful upload component, such as baidu’s open source components WebUploader, ze up6 optimal software, these components can meet the basic file upload some daily required functions, such as asynchronous upload files, folders, upload drag-and-drop, paste upload, upload progress monitoring, file thumbnails, Even large files resumable breakpoint, large files in seconds.
Uploading folders in Web projects is now a mainstream requirement. There are similar requirements in OA, or enterprise ERP systems. Uploading folders and keeping the hierarchy is a good way to guide users and make them easier to use. Can provide more advanced application support.
Folder data table structure
CREATE TABLE IF NOT EXISTS up6_folders (
f_id char(32) NOT NULL ,
f_nameLoc varchar(255) default ”,
f_pid char(32) default ”,
f_uid int(11) default ‘0’,
f_lenLoc bigint(19) default ‘0’,
f_sizeLoc varchar(50) default ‘0’,
f_pathLoc varchar(255) default ”,
f_pathSvr varchar(255) default ”,
f_pathRel varchar(255) default ”,
f_folders int(11) default ‘0’,
f_fileCount int(11) default ‘0’,
f_filesComplete int(11) default ‘0’,
f_complete tinyint(1) default ‘0’,
f_deleted tinyint(1) default ‘0’,
f_time timestamp NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
f_pidRoot char(32) default ”,
PRIMARY KEY (f_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
File data table structure
CREATE TABLE IF NOT EXISTS up6_files (
f_id char(32) NOT NULL,
F_pid char(32) default “, / Parent folder ID/
F_pidRoot CHAR (32) default “, / root folder ID/
F_fdTask TINyInt (1) default ‘0’, / Whether a folder information /
F_fdChild TinyInt (1) default ‘0’, / Whether file in folder /
f_uid int(11) default ‘0’,
F_nameLoc vARCHar (255) default “, / local name of the file (original file name) /
F_nameSvr vARCHar (255) default “, / File name in the server /
F_pathLoc vARCHar (512) default “, / The local path of the file /
F_pathSvr vARCHar (512) default “, / location of file in remote server /
f_pathRel varchar(512) default ”,
F_md5 varchar(40) default “, / file MD5/
F_lenLoc BIGInt (19) default ‘0’, / file size /
F_sizeLoc vARCHar (10) default ‘0’, / File size (formatted) /
F_pos bigint(19) default ‘0’, / continuation position /
F_lenSvr BigInt (19) default ‘0’, / Uploaded size /
F_perSvr vARCHar (7) default ‘0%’, / Upload percentage /
F_complete TinyInt (1) default ‘0’, / Whether upload completed /
f_time timestamp NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
f_deleted tinyint(1) default ‘0’,
f_scan tinyint(1) default ‘0’,
PRIMARY KEY (f_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
The core of the project is uploading files in chunks. The front and back ends need to be highly coordinated, and both parties need to agree on some data to complete large file blocks. We should focus on the following problems in the project.
* How to slice;
* How to compose a file;
* Interrupt which shard to start from.
How to split, using powerful JS libraries, to ease our work, there are already wheels on the market for large file chunks, although programmer nature once forced me to reinvent the wheel. But because of time and work, I had to let it go. Finally, I chose Baidu’s WebUploader to achieve the front-end requirements.
How do we do that? Before we do that, we have to solve the problem of how do we tell which file the partition belongs to. In the beginning, I used the front end to generate a unique UUID as a file identifier, which I carried on each shard request. However, I gave up the second transmission and adopted Md5 to maintain the relationship between blocks and files.
The problem of merging files and records into chunks on the server side has been solved by the industry. Refer to Xunlei, you will find that every time the download, there will be two files, a file body, the other is a file temporary file, temporary file stores the state of the corresponding byte bits of each block.
All of these need to be closely connected with the front and back ends. The front end needs to fragment files according to a fixed size, and the request must be accompanied by the fragment number and size. After the front-end sends the request to the background smoothly, the server only needs to calculate the starting position according to the fragment number and the size of each fragment given in the request data (the fragment size is fixed and the same), and write the read file fragment data into the file.
In order to facilitate development, I have divided the service side business logic into initialization, block processing, file upload completion and so on.
The service logic module of the server is as follows
Functional analysis:
Folder generation module
After the folder is uploaded, the scan code of the server is as follows
public class fd_scan
{
DbHelper db;
Connection con;
PreparedStatement cmd_add_f = null;
PreparedStatement cmd_add_fd = null;
public FileInf root = null; / / the root node
public fd_scan()
{
this.db = new DbHelper();
this.con = this.db.GetCon();
}
public void makeCmdF()
{
StringBuilder sb = new StringBuilder();
sb.append(“insert into up6_files (“);
sb.append(” f_id”); / / 1
sb.append(“,f_pid”); / / 2
sb.append(“,f_pidRoot”); / / 3
sb.append(“,f_fdTask”); / / 4
sb.append(“,f_fdChild”); / / 5
sb.append(“,f_uid”); / / 6
sb.append(“,f_nameLoc”); / / 7
sb.append(“,f_nameSvr”); / / 8
sb.append(“,f_pathLoc”); / / 9
sb.append(“,f_pathSvr”); / / 10
sb.append(“,f_pathRel”); / / 11
sb.append(“,f_md5”); / / 12
sb.append(“,f_lenLoc”); / / 13
sb.append(“,f_sizeLoc”); / / 14
sb.append(“,f_lenSvr”); / / 15
sb.append(“,f_perSvr”); / / 16
sb.append(“,f_complete”); / / 17
sb.append(“) values(“);
sb.append(” ?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“)”);
try {
this.cmd_add_f = this.con.prepareStatement(sb.toString());
this.cmd_add_f.setString(1, “”); //id
this.cmd_add_f.setString(2, “”); //pid
this.cmd_add_f.setString(3, “”); //pidRoot
this.cmd_add_f.setBoolean(4, true); //fdTask
this.cmd_add_f.setBoolean(5, false); //f_fdChild
this.cmd_add_f.setInt(6, 0); //f_uid
this.cmd_add_f.setString(7, “”); //f_nameLoc
this.cmd_add_f.setString(8, “”); //f_nameSvr
this.cmd_add_f.setString(9, “”); //f_pathLoc
this.cmd_add_f.setString(10, “”); //f_pathSvr
this.cmd_add_f.setString(11, “”); //f_pathRel
this.cmd_add_f.setString(12, “”); //f_md5
this.cmd_add_f.setLong(13, 0); //f_lenLoc
this.cmd_add_f.setString(14, “”); //f_sizeLoc
this.cmd_add_f.setLong(15, 0); //f_lenSvr
this.cmd_add_f.setString(16, “”); //f_perSvr
this.cmd_add_f.setBoolean(17, true); //f_complete
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void makeCmdFD()
{
StringBuilder sb = new StringBuilder();
sb.append(“insert into up6_folders (“);
sb.append(” f_id”); / / 1
sb.append(“,f_pid”); / / 2
sb.append(“,f_pidRoot”); / / 3
sb.append(“,f_nameLoc”); / / 4
sb.append(“,f_uid”); / / 5
sb.append(“,f_pathLoc”); / / 6
sb.append(“,f_pathSvr”); / / 7
sb.append(“,f_pathRel”); / / 8
sb.append(“,f_complete”); / / 9
sb.append(“) values(“); //
sb.append(” ?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“,?” );
sb.append(“)”);
try {
this.cmd_add_fd = this.con.prepareStatement(sb.toString());
this.cmd_add_fd.setString(1, “”); //id
this.cmd_add_fd.setString(2, “”); //pid
this.cmd_add_fd.setString(3, “”); //pidRoot
this.cmd_add_fd.setString(4, “”); //name
this.cmd_add_fd.setInt(5, 0); //f_uid
this.cmd_add_fd.setString(6, “”); //pathLoc
this.cmd_add_fd.setString(7, “”); //pathSvr
this.cmd_add_fd.setString(8, “”); //pathRel
this.cmd_add_fd.setBoolean(9, true); //complete
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
protected void GetAllFiles(FileInf inf,String root)
{
File dir = new File(inf.pathSvr);
File [] allFile = dir.listFiles();
for(int i = 0; i < allFile.length; i++)
{
if(allFile[i].isDirectory())
{
FileInf fd = new FileInf();
String uuid = UUID.randomUUID().toString();
uuid = uuid.replace(“-“, “”);
fd.id = uuid;
fd.pid = inf.id;
fd.pidRoot = this.root.id;
fd.nameSvr = allFile[i].getName();
fd.nameLoc = fd.nameSvr;
fd.pathSvr = allFile[i].getPath();
fd.pathSvr = fd.pathSvr.replace(“\”, “/”);
fd.pathRel = fd.pathSvr.substring(root.length() + 1);
fd.perSvr = “100%”;
fd.complete = true;
this.save_folder(fd);
this.GetAllFiles(fd, root);
}
else
{
FileInf fl = new FileInf();
String uuid = UUID.randomUUID().toString();
uuid = uuid.replace(“-“, “”);
fl.id = uuid;
fl.pid = inf.id;
fl.pidRoot = this.root.id;
fl.nameSvr = allFile[i].getName();
fl.nameLoc = fl.nameSvr;
fl.pathSvr = allFile[i].getPath();
fl.pathSvr = fl.pathSvr.replace(“\”, “/”);
fl.pathRel = fl.pathSvr.substring(root.length() + 1);
fl.lenSvr = allFile[i].length();
fl.lenLoc = fl.lenSvr;
fl.perSvr = “100%”;
fl.complete = true;
this.save_file(fl);
}
}
}
protected void save_file(FileInf f)
{
try {
this.cmd_add_f.setString(1, f.id); //id
this.cmd_add_f.setString(2, f.pid); //pid
this.cmd_add_f.setString(3, f.pidRoot); //pidRoot
this.cmd_add_f.setBoolean(4, f.fdTask); //fdTask
this.cmd_add_f.setBoolean(5, true); //f_fdChild
this.cmd_add_f.setInt(6, f.uid); //f_uid
this.cmd_add_f.setString(7, f.nameLoc); //f_nameLoc
this.cmd_add_f.setString(8, f.nameSvr); //f_nameSvr
this.cmd_add_f.setString(9, f.pathLoc); //f_pathLoc
this.cmd_add_f.setString(10, f.pathSvr); //f_pathSvr
this.cmd_add_f.setString(11, f.pathRel); //f_pathRel
this.cmd_add_f.setString(12, f.md5); //f_md5
this.cmd_add_f.setLong(13, f.lenLoc); //f_lenLoc
this.cmd_add_f.setString(14, f.sizeLoc); //f_sizeLoc
this.cmd_add_f.setLong(15, f.lenSvr); //f_lenSvr
this.cmd_add_f.setString(16, f.perSvr); //f_perSvr
this.cmd_add_f.setBoolean(17, f.complete); //f_complete
this.cmd_add_f.executeUpdate();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} / /
}
protected void save_folder(FileInf f)
{
try {
this.cmd_add_fd.setString(1, f.id); //id
this.cmd_add_fd.setString(2, f.pid); //pid
this.cmd_add_fd.setString(3, f.pidRoot); //pidRoot
this.cmd_add_fd.setString(4, f.nameSvr); //name
this.cmd_add_fd.setInt(5, f.uid); //f_uid
this.cmd_add_fd.setString(6, f.pathLoc); //pathLoc
this.cmd_add_fd.setString(7, f.pathSvr); //pathSvr
this.cmd_add_fd.setString(8, f.pathRel); //pathRel
this.cmd_add_fd.setBoolean(9, f.complete); //complete
this.cmd_add_fd.executeUpdate();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void scan(FileInf inf, String root) throws IOException, SQLException
{
this.makeCmdF();
this.makeCmdFD();
this.GetAllFiles(inf, root);
this.cmd_add_f.close();
this.cmd_add_fd.close();
this.con.close();
}
}
Upload block, block processing logic should be the most simple logic, up6 have put the file on the block, and each block of data for the identification, the identification indexes including file blocks, size, migration, file MD5, file MD5 (open) and other information, the service side after receiving this information can be very convenient to deal with. For example, storing block data in a distributed storage system
Block uploads are the basis of our whole project, such as resumable, pause and so on.
This one is relatively simple. Front end is the use of webuploader, block and other basic functions have been encapsulated, easy to use.
With the file API webUpload provides, the front end is surprisingly simple.
What we divide we unite. Shards large files, but shards do not have the original file function, so we need to synthesize the shards into the original file. All we need to do is write the shard into the file in its original position. Since we have already covered the principle, we know the size and number of the partition, so I can know where the partition starts in the file. So it makes sense to use RandomAccessFile, which can be moved back and forth within the file. But most of the functionality in andomAccessFile has been replaced by the “memory-mapped files” of NIO in JDK1.4. In this project I wrote a composite file using RandomAccessFile and MappedByteBuffer respectively. The corresponding method is uploadFileRandomAccessFile and uploadFileByMappedByteBuffer respectively. The code for the two methods is as follows.
A second transfer function
Server logic
Second transfer function, I believe that we have reflected the network disk upload, found that the file uploaded in seconds. In fact, those who have studied the principle should know that in fact, it is to check the MD5 of files, record the MD5 of files uploaded to the system, obtain the MD5 value of file content or partial value before uploading a file, and then match the data on the system.
Breakpoint-http implements the principle of second transmission. When the client selects a file and clicks upload, it triggers to obtain the MD5 value of the file. After obtaining the MD5 value, it calls a system interface (/index/checkFileMd5) to check whether the MD5 value already exists. Use the MD5 value of the file as the key. Value is the address where the file is stored. The interface returns to check the status before proceeding to the next step. I think you can see it in the code.
Well, the front-end MD5 values also use webuploader functionality, which is a good tool.
The md5_complete event is triggered after the control has calculated the MD5 of the file and the MD5 value is passed. The developer only needs to handle this event.
Breakpoint continuingly
Up6 already handles breakpoint continuations automatically and does not need to be handled separately.
These parameters are received and processed in f_post.jsp, and the developer needs to focus only on the business logic and nothing else.
Resumable upload refers to the failure of uploading a file halfway due to interruption caused by human factors (pause) or force majeure (disconnection or poor network). Then, when the environment recovers, upload the file again, rather than from a fresh start.
As mentioned earlier, the function of resumable breakpoint is based on the implementation of the transfer on the block, a large file is divided into many small pieces, the server can upload each successful block to the ground down, the client at the beginning of the upload file call interface quick verification, conditional selection to skip a certain block.
/index/checkFileMd5 = /index/checkFileMd5 = /index/checkFileMd5 = /index/checkFileMd5 = /index/checkFileMd5 And then the front end does a conditional screening to figure out which chunks are not uploaded, and then upload them.
Refer to the article: blog.ncmem.com/wordpress/2…
Welcome to join the group to discuss “374992201”