Based on the mongo GridFS
As the name suggests, the GridFS is a specification for storing files in a grid format. The GridFS divides files into multiple blocks, and each block acts as a separate document. By default, the block size defaults to 255kB, meaning that the document is stored in multiple 255kB blocks except for the last one. The GridFS uses two collections to store file information, one for file content (fs.chunks) and one for file metadata (fs.files). GridFS is also introduced to solve the problem that a single Document cannot exceed 4M(the new version is 16M). You can change the size of a single file block by configuring the GridFS. In this way, files smaller than 4M(16M) can be prevented from being divided into blocks and the read and write performance of small files can be improved.
The illustration
We upload a normal file to the GridFS for storage, which generates two sets, fs.files and fs.chunks, in a one-to-many relationship. The _id in fs.files corresponds to files_id in fs.chunks, and the fs.files record corresponds to multiple Fs. chunks. Fs. files stores the description information (size, name, format, etc.) of the file. There is an N field in the Fs. chunks table that represents the sequence of the chunks. The GridFS reads all chunks of the file and sorts them according to the N field. The binary contents of each chunk are pieced together so that the file contents are restored. Of course, the MongoDB driver does this for us.
- Fs.files field and description (extendable field)
| field | describe |
|---|---|
| _id | The file ID is recommended to be generated using uUID when storing the file. The _id is generated by MongoDB by default and corresponds to an object structure in Java, which is not conducive to interaction |
| filename | The file name |
| chunkSize | Size of a single chunk (mode 255KB (261120b)) |
| uploadDate | Upload time |
| aliases | The alias |
| md5 | Document verification to ensure safety |
| length | The file size |
In addition to these default fields, the fs.file set can be extended to add uploadUser (uploadUser) and suffix(file suffix) to facilitate type filtering on file management pages
- Fs. chunks field and description (not expandable)
| field | describe |
|---|---|
| _id | Unique ID, automatically generated by MongoDB |
| files_id | Corresponds to the _id in fs.files |
| data | File block binary content |
| n | The sequence number of the current chunk |
application
The framework
<! -- springboot -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.7. RELEASE</version>
</parent>
<! -- mongodb -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Copy the codeThe operations
For your reading convenience, the business processes are placed in the interface
upload
Interface implementation
@PostMapping("upload")
public String upload(HttpServletRequest request) {
MultipartFile file = ((MultipartHttpServletRequest) request).getFile("upload_file");
try {
GridFS fs = new GridFS(mongoTemplate.getDb());
// Generate file ID
String fileId = UUID.randomUUID().toString().replace("-"."");
// Create a file object
GridFSInputFile gif = fs.createFile(file.getInputStream());
gif.setId(fileId);
String filename = file.getOriginalFilename();
// Set the file name
gif.setFilename(filename);
/ / type
gif.setContentType(file.getContentType());
// File extension TXT PDF XLS...
String suffix = filename.split("\ \.") [1];
// Metadata can put extended fields in this
gif.setMetaData(new BasicDBObject("suffix", type));
gif.save();
return fileId;
} catch (IOException ex) {
ex.printStackTrace();
}
return null;
}
Copy the codeInvoke the sample
Upload mulan words.txt
Chirp, chirp, mulan knit. Hear not the sound of the loom, but the sigh of the woman. Ask a woman what she thinks, ask a woman what she remembers. Women have no thoughts, women have no memory. Last night saw the army post, Khan big point soldiers, army book twelve volumes, volumes have ye name. Ye no big son, magnolia no elder brother, is willing to saddle horse for the city, from now on for ye zheng. Buy a horse to the east, a saddle - cloth to the west, a bridle to the south and a whip to the north. Denier word ye niang go, evening stay Yellow River edge, do not smell ye Niang call female voice, but smell the Yellow River water splatter. Dan words to the Yellow River, dusk to black mountain head, do not smell ye Niang call female voice, but smell Yanshan Hu Qi chiru. Thousands of miles to the army, if guan Shan fly. New gas pass gold watchman, cold light iron clothes. A general dies in battle, a strong man returns ten years. Return to see the son of Heaven, the son of Heaven sitting in the hall. Policy xun twelve turn, reward hundred thousand strong. Khan asked, Mulan not shangshu Lang, willing to chi li foot, send son also hometown. Ye Niang smell female, guo xiangfu will; Sister wenmei, when the household red makeup; Little brother smell elder sister come, whet knife huo Huo to pig sheep. Open my east court door, sit on my west court bed, take off my war robes, wear my old clothes. When the window, on the mirror post yellow. Go out to see companions, companions are busy: peer twelve years, I do not know Mulan is a girl. Male rabbit feet flapping, female rabbit eyes blurred; Can Ann tell whether I am a male or a female?Copy the codeExample Postman call
Fs database files
fs.chunks
download
Interface implementation
@GetMapping("/download/{fileId}") public void download(@PathVariable String fileId, HttpServletRequest request, HttpServletResponse response) { GridFS fs = new GridFS(mongoTemplate.getDb()); GridFSDBFile file = fs.findone (new BasicDBObject("_id", fileId)); if (file ! = null) {// try resurce appears in a manner that automatically closes the stream without having to deal with the try (BufferedOutputStream bos = new BufferedOutputStream(response.getOutputStream()); InputStream is = file.getInputStream(); BufferedInputStream bis = new BufferedInputStream(is);) String suffix = (String) file.getmetadata ().get("suffix"); response.setContentType(SVG.equals(suffix) ? "image/svg+xml; charset=UTF-8" : "application/octet-stream; charset=UTF-8"); response.addHeader("Content-Disposition", "attachment; " + createContentDisposition(request, file.getFilename())); response.setContentLength((int) file.getLength()); int length = 0; byte[] temp = new byte[2048]; while ((length = bis.read(temp)) ! = -1) { bos.write(temp, 0, length); } } catch (IOException e) { e.printStackTrace(); Content-disposition ** @param Request * @param filename * @return */ private String createContentDisposition(HttpServletRequest request, String filename) { UserAgent userAgent = UserAgentUtil.parse(request.getHeader("User-Agent")); Browser browser = userAgent.getBrowser(); try { String disposition = URLEncoder.encode(filename, StandardCharsets.UTF_8.name()); if (browser.isMatch("firefox") || browser.isMatch("applewebkit") || browser.isMatch("safari")) { disposition = new String(filename.getBytes(StandardCharsets.UTF_8.name()), StandardCharsets.ISO_8859_1.name()); } return browser.isMatch("opera") ? "filename*=UTF-8''" + disposition : "filename = "" + disposition + """; } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return null; } private final static Map<String, String> FILE_CONTENT_MAP = new HashMap<>(16); static { FILE_CONTENT_MAP.put("doc", "application/msword; charset=UTF-8"); FILE_CONTENT_MAP.put("docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=UTF-8"); FILE_CONTENT_MAP.put("xls", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xlt", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xla", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet; charset=UTF-8"); FILE_CONTENT_MAP.put("xltx", "application/vnd.openxmlformats-officedocument.spreadsheetml.template; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsm", "application/vnd.ms-excel.sheet.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xltm", "application/vnd.ms-excel.template.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xlam", "application/vnd.ms-excel.addin.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsb", "application/vnd.ms-excel.sheet.binary.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("ppt", "application/vnd.ms-powerpoint; charset=UTF-8"); FILE_CONTENT_MAP.put("pptx", "application/vnd.openxmlformats-officedocument.presentationml.presentation; charset=UTF-8"); } private static final String SVG = "svg";Copy the codeInvoke the sample
Browser: Enter the download interface address + file ID
Delete the file
Interface implementation
@DeleteMapping("/{fileId}")
public void delete(@PathVariable String fileId) {
GridFS fs = new GridFS(mongoTemplate.getDb());
fs.remove(fileId);
}
Copy the codeThe appendix
In order to facilitate the need of friends to use and study, the appendix complete code, praise points praise (PS: whenever we online research (CV) code implementation, always change back to change, CTRL + C here is done)
package com.demo.fsserver.api;
import cn.hutool.http.useragent.Browser;
import cn.hutool.http.useragent.UserAgent;
import cn.hutool.http.useragent.UserAgentUtil;
import com.mongodb.BasicDBObject;
import com.mongodb.gridfs.GridFS;
import com.mongodb.gridfs.GridFSDBFile;
import com.mongodb.gridfs.GridFSInputFile;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.MultipartHttpServletRequest;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
@RestController
@RequestMapping("/fs/api")
public class FsController {
@Autowired
private MongoTemplate mongoTemplate;
/**
* 上传
*
* @param request
* @return
*/
@PostMapping("upload")
public String upload(HttpServletRequest request) {
MultipartFile file = ((MultipartHttpServletRequest) request).getFile("upload_file");
try {
GridFS fs = new GridFS(mongoTemplate.getDb());
String fileId = UUID.randomUUID().toString().replace("-", "");
GridFSInputFile gif = fs.createFile(file.getInputStream());
gif.setId(fileId);
String filename = file.getOriginalFilename();
gif.setFilename(filename);
String suffix = filename.split("\.")[1];
gif.setContentType(file.getContentType());
gif.setMetaData(new BasicDBObject("suffix", suffix));
gif.save();
return fileId;
} catch (IOException ex) {
ex.printStackTrace();
}
return null;
}
/**
* 下载文件
*
* @param fileId
* @param request
* @param response
*/
@GetMapping("/download/{fileId}")
public void download(@PathVariable String fileId, HttpServletRequest request, HttpServletResponse response) {
GridFS fs = new GridFS(mongoTemplate.getDb());
GridFSDBFile file = fs.findOne(new BasicDBObject("_id", fileId));
if (file != null) {
try (BufferedOutputStream bos = new BufferedOutputStream(response.getOutputStream());
InputStream is = file.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);) {
String suffix = (String) file.getMetaData().get("suffix");
response.setContentType(SVG.equals(suffix) ? "image/svg+xml;charset=UTF-8" : "application/octet-stream;charset=UTF-8");
response.addHeader("Content-Disposition",
"attachment; " + createContentDisposition(request, file.getFilename()));
response.setContentLength((int) file.getLength());
int length = 0;
byte[] temp = new byte[2048];
while ((length = bis.read(temp)) != -1) {
bos.write(temp, 0, length);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* 创建 Content-Disposition
*
* @param request
* @param filename
* @return
*/
private String createContentDisposition(HttpServletRequest request, String filename) {
UserAgent userAgent = UserAgentUtil.parse(request.getHeader("User-Agent"));
Browser browser = userAgent.getBrowser();
try {
String disposition = URLEncoder.encode(filename, StandardCharsets.UTF_8.name());
if (browser.isMatch("firefox") || browser.isMatch("applewebkit") || browser.isMatch("safari")) {
disposition = new String(filename.getBytes(StandardCharsets.UTF_8.name()), StandardCharsets.ISO_8859_1.name());
}
return browser.isMatch("opera") ? "filename*=UTF-8''" + disposition : "filename = "" + disposition + """;
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return null;
}
/**
* 删除文件
*
* @param fileId
*/
@DeleteMapping("/{fileId}")
public void delete(@PathVariable String fileId) {
GridFS fs = new GridFS(mongoTemplate.getDb());
fs.remove(fileId);
}
private final static Map<String, String> FILE_CONTENT_MAP = new HashMap<>(16);
static {
FILE_CONTENT_MAP.put("doc", "application/msword;charset=UTF-8");
FILE_CONTENT_MAP.put("docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document;charset=UTF-8");
FILE_CONTENT_MAP.put("xls", "application/vnd.ms-excel;charset=UTF-8");
FILE_CONTENT_MAP.put("xlt", "application/vnd.ms-excel;charset=UTF-8");
FILE_CONTENT_MAP.put("xla", "application/vnd.ms-excel;charset=UTF-8");
FILE_CONTENT_MAP.put("xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8");
FILE_CONTENT_MAP.put("xltx", "application/vnd.openxmlformats-officedocument.spreadsheetml.template;charset=UTF-8");
FILE_CONTENT_MAP.put("xlsm", "application/vnd.ms-excel.sheet.macroEnabled.12;charset=UTF-8");
FILE_CONTENT_MAP.put("xltm", "application/vnd.ms-excel.template.macroEnabled.12;charset=UTF-8");
FILE_CONTENT_MAP.put("xlam", "application/vnd.ms-excel.addin.macroEnabled.12;charset=UTF-8");
FILE_CONTENT_MAP.put("xlsb", "application/vnd.ms-excel.sheet.binary.macroEnabled.12;charset=UTF-8");
FILE_CONTENT_MAP.put("ppt", "application/vnd.ms-powerpoint;charset=UTF-8");
FILE_CONTENT_MAP.put("pptx", "application/vnd.openxmlformats-officedocument.presentationml.presentation;charset=UTF-8");
}
private static final String SVG = "svg";
}
Copy the codeApplication of summary
Fast development of three interfaces, upload, download, delete, the amount of code is not very small, in the service applied to MongoDB, to achieve a storage service is so simple, fs.files can freely expand fields in mateData, no need to redesign the business table; The code has been tested and optimized, and can be used or debugged by friends who need it
other
You can also read my other articles if you are interested. ^
MongoDB stores query schemes in separate tables
Redis cache and local cache design practices
My home page