Based on the mongo GridFS

As the name suggests, the GridFS is a specification for storing files in a grid format. The GridFS divides files into multiple blocks, and each block acts as a separate document. By default, the block size defaults to 255kB, meaning that the document is stored in multiple 255kB blocks except for the last one. The GridFS uses two collections to store file information, one for file content (fs.chunks) and one for file metadata (fs.files). GridFS is also introduced to solve the problem that a single Document cannot exceed 4M(the new version is 16M). You can change the size of a single file block by configuring the GridFS. In this way, files smaller than 4M(16M) can be prevented from being divided into blocks and the read and write performance of small files can be improved.

The illustration

We upload a normal file to the GridFS for storage, which generates two sets, fs.files and fs.chunks, in a one-to-many relationship. The _id in fs.files corresponds to files_id in fs.chunks, and the fs.files record corresponds to multiple Fs. chunks. Fs. files stores the description information (size, name, format, etc.) of the file. There is an N field in the Fs. chunks table that represents the sequence of the chunks. The GridFS reads all chunks of the file and sorts them according to the N field. The binary contents of each chunk are pieced together so that the file contents are restored. Of course, the MongoDB driver does this for us.

• Fs.files field and description (extendable field)

In addition to these default fields, the fs.file set can be extended to add uploadUser (uploadUser) and suffix(file suffix) to facilitate type filtering on file management pages

• Fs. chunks field and description (not expandable)

application

The framework

<! -- springboot -->
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.7. RELEASE</version>
</parent>
<! -- mongodb -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Copy the code

The operations

For your reading convenience, the business processes are placed in the interface

upload

Interface implementation

@PostMapping("upload")
public String upload(HttpServletRequest request) {
    MultipartFile file = ((MultipartHttpServletRequest) request).getFile("upload_file");
    try {
        GridFS fs = new GridFS(mongoTemplate.getDb());
        // Generate file ID
        String fileId = UUID.randomUUID().toString().replace("-"."");
        // Create a file object
        GridFSInputFile gif = fs.createFile(file.getInputStream());
        gif.setId(fileId);
        String filename = file.getOriginalFilename();
        // Set the file name
        gif.setFilename(filename);
        / / type
        gif.setContentType(file.getContentType());
        // File extension TXT PDF XLS...
        String suffix = filename.split("\ \.") [1];
        // Metadata can put extended fields in this
        gif.setMetaData(new BasicDBObject("suffix", type));
        gif.save();
        return fileId;
    } catch (IOException ex) {
        ex.printStackTrace();
    }
    return null;
}
Copy the code

Invoke the sample

Upload mulan words.txt

Chirp, chirp, mulan knit. Hear not the sound of the loom, but the sigh of the woman. Ask a woman what she thinks, ask a woman what she remembers. Women have no thoughts, women have no memory. Last night saw the army post, Khan big point soldiers, army book twelve volumes, volumes have ye name. Ye no big son, magnolia no elder brother, is willing to saddle horse for the city, from now on for ye zheng. Buy a horse to the east, a saddle - cloth to the west, a bridle to the south and a whip to the north. Denier word ye niang go, evening stay Yellow River edge, do not smell ye Niang call female voice, but smell the Yellow River water splatter. Dan words to the Yellow River, dusk to black mountain head, do not smell ye Niang call female voice, but smell Yanshan Hu Qi chiru. Thousands of miles to the army, if guan Shan fly. New gas pass gold watchman, cold light iron clothes. A general dies in battle, a strong man returns ten years. Return to see the son of Heaven, the son of Heaven sitting in the hall. Policy xun twelve turn, reward hundred thousand strong. Khan asked, Mulan not shangshu Lang, willing to chi li foot, send son also hometown. Ye Niang smell female, guo xiangfu will; Sister wenmei, when the household red makeup; Little brother smell elder sister come, whet knife huo Huo to pig sheep. Open my east court door, sit on my west court bed, take off my war robes, wear my old clothes. When the window, on the mirror post yellow. Go out to see companions, companions are busy: peer twelve years, I do not know Mulan is a girl. Male rabbit feet flapping, female rabbit eyes blurred; Can Ann tell whether I am a male or a female?Copy the code

Example Postman call

Fs database files

fs.chunks

download

Interface implementation

@GetMapping("/download/{fileId}") public void download(@PathVariable String fileId, HttpServletRequest request, HttpServletResponse response) { GridFS fs = new GridFS(mongoTemplate.getDb()); GridFSDBFile file = fs.findone (new BasicDBObject("_id", fileId)); if (file ! = null) {// try resurce appears in a manner that automatically closes the stream without having to deal with the try (BufferedOutputStream bos = new BufferedOutputStream(response.getOutputStream()); InputStream is = file.getInputStream(); BufferedInputStream bis = new BufferedInputStream(is);) String suffix = (String) file.getmetadata ().get("suffix"); response.setContentType(SVG.equals(suffix) ? "image/svg+xml; charset=UTF-8" : "application/octet-stream; charset=UTF-8"); response.addHeader("Content-Disposition", "attachment; " + createContentDisposition(request, file.getFilename())); response.setContentLength((int) file.getLength()); int length = 0; byte[] temp = new byte[2048]; while ((length = bis.read(temp)) ! = -1) { bos.write(temp, 0, length); } } catch (IOException e) { e.printStackTrace(); Content-disposition ** @param Request * @param filename * @return */ private String createContentDisposition(HttpServletRequest request, String filename) { UserAgent userAgent = UserAgentUtil.parse(request.getHeader("User-Agent")); Browser browser = userAgent.getBrowser(); try { String disposition = URLEncoder.encode(filename, StandardCharsets.UTF_8.name()); if (browser.isMatch("firefox") || browser.isMatch("applewebkit") || browser.isMatch("safari")) { disposition = new String(filename.getBytes(StandardCharsets.UTF_8.name()), StandardCharsets.ISO_8859_1.name()); } return browser.isMatch("opera") ? "filename*=UTF-8''" + disposition : "filename = "" + disposition + """; } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return null; } private final static Map<String, String> FILE_CONTENT_MAP = new HashMap<>(16); static { FILE_CONTENT_MAP.put("doc", "application/msword; charset=UTF-8"); FILE_CONTENT_MAP.put("docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=UTF-8"); FILE_CONTENT_MAP.put("xls", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xlt", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xla", "application/vnd.ms-excel; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet; charset=UTF-8"); FILE_CONTENT_MAP.put("xltx", "application/vnd.openxmlformats-officedocument.spreadsheetml.template; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsm", "application/vnd.ms-excel.sheet.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xltm", "application/vnd.ms-excel.template.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xlam", "application/vnd.ms-excel.addin.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("xlsb", "application/vnd.ms-excel.sheet.binary.macroEnabled.12; charset=UTF-8"); FILE_CONTENT_MAP.put("ppt", "application/vnd.ms-powerpoint; charset=UTF-8"); FILE_CONTENT_MAP.put("pptx", "application/vnd.openxmlformats-officedocument.presentationml.presentation; charset=UTF-8"); } private static final String SVG = "svg";Copy the code

Invoke the sample

Browser: Enter the download interface address + file ID

Delete the file

Interface implementation

@DeleteMapping("/{fileId}")
public void delete(@PathVariable String fileId) {
    GridFS fs = new GridFS(mongoTemplate.getDb());
    fs.remove(fileId);
}
Copy the code

The appendix

In order to facilitate the need of friends to use and study, the appendix complete code, praise points praise (PS: whenever we online research (CV) code implementation, always change back to change, CTRL + C here is done)

package com.demo.fsserver.api;

import cn.hutool.http.useragent.Browser;
import cn.hutool.http.useragent.UserAgent;
import cn.hutool.http.useragent.UserAgentUtil;
import com.mongodb.BasicDBObject;
import com.mongodb.gridfs.GridFS;
import com.mongodb.gridfs.GridFSDBFile;
import com.mongodb.gridfs.GridFSInputFile;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.MultipartHttpServletRequest;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

@RestController
@RequestMapping("/fs/api")
public class FsController {

    @Autowired
    private MongoTemplate mongoTemplate;

    /**
     * 上传
     *
     * @param request
     * @return
     */
    @PostMapping("upload")
    public String upload(HttpServletRequest request) {
        MultipartFile file = ((MultipartHttpServletRequest) request).getFile("upload_file");
        try {
            GridFS fs = new GridFS(mongoTemplate.getDb());
            String fileId = UUID.randomUUID().toString().replace("-", "");
            GridFSInputFile gif = fs.createFile(file.getInputStream());
            gif.setId(fileId);
            String filename = file.getOriginalFilename();
            gif.setFilename(filename);
            String suffix = filename.split("\.")[1];
            gif.setContentType(file.getContentType());
            gif.setMetaData(new BasicDBObject("suffix", suffix));
            gif.save();
            return fileId;
        } catch (IOException ex) {
            ex.printStackTrace();
        }
        return null;
    }

    /**
     * 下载文件
     *
     * @param fileId
     * @param request
     * @param response
     */
    @GetMapping("/download/{fileId}")
    public void download(@PathVariable String fileId, HttpServletRequest request, HttpServletResponse response) {
        GridFS fs = new GridFS(mongoTemplate.getDb());
        GridFSDBFile file = fs.findOne(new BasicDBObject("_id", fileId));
        if (file != null) {
            try (BufferedOutputStream bos = new BufferedOutputStream(response.getOutputStream());
                 InputStream is = file.getInputStream();
                 BufferedInputStream bis = new BufferedInputStream(is);) {
                String suffix = (String) file.getMetaData().get("suffix");
                response.setContentType(SVG.equals(suffix) ? "image/svg+xml;charset=UTF-8" : "application/octet-stream;charset=UTF-8");
                response.addHeader("Content-Disposition",
                        "attachment; " + createContentDisposition(request, file.getFilename()));
                response.setContentLength((int) file.getLength());
                int length = 0;
                byte[] temp = new byte[2048];
                while ((length = bis.read(temp)) != -1) {
                    bos.write(temp, 0, length);
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

    }

    /**
     * 创建 Content-Disposition
     *
     * @param request
     * @param filename
     * @return
     */
    private String createContentDisposition(HttpServletRequest request, String filename) {
        UserAgent userAgent = UserAgentUtil.parse(request.getHeader("User-Agent"));
        Browser browser = userAgent.getBrowser();
        try {
            String disposition = URLEncoder.encode(filename, StandardCharsets.UTF_8.name());
            if (browser.isMatch("firefox") || browser.isMatch("applewebkit") || browser.isMatch("safari")) {
                disposition = new String(filename.getBytes(StandardCharsets.UTF_8.name()), StandardCharsets.ISO_8859_1.name());
            }
            return browser.isMatch("opera") ? "filename*=UTF-8''" + disposition : "filename = "" + disposition + """;
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        return null;
    }


    /**
     * 删除文件
     *
     * @param fileId
     */
    @DeleteMapping("/{fileId}")
    public void delete(@PathVariable String fileId) {
        GridFS fs = new GridFS(mongoTemplate.getDb());
        fs.remove(fileId);
    }

    private final static Map<String, String> FILE_CONTENT_MAP = new HashMap<>(16);

    static {
        FILE_CONTENT_MAP.put("doc", "application/msword;charset=UTF-8");
        FILE_CONTENT_MAP.put("docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document;charset=UTF-8");
        FILE_CONTENT_MAP.put("xls", "application/vnd.ms-excel;charset=UTF-8");
        FILE_CONTENT_MAP.put("xlt", "application/vnd.ms-excel;charset=UTF-8");
        FILE_CONTENT_MAP.put("xla", "application/vnd.ms-excel;charset=UTF-8");
        FILE_CONTENT_MAP.put("xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8");
        FILE_CONTENT_MAP.put("xltx", "application/vnd.openxmlformats-officedocument.spreadsheetml.template;charset=UTF-8");
        FILE_CONTENT_MAP.put("xlsm", "application/vnd.ms-excel.sheet.macroEnabled.12;charset=UTF-8");
        FILE_CONTENT_MAP.put("xltm", "application/vnd.ms-excel.template.macroEnabled.12;charset=UTF-8");
        FILE_CONTENT_MAP.put("xlam", "application/vnd.ms-excel.addin.macroEnabled.12;charset=UTF-8");
        FILE_CONTENT_MAP.put("xlsb", "application/vnd.ms-excel.sheet.binary.macroEnabled.12;charset=UTF-8");
        FILE_CONTENT_MAP.put("ppt", "application/vnd.ms-powerpoint;charset=UTF-8");
        FILE_CONTENT_MAP.put("pptx", "application/vnd.openxmlformats-officedocument.presentationml.presentation;charset=UTF-8");
    }

    private static final String SVG = "svg";
}
Copy the code

Application of summary

Fast development of three interfaces, upload, download, delete, the amount of code is not very small, in the service applied to MongoDB, to achieve a storage service is so simple, fs.files can freely expand fields in mateData, no need to redesign the business table; The code has been tested and optimized, and can be used or debugged by friends who need it

other

You can also read my other articles if you are interested. ^

MongoDB stores query schemes in separate tables

Redis cache and local cache design practices

My home page