How to build a distributed file management system with FastDFS

Speaking of distributed file management system, you may easily think of HDFS, GFS and other systems, the former is a part of Hadoop, the latter is a distributed file management system provided by Google. In addition, China’s Taobao and Tencent also have their own distributed File management systems, called TFS (Taobao File System and Tencent File System).

FastDFS is probably closer to our Java engineers than any of the distributed file management systems mentioned above, because file uploads are so common, and the combination of FastDFS and Nginx is the best way to build a standalone distributed file management system. So Songo is here today to talk a little bit about that.

If you are not familiar with how to upload files in a traditional development environment, please refer to this tutorial:

Spring Boot + Vue, hand in hand to teach you to do file upload

1. What is FastDFS

1.1 FastDFS profile

FastDFS is a lightweight distributed file management system developed by Yu Qing from Taobao in 2008. FastDFS is implemented in C language and supports Unix-like systems such as Linux, FreeBSD and MacOS. FastDFS is similar to Google FS. It is an application-level file system, not a general file system. It can only be accessed through a proprietary API, and currently provides C, Java SDK, and PHP extension SDK.

This open source software from the release so far, after decades, the vitality of this open source software is still strong, still highly respected in the industry, of course, this is also due to the author has been constantly improving the software.

FastDFS is specially designed for Internet applications. It solves the problem of large-capacity file storage and pursues high performance and scalability. It can be regarded as a file-based key/value storage system.

1.2 Why FastDFS

File management can be Easy in traditional enterprise-level development environments where high concurrency requirements are not high and data volumes may not be large.

But the Internet application visits a large amount of data, in the Internet application, we must consider to solve the problem of large file storage and high performance access, and FastDFS is particularly suitable for this thing, common picture storage, video storage, document storage and so on we can use FastDFS to do.

1.3 FastDFS architecture

As a distributed file management system, FastDFS has four main functions:

File storage
File synchronization
File upload
File download

This aspect of the function, basically can solve our common file management needs.

Here is a system architecture diagram from the FastDFS website:

As you can see from the image above, the FastDFS architecture consists of two parts, Tracker and Storage. Tracker is used to track files, which is like an index of files, and Storage is used to store files.

The file we upload is finally saved in Storage, and the metadata information of the file is saved in Tracker, through which load balancing can be achieved for Storage.

A Storage Cluster can be composed of multiple groups. Different groups do not communicate with each other. A group is equivalent to a small Cluster. Storage servers in a group synchronize files through connections to ensure high availability.

2. FastDFS installation

After introducing FastDFS, I believe you are ready to try it. Let’s take a look at FastDFS installation.

For the convenience of testing, I will not open multiple virtual machines. I will install Tracker and Storage on the same server.

We generally use FastDFS to upload pictures, and Nginx is generally used for the following image access after the successful upload, so HERE I will introduce the installation from three aspects:

The Tracker is installed
Storage installation
Nginx installation

2.1 the Tracker installation

To install, we first need to prepare an environment, two libraries and an installation package.

1. An environment

FastDFS is developed in C language, so before installing FastDFS, if you do not have the GCC environment, you need to install the following command:

yum install gcc-c++
Copy the code

2. Two libraries

Since FastDFS relies on the libevent library, install the libevent library as follows:

yum -y install libevent
Copy the code

The other library is libFastCommon, which is official with FastDFS and contains some of the base libraries needed to run FastDFS.

Libfastcommon: github.com/happyfish10…

GitHub is slow to access, so songgo has downloaded the installation file and put it on baidu’s web disk. You can reply to fastdfs to get the download link.

Copy the downloaded libfastcommon to /usr/local/ and run the following commands:

cd /usr/localThe tar - ZXVF V1.0.43. Tar. GzcdLibfastcommon - 1.0.43 /. / make. Sh. / make. Sh installCopy the code

3. An installation package

Next we download Tracker. Note that since Tracker and Storage are the same installation package, we can download it once (not again in Section 2.2).

The installation file can be downloaded from the FastDFS GitHub repository at github.com/happyfish10…

GitHub is slow to access, so songgo has downloaded the installation file and put it on baidu’s web disk. You can reply to fastdfs to get the download link.

Copy the downloaded files to the /usr/local directory and run the following commands to install them:

cd /usr/localThe tar - ZXVF V6.06. Tar. GzcdFastdfs - 6.06 /. / make. Sh. / make. Sh installCopy the code

After the installation is successful, run the following command to copy the configuration files in the conf directory of the installation directory to the /etc/fdfs directory:

cd conf/
cp ./* /etc/fdfs/
Copy the code

4. Configuration

Then go to /etc/fdfs/directory to configure:

Open the tracker.conf file:

vi tracker.conf
Copy the code

Modify the following configuration:

The default port is 22122, which can be changed according to actual needs. I will not change it here. Next, configure a directory for storing metadata (note that the directory exists).

5. Start

Next, run the following command to start the Tracker:

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
Copy the code

After that, our Tracker is installed successfully.

2.2 Storage installation

For simplicity, let’s build a Storage instance. Libevent and libfastcommon are also required for Storage installation. For example, libevent and libfastcommon are required for Storage installation.

The installation of Storage is the same as that of Tracker, and the execution commands are the same. Because I have installed Tracker and Storage on the same server, there is no need to execute the installation command (equivalent to the installation of Storage when Tracker is installed).

The only thing to do is to go to /etc/fdfs and configure Storage:

vi storage.conf
Copy the code

The tracker_server template has two addresses. We only have one address here. After configuration, remember to comment out the other one that is not used.

After the configuration, run the following command to start the Storage:

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
Copy the code

After these two startup are complete, we can now do file upload, but generally if it is an image file, we also need to provide an image access function, currently the best solution is of course Nginx, so we here together with Nginx configuration, and then test.

2.3 Nginx installation

Nginx is an important partner of FastDFS.

Nginx installation consists of two steps:

Install Nginx
Install fastdfs-nginx-module on Storage

The first step is simple, songo wrote a special article on Nginx literacy, so you can refer to Nginx installation directly here: Nginx minimalism tutorial!

Now let’s look at step two.

Download fastdfs-nginx-module at github.com/happyfish10…

GitHub is slow to access, so songgo has downloaded the installation file and put it on baidu’s web disk. You can reply to fastdfs to get the download link.

After the download is complete, copy the downloaded file to the /usr/local directory. Then go to the /usr/local directory and run the following commands:

cd /usr/localThe tar - ZXVF V1.22. Tar. GzCopy the code

Then copy the /usr/local/fastdfs-nginx-module-1.22/src/mod_fastdfs.conf file to the /etc/fdfs/directory and modify the contents of the file.

vi /etc/fdfs/mod_fastdfs.conf
Copy the code

Next, go back to the unzipped directory of the nginx installation file downloaded in step 1 and run the following command to reconfigure and compile the installation:

./configure --add-module=/usr/local/ fastdfs - nginx - module - 1.22 / SRC make make installCopy the code

After the installation is complete, modify the nginx configuration file as follows:

vi /usr/local/nginx/conf/nginx.conf
Copy the code

Configure nginx request forwarding here.

After the configuration is complete, nginx is started. If the following log is displayed, nginx is started successfully:

ngx_http_fastdfs_set pid=9908
Copy the code

Fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module

Fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module fastdfs-nginx-module

Storage consists of many groups, each group is a small cluster, in each group, the data will be synchronized, but if the data has not been synchronized, this time there is a request sent, what should I do? Fastdfs-nginx-module will help you get the file directly from the source Storage.

The installation was successful.

3.Java client invocation

With the installation successful, let’s test the file upload and download with the Java client.

First let’s create a normal Maven project and add the following dependencies:

<dependency>
    <groupId>net.oschina.zcx7878</groupId>
    <artifactId>fastdfs-client-java</artifactId>
    <version>1.27.0.0</version>
</dependency>
Copy the code

Then add the FastDFS configuration file fastdfs-client.properties to the resources directory of your project as follows:

fastdfs.connect_timeout_in_seconds = 5 fastdfs.network_timeout_in_seconds = 30 fastdfs.charset = UTF-8 fastdfs.http_anti_steal_token = false fastdfs.http_secret_key = FastDFS1234567890 fastdfs.http_tracker_http_port = 80 Fastdfs.tracker_servers = 192.168.91.128:22122 Fastdfs.connection_pool. enabled = true fastdfs.connection_pool.max_count_per_entry = 500 fastdfs.connection_pool.max_idle_time = 3600 fastdfs.connection_pool.max_wait_time_in_ms = 1000Copy the code

The configuration here is basically obvious, so I’m not going to explain each one. Set fastdfs. tracker_Servers, which is the address of Tracker, as required.

The fastdfs.http_secret_key configuration is ignored here, and I’ll explain it later.

3.1 Uploading Files

After the configuration is complete, the file upload code is as follows:

@Test
void testUpload(a) {
    try {
        ClientGlobal.initByProperties("fastdfs-client.properties");
        TrackerClient tracker = new TrackerClient();
        TrackerServer trackerServer = tracker.getConnection();
        StorageServer storageServer = null;
        StorageClient1 client = new StorageClient1(trackerServer, storageServer);
        NameValuePair nvp[] = null;
        // Upload to the file system
        String fileId = client.upload_file1("C:\\Users\\javaboy\\Pictures\\picpick\\1.png"."png",
                nvp);
        logger.info(fileId);
    } catch(Exception e) { e.printStackTrace(); }}Copy the code

Here, the configuration file is loaded, a TrackerClient object is constructed, a TrackerServer is obtained from this object, and an instance of StorageClient1 is created. NameValuePair stores the metadata of the file, using key/value if any, or null if none.

Finally, the client upload_file1 method is called to upload the file. The first parameter is the file path, the second parameter is the file extension, the third parameter is the file metadata information, and the return value of this method is the file access path. If you run this method, the following logs are displayed:

The 2020-02-29 17:46:03. 6184-017 the INFO [main] O.J.F astdfs. FastdfsApplicationTests: group1/M00/00/00/wKhbgF5aMteAWy0gAAJkI7-2yGk361.pngCopy the code

Group1 / M00/00/00 / wKhbgF5aMteAWy0gAAJkI7-2 ygk361. PNG is the path to the file, at this time, Type http://192.168.91.128/group1/M00/00/00/wKhbgF5aMteAWy0gAAJkI7-2yGk361.png in the browser can see the uploaded pictures.

3.2 Downloading Files

@Test
void testDownload(a) {
    try {
        ClientGlobal.initByProperties("fastdfs-client.properties");
        TrackerClient tracker = new TrackerClient();
        TrackerServer trackerServer = tracker.getConnection();
        StorageServer storageServer = null;
        StorageClient1 client = new StorageClient1(trackerServer, storageServer);
        byte[] bytes = client.download_file1("group1/M00/00/00/wKhbgF5aMteAWy0gAAJkI7-2yGk361.png");
        FileOutputStream fos = new FileOutputStream(new File("C:\\Users\\javaboy\\Pictures\\picpick\\666.png"));
        fos.write(bytes);
        fos.close();
    } catch(Exception e) { e.printStackTrace(); }}Copy the code

This code is easy to understand. Call the download_file1 method to get an array of bytes and write it to a local file through the IO stream.

4. Security issues

Now, anyone can access our server to upload files, which is definitely not possible. This problem can be solved by adding a token when uploading.

First we enable token verification on the server side:

vi /etc/fdfs/http.conf
Copy the code

After the configuration is complete, restart the server:

./nginx -s stop
./nginx
Copy the code

Next, prepare a method to get the token on the front end as follows:

@Test
public void getToken(a) throws Exception {
    int ts = (int) Instant.now().getEpochSecond();
    String token = ProtoCommon.getToken("M00/00/00/wKhbgF5aMteAWy0gAAJkI7-2yGk361.png", ts, "FastDFS1234567890");
    StringBuilder sb = new StringBuilder();
    sb.append("? token=").append(token);
    sb.append("&ts=").append(ts);
    System.out.println(sb.toString());
}
Copy the code

The protocommon.getToken method is used to obtain the token. The first parameter of this method is the id of the file you want to access. ** The address does not contain group. ** The second parameter is the timestamp, and the third parameter is the key. The key must be the same as that configured on the server.

The generated string concatenation, appended to the access path behind, such as: http://192.168.91.128/group1/M00/00/00/wKhbgF5aMteAWy0gAAJkI7-2yGk361.png? Token = 7 e329cc50307000283a3ad3592bb6d32 & ts = 1582975854. If there is no token in the access path, the access fails.

All right, you’re done! I’ll tell you next time how I play with this thing in Spring Boot.

How to build a distributed file management system with FastDFS

1. What is FastDFS

1.1 FastDFS profile

1.2 Why FastDFS

1.3 FastDFS architecture

2. FastDFS installation

2.1 the Tracker installation

2.2 Storage installation

2.3 Nginx installation

3.Java client invocation

3.1 Uploading Files

3.2 Downloading Files

4. Security issues

Related Posts

Complete HTTP Request Response Process (I)

Basic configuration: GORM use, database connection and configuration and database operations

Algorithm – Graph algorithm – breadth first search