Introduce a FastDFS

Source: China www.oschina.net/question/ta… GitHub open Source: github.com/happyfish10… Reference: blog.chinaunix.net/uid-2019631…

1.1 introduction

FastDFS is an open source high-performance Distributed File System developed by Yu Qing, senior architect of Taobao Development Platform Department. As a distributed file system, manage it to a file, the main features include features include: file storage, file synchronization, file access (such as file upload, download), mainly to solve the problem of mass data storage, especially suitable for small and medium-sized file (recommended range: 4 KB < file_size < 500 MB) as the carrier of the online service.

The FastDFS system has three roles: Tracker Server, Storage Server and Client.

  • Tracker Server: FastDFS coordinator for balancing Responsible for the management of all storage servers and groups. After startup, each storage will connect to Tracker, inform it of its own group and other information, and maintain periodic heartbeat. According to the heartbeat information of the storage, Create a mapping table for group==>[Storage ServerList].
  • Storage Server: a Storage Server that provides capacity and backup services. A group is organized by a volume, group, or volume. A group contains multiple storages and their data is mutually backed up. The storage space of the storage with the smallest capacity in the group is determined.
  • Client: As the initiator of the business request, the Client interacts with the tracker server or storage node through a dedicated interface using TCP/IP protocol, which is the server on which our project is deployed.
1.2 FastDFS architecture diagram

1.3 FastDFS Storage Policy

To support large capacity, storage nodes (servers) are organized into volumes (or groups). A storage system consists of one or more volumes whose files are independent of each other. The file capacity of all volumes is the total file capacity of the entire storage system. A volume can be composed of one or more storage servers. All files on the storage servers under a volume are the same. Multiple storage servers in a volume provide redundant backup and load balancing.

When a server is added to a volume, the system automatically synchronizes existing files. After the synchronization is complete, the system automatically switches the new server to online services. When the storage space is insufficient or about to be used up, you can dynamically add volumes. You only need to add one or more servers and configure them as a new volume, thus increasing the capacity of the storage system.

1.4 FastDFS upload process

FastDFS provides users with basic file access interfaces, such as Upload, Download, Append, and Delete, in the form of client libraries.

The Storage Server periodically sends its Storage information to the Tracker Server. If there is more than one Tracker Server in a Tracker Server Cluster, the relationship between the Tracker servers is equal. Therefore, the client can select any Tracker when uploading.

When the Tracker receives a request from the client to upload a file, it will assign a group for the file. After the group is selected, the Tracker must decide which storage server in the group to assign to the client. After a storage server is allocated, the client sends a file write request to the storage. The storage allocates a data storage directory for the file. Then assign a fileID to the file, and finally generate a file name to store the file based on the above information. As shown below:

1.5 FastDFS file synchronization

When a file is written to a storage server in a group, the client considers that the file is successfully written. After the storage Server writes the file, the background thread synchronizes the file to other storage servers in the same group.

After each storage writes a file, it also writes a binlog. The binlog does not contain file data, but only file name and other meta information. This binlog is used for background synchronization. Progress is recorded as a timestamp, so it is best to keep the clocks of all servers in the cluster in sync.

The synchronization progress of a storage is reported to tracker as part of metadata. Tracke uses the synchronization progress as a reference when selecting a storage to read.

For example, if there are three storage servers A, B, and C in A group, A synchronizes to T1 from C (all files written before T1 have been synchronized to B), and B synchronizes to T2 from C (T2 > T1), when the tracker receives the synchronization progress information, it will arrange it. Use the smallest one as the synchronization timestamp of C. In this case, T1 is C’s synchronization timestamp T1 (all data written before T1 has been synchronized to C). Similarly, according to the above rules, the tracker generates A synchronization timestamp for A and B.

1.6 FastDFS file download

After the client Uploadfile succeeds, it will get a file name generated by the storage. Then the client can access the file according to the file name.



As with the Upload file, the client can select any Tracker server when downloading the file. When tracker sends a download request to a tracker, the file name must be displayed. Tracke retrieves information about the file name, such as the file group, size, and creation time, and then selects a storage for the request to serve the read request

Two FastDFS environment installation

2.1 to prepare

Operating environment CentOS 6.4

Put all downloaded files into package for unified management

2.2 Download libfastCommon

1. Download libfastcommon

Wget # https://github.com/happyfish100/libfastcommon/archive/V1.0.38.tar.gzCopy the code

2. Decompress the file

# tar -zxvf v1.0.38.tar. gz # CD libfastcommon-1.0.38Copy the code

3. Compile and install

# ./make.sh
# ./make.sh install
Copy the code

Libfastcommon. so is installed in /usr/lib64 (32-bit), /usr/lib/ (64-bit), /usr/lib64 (64-bit)



5. Create a soft link, because the FastDFS main program set the lib directory is /usr/local/lib, so we need to create a soft link

# ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
# ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
# ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
Copy the code
2.3 Download and Install FastDFS

1 download FastDFS

Wget # https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gzCopy the code

2 decompression

# tar -zxvf V5.11.tar.gz
# cd fastdfs-5.11
Copy the code

3 Compilation and Installation

# ./make.sh
# ./make.sh install
Copy the code

4 Default installation directory

  1. Service script:cd /etc/init.ddirectory
/etc/init.d/fdfs_storaged
/etc/init.d/fdfs_tracker
Copy the code
  1. Configuration file:cd /etc/fdfsThere are four. Sample files (automatically generated FDFS template configuration files)
/etc/fdfs/client.conf.sample
/etc/fdfs/storage.conf.sample
/etc/fdfs/storage_ids.conf.sample
/etc/fdfs/tracker.conf.sample
Copy the code
  1. Command script:
fdfs_appender_test
fdfs_appender_test1
fdfs_append_file
fdfs_crc32
fdfs_delete_file
fdfs_download_file
fdfs_file_info
fdfs_monitor
fdfs_storaged
fdfs_test
fdfs_test1
fdfs_trackerd
fdfs_upload_appender
fdfs_upload_file
stop.sh
restart.sh
Copy the code

You can also view the executable command ls -la /usr/bin/fdfs*

-rwxr-xr-x. 1 root root 317480 October 9 09:48 /usr/bin/fdfs_appender_test -rwxr-xr-x. 1 root root 317256 October 9 09:48 /usr/bin/fdfs_appender_test1 -rwxr-xr-x. 1 root root 304104 October 9 09:48 /usr/bin/fdfs_append_file -rwxr-xr-x. 1 root Root 303808 October 9 09:48 /usr/bin/fdfs_crc32-rwxr-xr-x. 1 root root 304144 October 9 09:48 /usr/bin/fdfs_delete_file 1 root root 304904 October 9 09:48 /usr/bin/fdfs_download_file -rwxr-xr-x. 1 root root 304496 October 9 09:48 /usr/bin/fdfs_file_info -rwxr-xr-x. 1 root root 322400 October 9 09:48 /usr/bin/fdfs_monitor-rwxr-xr-x. 1 root root 1111520 10月 9 09:48 /usr/bin/fdfs_storaged -rwxr-xr-x. 1 root root 327384 月 9 09:48 /usr/bin/fdfs_test -rwxr-xr-x. 1 root Root 326592 Oct 9 09:48 /usr/bin/fdfs_test1-rwxr-xr-x. 1 root root 453848 Oct 9 09:48 /usr/bin/fdfs_trackerd 1 root root 305096 October 9 09:48 /usr/bin/fdfs_upload_appender -rwxr-xr-x. 1 root root 306120 October 9 09:48 /usr/bin/fdfs_upload_fileCopy the code

5 Setting the soft link of the Script The bin directory of the FastDFS service script is /usr/local/bin, but the actual command is installed in /usr/bin/

You can set up a soft link from /usr/bin to /usr/local/bin

# ln -s /usr/bin/fdfs_trackerd   /usr/local/bin
# ln -s /usr/bin/fdfs_storaged   /usr/local/bin
# ln -s /usr/bin/stop.sh         /usr/local/bin
# ln -s /usr/bin/restart.sh      /usr/local/bin
Copy the code
2.4 Configuring the Tracker Service

1. Go to the /etc/fdfs directory and run the cp command to copy tracker.conf.sample and rename it to tracker.con(delete the suffix of

# cd /etc/fdfs/
# cp tracker.conf.sample  tracker.conf
# vi tracker.conf
Copy the code

Conf: vi tracker.conf: vi tracker.conf: vi tracker.conf: vi tracker.conf: vi tracker.conf: vi tracker.conf

Default: 22122 port=22122 # Tracker Root path to store data and log. The root directory must exist. The subdirectory will automatically create base_path=/home/lyy/fastdfs # HTTP port. HTTP. Server_port =80Copy the code

3. Create the base data directory of tracker, that is, the directory corresponding to base_path

mkdir -p /home/lyy/fastdfs
Copy the code

4. Open port 22122

-a INPUT -m state --state NEW -m TCP -p TCP --dport 22122 -j ACCEPT  # service iptables restartCopy the code

5, start the tracker (support start | stop | restart) :

You can use this method to start # /etc/init.d/fdfs_trackerd start or if the soft link # service fdfs_trackerd start is createdCopy the code

After successful startup, data and logs directories will be created under /home/lyy/fastdfs(base_path configured).



6. View startup logs

Go to the logs directory in the base_path(/home/lyy/fastdfs) you just specified and view the tracker.log file



7. Check the port status

netstat -apn|grep fdfs
Copy the code

If port 22122 is being monitored, the Tracker service is installed successfully, as shown in the following figure:



Tracker close command:service fdfs_trackerd stop

8. Set the Tracker to boot

/ /etc/init.d/fdfs_trackerd start / /etc/init.d/fdfs_trackerd startCopy the code
2.5 Configuring Storage Services

1. Go to the /etc/fdfs directory and run the cp command to copy storage.conf.sample and rename it to storage.conf (delete

# cd /etc/fdfs/
# cp storage.conf.sample storage.conf
# vi storage.conf
Copy the code

2. Edit storage.conf and modify related parameters (base_path, store_path0, tracker_server, and HTTP.server_port).

Group_name =group1 # Port number of the storage server group_name=group1 # Port number of the storage server group_name=group1 Storge default 23000 port=23000 # Heartbeat interval in seconds heart_beat_interval=30 # Number of storage paths Needs and matching store_path_count store_path number = 1 # storage to store the data and log with path, must be created in advance base_path = / home/lyy/storage/fastdfs # if is empty, Base_path store_path0=/home/lyy/storage/file # 256), then when the storage server runs for the first time, Subdir_count_per_path =256 Write a line tracker_server=192.168.50.198:22122 for each tracker server # Time period to allow system synchronization (default is full day). It is used to avoid problems with peak synchronization. Server_port =80 sync_start_time=00:00 synC_end_time =23:59Copy the code

3. Create a Storage basic data directory corresponding to the base_path directory

# mkdir -p/home/lyy/storage/fastdfs # # is this configuration store_path0 path mkdir -p/home/lyy/storage/fileCopy the code

4. Start Storage

# service fdfs_storaged start # service fdfs_storaged start # service fdfs_storaged startCopy the code

5, check the Storage start log view Storage start logging: in a just base_path (/ home/lyy/Storage/fastdfs) in logs directory, check the Storage. The log file

6. Check port status:netstat -apn|grep fdfs



To close the Storage command:service fdfs_storaged stop

Restart Storage command:service fdfs_storaged restart

7. Run the monitor command to check whether the storage is bound successfully and whether the storage and Tracker communicate successfully

/usr/bin/fdfs_monitor /etc/fdfs/storage.conf
Copy the code



8, set the Storage startup

/etc/init.d/fdfs_storaged start: /etc/init.d/fdfs_storaged startCopy the code

9, Storage directory

After Tracker and Storage are successfully started, data and logs directories are created under base_path(/home/lyy/ Storage /file) to record Storage Server information.

In the store_path0 directory, N*N subdirectories are created:

Three install Nginx

1. Download Nginx installation package

Wget # http://nginx.org/download/nginx-1.15.2.tar.gzCopy the code

Unzip nginx:

# tar - ZXVF nginx - 1.12.1. Tar. GzCopy the code

3. Go to the nginx directory

# CD nginx - 1.15.2Copy the code

4. Install the dependency libraries

# yum update
# yum install libpcre3 libpcre3-dev openssl libssl-dev libperl-dev
Copy the code

5. Configure nginx and load fastdfs-nginx-module

# ./configure
Copy the code

6. Compile and install nginx

# make
# make install
Copy the code

Nginx installation path:whereis nginx

8. Start and stop:

#./nginx -s stop #./nginx -s quit #./nginx -s reloadCopy the code

9. Set boot

# vi/etc/rc. Local add a line: / usr/local/nginx/sbin/nginx # # set the execute permissions chmod 755 rc. LocalCopy the code

10. Query the nginx version

/usr/local/nginx/sbin/nginx -V
Copy the code



Open port 80 on the firewall

-c INPUT -m state --state NEW -m TCP -p TCP --dport 80 -j ACCEPT  # service iptables restartCopy the code

12. Access nginx through a browser

To check whether nginx started successfully, enter our IP address (mine: 192.168.50.198). Default is port 80.

File upload test

4.1 Modifying the nginx.conf file
Vi/usr/local/nginx/conf/nginx. Conf add the following lines, To map/group1 M00 to/home/lyy/storage/file/data location/group1 M00 {alias/home/lyy/storage/file/data; } # # restart nginx/usr/local/nginx/sbin/nginx -s reloadCopy the code

4.2 Client configuration file in Tracker server

1. Modify clinet configuration

# cd /etc/fdfs
# cp client.conf.sample client.conf
# vi client.conf
Copy the code

Modify the following configuration:

Base_path =/home/lyy/ Client # tracker_server=192.168.50.198:22122Copy the code

Creating a folder

mkdir -p /home/lyy/client
Copy the code
4.3 Upload Test

In Linux, run the following command to upload the 1.jpg image

/usr/bin/fdfs_upload_file /etc/fdfs/client.conf 1.jpg 
Copy the code

Return the file ID number on success:

group1/M00/00/00/wKgyxl2dfNWAG9cOAAOUD-69J7A524.jpg

Group name: group1 Disk: M00 Directory: 00/00 File name: wkgyxL2DFNWAG9COAAOUd-69j7a524.jpgCopy the code
4.4 The Access succeeds Through WGET or Browser
  • The browser

  • Wget way

Install and configure the Fastdfs-nginx-module

5.1 Fastdfs-nginx-module Module Description

FastDFS uses the Tracker server to store files on the Storage server. However, files need to be replicated between Storage servers in the same group, resulting in synchronization delay.

Suppose the Tracker server uploads the file to 192.168.50.196, and the file ID is returned to the client.

In this case, the FastDFS storage cluster mechanism synchronizes the file to the same storage group 192.168.50.197. If a client uses this file ID to fetch files from 192.168.50.197 before the file replication is complete, the file cannot be accessed.

Fastdfs-nginx-module redirects the file link to the source server to retrieve the file, avoiding the file failure caused by replication delay on the client.

5.2 Installing the Fastdfs-nginx-Module installation package

1. Download the Fastdfs-nginx-Module installation package

# # download the installation package wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz decompression installation package tar - ZXVF V1.20. Tar. GzCopy the code

2. Configure Nginx

# # stopped first nginx service/usr/local/nginx/sbin/nginx # CD nginx - - s stop into nginx directory 1.15.2 # #. To add modules/configure - add - the module = / usr/local/package/fastdfs - nginx - module - 1.20 / SRC to recompile, installation # make && make installCopy the code
An error may be reported: compilation is interrupted. Make [1]: /usr/local/package/nginx-1.15.2 make: /usr/local/package/nginx-1.15.2 *** [build] error 2 Fastdfs-nginx-module-1.20 / SRC /config /configure ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/" CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"Copy the code

3. Check the nginx version

/usr/local/nginx/sbin/nginx -V
Copy the code

If the following figure is displayed, the configuration is successful

5.3 Configuring the Nginx and Fastdfs-nginx-module Modules

1. Configure mod-fastdfs.conf and copy it to /etc/fdfs

# CD/usr/local/package/fastdfs - nginx - module - 1.20 / SRC / # cp mod_fastdfs. Conf/etc/FDFS /Copy the code

2, modify mod-fastdfs.conf

#  vi /etc/fdfs/mod_fastdfs.conf 
Copy the code

Modify the following configuration:

Connect_timeout =10 # tracker_server= 192.168.50.1980:22122 # Url_have_group_name = true # Storage the store_path0 path configured in storage.conf must be the same as store_path0=/home/lyy/ Storage /fileCopy the code

3. Copy the FastDFS decompressed directory (anti-steal. JPG, http.conf, mime.types) to /etc/fdfs

# CD/usr/local/package/fastdfs - 5.11 / conf / # cp anti - steal. JPG HTTP. Conf mime. The types/etc/FDFS /Copy the code

4. Configure nginx and modify nginx.conf

 vi /usr/local/nginx/conf/nginx.conf
Copy the code

Add the fastdfs-nginx module to port 80 and comment out the previous configuration

5. Start nginx

/usr/local/nginx/sbin/nginx
Copy the code

6. The access succeeds through wGET and browser

  • The browser

  • Wget way



    Structure diagram (found online, not original) :

Six FastDFS common test commands

6.1 – Uploading files
/usr/bin/fdfs_upload_file <config_file> <local_filename> / usr/bin/fdfs_upload_file/etc/FDFS/client. The conf 1. JPG after a successful return file ID: group1 / M00/00/00 / wKgyxl2dfNWAG9cOAAOUD - 69 j7a524. JPGCopy the code
6.2 – Download files
Command: /usr/bin/fdfs_download_file <config_file> <file_id> [local_filename] Example:  /usr/bin/fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgyxl2dfNWAG9cOAAOUD-69J7A524.jpgCopy the code

View the results:

6.3 – Deleting files
Command: /usr/bin/fdfs_delete_file <config_file> <file_id> Example:  /usr/bin/fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgyxl2dfNWAG9cOAAOUD-69J7A524.jpgCopy the code

Return result: