•FastDFS is a lightweight open source distributed file system
•FastDFS mainly solves the problem of large file storage and high concurrent access, and implements load balancing for file access
•FastDFS implements software-based RAID and can use cheap IDE hard drives for storage • Supports online storage server expansion
• Only one file supports the same content, saving disk space. •FastDFS can be accessed only through ClientAPI. POSIX access is not supported
•FastDFS is especially suitable for medium to large sites to store resource files (images, documents, audio, video, etc.)
FastDFS profile
•FastDFS is a lightweight open source distributed file system
•FastDFS mainly solves the problem of large file storage and high concurrent access, and implements load balancing for file access
•FastDFS implements software-based RAID, which can be stored using inexpensive IDE hard drives
• Supports online capacity expansion of storage servers
• Save only one file that supports the same content, saving disk space
•FastDFS can only be accessed through ClientAPI. POSIX access is not supported
•FastDFS is especially suitable for medium to large sites to store resource files (images, documents, audio, video, etc.)
System Architecture – Architecture diagram
System Architecture – Flowchart for uploading files
• 1. The client queries the storage to which the tracker is uploaded.
• 2. Tracker returns an available storage;
• 3. The client communicates with the storage to upload the file.
System Architecture – Flowchart for downloading files
• 1. The client queries the storage where the tracker downloaded the file, using the file id (group name and file name).
• 2. Tracker returns an available storage;
• 3. The client communicates with the storage to download the file.
Related terms
• Tracker Server: a tracking Server that performs scheduling and load balancing on access. Records the status of the Storage Server, which connects the Client and Storageserver.
• Storage Server: A Storage Server on which files and meta data are stored
• Group: a group, also called a volume. The files on the servers in the same group are identical
• File id: consists of two parts: group name and file name (including path)
• Meta Data: file related attributes in Key Value Pair mode, for example, width=1024,heigth=768
Synchronization mechanism
• The storageservers in the same group are peers. Files can be uploaded or deleted on any storageserver.
• File synchronization is implemented only between storage servers in the same group in push mode, that is, the source server synchronizes files to the target server.
• Source data needs to be synchronized. Backup data does not need to be synchronized again. Otherwise, a loop is generated.
• The second rule has an exception, that is, when a new storageserver is added, the existing storageserver synchronizes all existing data (including source data and backup data) to the new server.
Communication protocol
• A protocol package consists of header and body
• Header contains 10 bytes and is in the following format:
– 8 bytes Body length
– 1 byte command
– 1 byte status
• The body packet format depends on the command. The body can be empty
Runtime directory structure – Tracker Server
• ${base_path}
|__data
| | __storage_groups. Dat: storage group information
| | __storage_servers. Dat: storage server list
|__logs
| __trackerd. Log: trackerserver log file
Runtime directory structure – Storage Server
• ${base_path}
|__data
| | __. Data_init_flag: current storageserver initialization information
| | __storage_stat. Dat: current storage server statistics
| | __sync: storing data synchronization related documents
| | | __binlog. Index: the current binlog file index number
| | | __binlog. # # # : store update records (log)
| | | __ ${ip_addr} _ ${port}. Mark: synchronous completion
| |
| | __ one level: 256 directory where the data files, such as: 00, 1 f
| | __ the secondary directory: 256 directory for storing the data file
|__logs
| __storaged. Log: storageserver log file
Installation and operation
•#step 1. download FastDFS source package and unpack it,
•# if you use HTTP to download file, please downloadlibevent 1.4.x and install it
• tar XZF FastDFS_v1. X.t ar. Gz
#, for example:
• tar XZF FastDFS_v1. 20. Tar. Gz
•
•#step 2. enter the FastDFS dir
• CD FastDFS
•
•#step 3. if HTTP supported, modify make.sh, uncommentthe line:
• # WITH_HTTPD = 1, then the execute:
•. / make. Sh
•
• # step 4. Make install
•. / make. Sh install
•
•#step 5. Edit /modify the config file of tracker andstorage
•
•#step 6 run server programs
• # start the tracker server:
• / usr/local/bin/fdfs_trackerd
•
• # start the storage server:
• / usr/local/bin/fdfs_storaged
FastDFS versus centralized storage
• FastDFS 英 文 : www.csource.org/
• FastDFS English: code.google.com/p/fastdfs/
Official website information, share for everyone to learn
Willing to understand the framework technology or source code friends directly please: 2042849237
More detailed source code reference sources