directory

           

⊙ What is a distributed file system

⊙ Why use a distributed file system

⊙FastDFS compares with HDFS

⊙ What is FastDFS

⊙ Common Terms

Even though FastDFS architecture

⊙FastDFS Upload process

⊙FastDFS download process

⊙ FastDFS installation on Linux

  

1. What is distributed file system

With the increasing number of file data, through the tomcat or nginx static resource file virtualization in a server on a single node is not saving, if use to store can also be multiple nodes, but bad for management and maintenance, so we need a system to manage the data files on multiple computers node, which is a distributed file system.

Distributed file system is a file system that allows files to be shared among multiple nodes over the network. Multiple computer nodes form a whole to provide file sharing and storage space for more users. For example, a web disk is a distributed file storage system. Although we are a distributed file system, we are transparent to the user, as if they were accessing a local disk.

Distributed file systems can provide redundancy, so they have high fault tolerance. Some nodes in the system go down, but the overall file service does not stop, can still provide services to users, the overall operation, data will not be lost. The distributed file system has strong scalability. It is easy to add or remove nodes without affecting online services. After nodes are added, they are released online and added to the cluster to provide services for users.

Distributed file systems can provide load balancing capabilities, which can be served by multiple nodes when reading file copies, and can be scaled horizontally to ensure improved performance and load.

 

2. Why use distributed file systems

  • Massive file data storage

  • High availability of file data (redundant backup)\

  • Read/write performance and load balancing

The above three points were not possible with Tomcat or Nginx before, which is why we use distributed file system

 

3. Compare FastDFS with HDFS

When it comes to distributed file storage, some people will definitely think of HDFS. Their main positioning and application scenarios are different. \

  • HDFS, a file system in Hadoop, mainly solves the problem of distributed data storage in parallel computing. Its single data files are usually large and are stored in blocks (sharded), so it is the scenario used for big data and big file storage.
  • FastDFS is mainly used on Internet sites to provide an online service for file upload and download. Therefore, FastDFS supports load balancing, dynamic expansion and other aspects well. FastDFS does not store files quickly. FastDFS is great for storing small and medium-sized files, such as user avatars, small audio and video files, etc. \

 

4. What is FastDFS

FastDFS is an open source lightweight distributed file system. It manages files, including file storage, file synchronization, and file access (file upload and download). It solves the problems of large-capacity storage and load balancing. \

It is especially suitable for online services with file as the carrier, such as photo album website, video website and so on. FastDFS is tailor-made for the Internet. It takes into account redundant backup, load balancing, and linear expansion, and emphasizes high availability and performance. It is easy to set up a high-performance file server cluster to provide file uploading and downloading services.

The FastDFS server has two roles: tracker and storage. The tracker mainly does scheduling work and plays a load balancing role in access.

Storage nodes store files and perform all functions of file management. FastDFS also manages file metadata. The meta data of a file is the related attributes of the file, which are represented by key value pairs. For example, width=1024, where key is width and value is 1024. File metadata is a list of file attributes that can contain multiple key-value pairs.

Both the tracker and the storage node can consist of one or more servers. Servers in both the tracker and the storage node can be added or taken offline at any time without affecting online services. All servers in the tracker are peer and can be added or reduced at any time according to the stress of the server.

To support large capacity, storage nodes (servers) are organized into volumes (or groups). A storage system consists of one or more volumes whose files are independent of each other. The file capacity of all volumes is the total file capacity of the entire storage system. A volume can be composed of one or more storage servers. All files on the storage servers under a volume are the same. Multiple storage servers in a volume provide redundant backup and load balancing.

When a server is added to a volume, the system automatically synchronizes existing files. After the synchronization is complete, the system automatically switches the new server to online services.

When the storage space is insufficient or about to be used up, you can dynamically add volumes. You only need to add one or more servers and configure them as a new volume, thus increasing the capacity of the storage system.

File identification in FastDFS is divided into two parts: the volume name and the file name.

 

5. Common terms

  • Tracker: a server used to coordinate scheduling, load balancer and record storage status information. \

  • Storage: storage server for storing files and their metadata. \

  • Group: the nodes in the same group provide redundancy backup. Different groups are used for capacity expansion. \

  • Mata Data: Metadata information of a file, such as length and width, image suffixes, and number of frames of a video.

 

6. FastDFS architecture

 

7. FastDFS upload process

  • Storage periodically sends a heartbeat to the Tracker to tell the Tracker that it is still alive, so Fastdfs can work

  • The client sends an upload request to the Tracker, which checks for available Storage\

  • If one is available, the client can upload file data to Storage \

  • After the file is written to the disk, the Storage returns the path information to the client \

  • The client can find the uploaded file based on this path information

 

8. FastDFS download process

  • Storage periodically sends a heartbeat to the Tracker installation to tell the Tracker that it is still alive so that Fastdfs can work

  • The client sends the download request to the Tracker, which returns to the client \ after the Tracker finds the Storage address

  • After obtaining the Storage address, the client goes to the Storage to find the file

  • Return the file to the client

 

FastDFS installation on Linux

Note: Be sure to start Tracker first before starting Storage

 

9.1 Installing the Basic Environment

Note: If Tracker and Storage are configured on different servers, the base environment should be installed on both servers.

 

1. Download the installation package

  • Libfatscommon: some common function packages that FastDFS has isolated \

  • FastDFS: FastDFS ontology \

  • Fastdfs-nginx-module: fastdfs and nginx associated module \

  • Nginx: Publish access services

 

2. Install the basic environment

  • yum install -y gcc gcc-c++ \

  • yum -y install libevent\

  • yum -y install zlib zlib-devel pcre pcre-devel gcc gcc-c++ openssl openssl-devel libevent libevent-devel perl unzip net-tools wget\

 

3. Install the libfatscommon library

  • The tar – ZXVF libfastcommon – 1.0.42. Tar. Gz

 

4. Go to the libfastcommon folder, compile and install

  • ./make.sh 
  • ./make.sh install

 

5. Install the fastdfs main program file

  • Tar – ZXVF fastdfs 6.04. Tar. Gz \

 

6. Install fastdfs

  •  ./make.sh\

  • ./make.sh install

 

7. Copy the configuration file

  • Cp/home/software/FastDFS/FastDFS – 6.04 / conf / * / etc/FDFS / \

Stop the tracker \

  • /usr/bin/stop.sh /etc/fdfs/tracker.con

 

9.2 Configuring the Tracker Service

 

Note: Tracker and storage are two different concepts of the same FastdFS main program. You can configure different configuration files to set tracker or storage

 

1. Go to the configuration file directory

  •  cd /etc/fdfs\

 

2. Modify the Tracker configuration file

  •  vim tracker.conf
Modify the tracker configuration file, which is the working directory of the tracker and saves data and logs
base_path=/usr/local/fastdfs/tracker
Copy the code

 

3. Create a working directory

  •  mkdir /usr/local/fastdfs/tracker -p

 

4. Start the process

  •  /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf\

 

9.3 Configuring the Storage Service

1. Go to the configuration file directory

  • cd /etc/fdfs

 

2. Modify the storage configuration file

  •  vim storage.conf\
Change the group name
group_name=test 
Change the workspace of the storage
base_path=/usr/local/fastdfs/storage 
Change the storage space of the storage
store_path0=/usr/local/fastdfs/storage 
Change the address and port number of tracker to be used for heartbeatTracker_server = 192.168.1.153:22122# follow up with an external service port number for Nginx
http.server_port=8888
Copy the code

 

3. Create a working directory

  •  mkdir /usr/local/fastdfs/storage -p\

 

4. Start the process

  • Prerequisite: Tracker \ must be started first

  • /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

 

9.4 Uploading tests using the Client

  

1. Go to the configuration file directory

  •  cd /etc/fdfs

 

2. Modify the client configuration file

  •  vim client.conf\
Modify the client workspace
base_path=/usr/local/fastdfs/client 
Change the address and port number of tracker to be used for heartbeat
tracker_server=192.168.1.153:22122
Copy the code

 

3. Create a working directory

  •  mkdir /usr/local/fastdfs/client

4. Take a picture and test it out

  • cd /usr/bin/\

  • ./fdfs_test /etc/fdfs/client.conf upload /home/logo.png

 

9.5 Configuring Nginx Fastdfs to Implement the File Server

Note: After fastdfs is installed, it is not accessible through HTTP, so you need to use Nginx, so you need to install the third party fastdfs module into nginx, and then use it. \

Note: Nginx must be on the same node as storage.

 

1. Decompress the Nginx Fastdfs package

  • The tar – ZXVF fastdfs – nginx – module – 1.22. Tar. Gz \

 

2. Copy the configuration file:

  • CD/fastdfs – nginx – module – 1.22 / SRC \

  • cp mod_fastdfs.conf /etc/fdfs

 

3. Modify the /fastdfs-nginx-module/ SRC /config file

/usr: \ /usr: \ /usr: \

 

4. Install nginx

1, install dependent environment \

(1) Install GCC

yum install gcc-c++

(2) Install the PCRE library to parse regular expressions

yum install -y pcre pcre-devel

(3) Zlib compression and decompression dependencies,

yum install -y zlib zlib-devel

(4) SSL-secured, encrypted socket protocol layer for HTTP secure transport, also known as HTTPS

yum install -y openssl openssl-devel

2, decompression, need to pay attention to, decompression is the source code, the source code needs to be compiled to install

The tar – ZXVF nginx – 1.16.1. Tar. Gz

3. Create a temporary nginx directory before compiling. If you do not create a temporary nginx directory, an error will be reported during nginx startup

mkdir /var/temp/nginx -p

2. Change the configuration

./configure

— prefix = / usr/local/nginx

– pid – path = / var/run/nginx/nginx. Pid

– the lock – path = / var/lock/nginx. Lock

– error – log – path = / var/log/nginx/error. The log

– HTTP – log – path = / var/log/nginx/access. Log

– with – http_gzip_static_module

– HTTP client – body – temp – path = / var/temp/nginx/client

– HTTP proxy – temp – path = / var/temp/nginx/proxy

– HTTP – fastcgi – temp – path = / var/temp/nginx/fastcgi

– HTTP – uwsgi – temp – path = / var/temp/nginx/uwsgi

– HTTP – scgi – temp – path = / var/temp/nginx/scgi

– add – the module = / home/software/FDFS/fastdfs – nginx – module – 1.22 / SRC

 

5. Make compilation

  • make

6. Installation

  • make install

7. Go to the sbin directory to start

  • nginx./nginx

8. Modify the mod_fastdfs.conf configuration file

  • cd /etc/fdfs
Modify mod_fastdfs workspace
base_path=/usr/local/fastdfs/tmp 
 Change the address and port number of tracker to be used for heartbeat
tracker_server=192.168.1.153:22122 
Change the group name
group_name=imooc 
# concatenate the group name at the URL
url_have_group_name = true 
File storage space
store0_path=/usr/local/fastdfs/storage
Copy the code

9. Create a workspace

  • mkdir /usr/local/fastdfs/tmp

10. Modify nginx.conf and add the following virtual hosts

server {
   listen 8888;
   server_name localhost;
If you have changed the group name, this field should match the group name you changed
   location /group1/M00
   {
      ngx_fastdfs_module;
   }
}
Copy the code

 

11. Start nginx

 

12. Browser access