FastDFS Distributed file clustering for Java Advanced Architecture

FastDFS profile

FastDFS is an open-source lightweight distributed file system that uses C and supports Unix-like operating systems such as Linux and BSD. It’s worth noting that Fastdfs is not a generic file system and can only be accessed through a dedicated API. Tailor-made for Internet applications, FastDFS solves the problem of large file storage. Fastdfs pursues high performance and high scalability. The main concepts of FastdFS:

Tracker-server: indicates the tracking server. Used for tracking files, mainly for scheduling. It records the status information of all storage groups and storage servers in memory and serves as the main hub for clients and data stores. More streamlined than GFS because no file index is recorded.
Storage-server: indicates a storage server. Used to store files. Use the operating system’s file system directly to manage and organize files.
Group: group, volume. Multiple servers exist in a group, the servers in a group store exactly the same files, and the server status of the same group is peer. Operations on files can be performed on servers in either group.
Metadata: Indicates metadata. Store information about files in key-value pairs.

Comparison of storage systems

In other words, there is no harm without comparison. Fastdfs is not a panacea. You need to choose a suitable storage system based on your business.

The storage system	Type of file suitable for storage	File distribution	The system performance	The complexity of the	FUSE(user file system)	POSIX()	Backup mechanism	Communication protocol interface	The community is	Implementation language
FastDFS	4 KB to 500 MB	Merge small files for storage	high	simple	Does not support	Does not support	Intra-group redundancy backup	HTTP API	Domestic users	C
TFS	All the files	Small files are merged into chunks to organize fragmentation	–	complex	Does not support	Does not support	Block storage in active/standby Dr Mode	HTTP API	less	C++
MFS	More than 64 k	Shard storage	The memory usage of the Master node is high	–	support	support	Multi-point backup, dynamic redundancy	Mount using FUSE	more	Perl
HDFS	A large file	Large file fragmentation block storage	–	simple	support	support	More than a copy	The native API	more	Java
Ceph	Object large file	OSD has one primary node and many secondary nodes	–	complex	support	support	More than a copy	The native API	less	C++
MogileFS	Lots of small pictures	–	high	complex	support	Does not support	Dynamic redundancy	The native API	The document is less	Perl
ClusterFS	A large file	–	–	simple	support	support	–	–	more	C

FastDFS client – server interaction principle

FastDFS + Nginx integration

Architecture diagram

Install FastDFS

mkdir /source
cd /sourceyum install -y gcc gcc-c++ make cmake wget libevent wget https://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz wget https://github.com/happyfish100/fastdfs/archive/V5.10.tar.gz tar - ZXVF V1.0.35. Tar. Gz tar - ZXVF V5.10. Tar. GzcdLibfastcommon-1.0.35./make. Sh./make-s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
cd ../
cdFastdfs - 5.10 /. / make. Sh. / make. Sh installcd. / rm - rf libfastcommon - 1.0.35 rm - rf fastdfs - 5.10 cp/etc/FDFS/tracker. Conf. Sample/etc/FDFS/tracker. Conf cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf mkdir -p /data/fdfs/tracker mkdir -p /data/fdfs/storage ln-s /usr/bin/stop.sh /usr/local/bin/stop.sh
ln -s /usr/bin/restart.sh /usr/local/bin/restart.shCopy the code

Modifying a Configuration File

Modify the tracker configuration file:

base_path=/data/fdfs/trackerCopy the code

Modify the storage configuration file:

Base_path = / data/storage/FDFS store_path0 = / data/storage/FDFS tracker_server = 192.168.80.3:22122Copy the code

Modifying the client configuration file:

Base_path = / data/FDFS/client tracker_server = 192.168.80.3:22122Copy the code

Start the

/etc/init.d/fdfs_trackerd start
/etc/init.d/fdfs_storaged startCopy the code

Netstat tunlap | grep: TCP 22122 0 0 0.0.0.0:0.0.0.0:22122 * 7247 / fdfs_trackerd TCP 0 0 192.168.80.3: LISTEN. "22122 192.168.80.3:39318 ESTABLISHED 7247/fdfs_trackerd TCP 0 0 192.168.80.3:39318 192.168.80.3:22122 ESTABLISHED 7444/fdfs_storagedCopy the code

Check the service status after startup. If active (exited) is displayed, restart the service.

test

/usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/ FastDFS_v5.05. Tar. Gz group1 / M00/00/00 / wKhQA1ysjSGAPjXbAAVFOL7FJU4 tar. GzCopy the code

Files are stored at:

ll /data/fdfs/storage/data/00/00/wKhQA1ysjSGAPjXbAAVFOL7FJU4.tar.gzCopy the code

Install Nginx and configure the module

Install the PCRE (Perl-compatible regular expression) library required by Nginx to allow Nginx to provide URL rewriting functionality using the rewrite module.
yum install pcre pcre-devel perl-ExtUtils-Embed -y
# Install openSSL-devel to allow Nginx to provide HTTPS services.
yum install openssl-devel -y
# Download software package
cd /sourceWget wget HTTP: / / http://59.80.44.46/nginx.org/download/nginx-1.14.2.tar.gz http://nchc.dl.sourceforge.net/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.g z# Decompress the packageTar XVF nginx-1.14.2.tar.gz tar XVF fastdfs-nginx-module_v1.16.tar.gzCreate the necessary soft connections
ln -s /usr/include/fastdfs/ /usr/local/include/fastdfs
ln -s /usr/include/fastcommon/ /usr/local/include/fastcommon
cp FastDFS/conf/http.conf /etc/fdfs/
cp FastDFS/conf/mime.types /etc/fdfs/
cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
mkdir -p /data/fdfs/fastdfs-nginx-moduleCopy the code

Modify the configuration file /etc/fdf/mod_fastdfs.conf:

connect_timeout=10
base_path=/data/fdfs/fastdfs-nginx-module
Configure the server addressTracker_server = 192.168.80.3: url_have_group_name = 22122true
store_path0=/data/fdfs/storageCopy the code

mkdir /applications
mkdir /tmp/nginx
useradd nginx -s /sbin/nologin -M
cdNginx-1.14.2. /configure \ --user=nginx \ --group=nginx \ --prefix=/applications/nginx-1.14.2 \ --with-http_ssl_module \  --with-http_gzip_static_module \ --http-client-body-temp-path=/tmp/nginx/client \ --http-proxy-temp-path=/tmp/nginx/proxy \ --http-fastcgi-temp-path=/tmp/nginx/cgi \ --with-poll_module \ --with-file-aio  \ --with-http_realip_module \ --with-http_addition_module \ --with-http_random_index_module \ --with-pcre \ --with-http_stub_status_module \ --with-stream \ --add-module=/source/fastdfs-nginx-module/src
make
make install
ln -s/applications/nginx chown-r nginx.nginx /applications/nginx chown-r nginx.nginx /applications/nginx-1.14.2/ chown -r nginx.nginx/TMP /nginx/Copy the code

Modify the configuration files/applications/nginx/conf/nginx. Conf, block is added in HTTP:

server {
        listen       8000;
        server_name  media;

        location / {
            root   html;
            index  index.html index.htm;
        }

        Intercept the file request and forward it to the module belowlocation ~/group[0-9] { ngx_fastdfs_module; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; }}Copy the code

Test configuration file correctness and start:

/applications/nginx/sbin/nginx -t
/applications/nginx/sbin/nginxCopy the code

Delete temporary files:

Rm -rf FastDFS rm -rf fastdfs-nginx-module rm -rf libfastcommon-1.0.35 rm -rf nginx-1.14.2Copy the code

Test Nginx file download function, input in a browser: http://192.168.80.3:8000/group1/M00/00/00/wKhQA1ysjSGAPjXbAAVFOL7FJU4.tar.gz

An nginx can access only one storage server’s data. Therefore, multiple storage servers need to configure multiple nginx, and then route the nginx according to the groupid (groupid) in the request path.

A simple script to start and stop the service:

# start-service
# usage: ./start-service
echo "============================================Sync datetime==========================================="
ntpdate time7.aliyun.com
echo "===========================================Getting status==========================================="
/etc/init.d/fdfs_trackerd status
/etc/init.d/fdfs_storaged status
echo "==========================================Starting service=========================================="
/etc/init.d/fdfs_storaged start
/etc/init.d/fdfs_trackerd start
echo "===========================================Getting status==========================================="
/etc/init.d/fdfs_trackerd status
/etc/init.d/fdfs_storaged status
echo "=========================================Testing config file========================================"
/applications/nginx/sbin/nginx -t
echo "=========================================Starting web server========================================"
/applications/nginx/sbin/nginx
echo "=======================================Getting network status======================================="
sleep 5s
netstat -tunlap | grep :22122
netstat -tunlap | grep :8000Copy the code

# stop-service
# usage: ./stop-service
echo "===========================================Getting status==========================================="
/etc/init.d/fdfs_trackerd status
/etc/init.d/fdfs_storaged status
echo "==========================================Stopping service=========================================="
/etc/init.d/fdfs_storaged stop
/etc/init.d/fdfs_trackerd stop
echo "========================================Stopping web server========================================="
kill `cat /applications/nginx/logs/nginx.pid`
echo "===========================================Getting status==========================================="
/etc/init.d/fdfs_trackerd status
/etc/init.d/fdfs_storaged status
echo "=======================================Getting network status======================================="
sleep 5s
netstat -tunlap | grep :22122
netstat -tunlap | grep :8000Copy the code

Token-based anti-theft chain implementation

FastDFS uses Token to prevent theft. Token is time-sensitive, including file ID, timestamp TS and Token. Request resources in FastDFS using urls with TS and tokens. An algorithm for generating tokens is provided in FastDFS, and the extension module validates the tokens. Since Token generation and verification are done on the server side, there are no security issues. Examples of links:

http://192.168.1.15:8080/group1/M01/01/01/wKgBD01c15nvKU1cAABAOeCdFS466570.c?token=b32cd06a53dea4376e43d71cc882f9cb&ts=1297930137Copy the code

/etc/ff/http. conf

# Enable Token verification
http.anti_steal.check_token=true
The # Token declaration cycle is 240 seconds
http.anti_steal.token_ttl=240
An encrypted string can be generated using openssl rand-base64 64
http.anti_steal.secret_key=2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw
# Token verification failed
http.anti_steal.token_check_fail=/data/fdfs/error.svgCopy the code

Using Java client validation:

Install the client to the local repository:

git clone https://github.com/happyfish100/fastdfs-client-java.git
cd fastdfs-client-java
mvn clean installCopy the code

Create a normal project using Maven and add dependencies to the POM file:

<dependencies> <dependency> <groupId>org.csource</groupId> <artifactId>fastdfs-client-java</artifactId> The < version > 1.27 - the SNAPSHOT < / version > < / dependency > < / dependencies >Copy the code

Create FastDFS config file fastdfs-client.properties in resources directory:

fastdfs.connect_timeout_in_seconds = 5
fastdfs.network_timeout_in_seconds = 30
fastdfs.charset = UTF-8
fastdfs.http_anti_steal_token = truefastdfs.http_secret_key = 2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrntxzLIDbPdvDDLJM8QHhKxSGWTcr+9VdG3yptkw Fastdfs. http_tracker_http_port = 8080 Fastdfs. tracker_Servers = 192.168.80.3:22122Copy the code

Create com. Bluemiaomiao. Demo. Java class files:

package com.bluemiaomiao; import org.csource.common.MyException; import org.csource.fastdfs.ClientGlobal; import org.csource.fastdfs.ProtoCommon; import java.io.IOException; import java.security.NoSuchAlgorithmException; import java.util.Properties; public class Demo { public static void main(String[] args) throws IOException, MyException, NoSuchAlgorithmException {// Load config file Properties prop = new Properties(); prop.load(Demo.class.getResourceAsStream("/fastdfs-client.properties")); ClientGlobal.initByProperties(prop); System.out.println(clientglobal.configinfo ()); // The address returned by the file upload tool is normally saved in the database. String remoteFileName ="group1/M00/00/00/wKhQA1ysjSGAPjXbAAVFOL7FJU4.tar.gz";
        // 获取当前时间戳
        int ts = (int)(System.currentTimeMillis()/1000);
        // 获取Token, 传入的文件ID不要含有分组信息
        String token = ProtoCommon.getToken("M00/00/00/wKhQA1ysjSGAPjXbAAVFOL7FJU4.tar.gz", ts, prop.getProperty("fastdfs.http_secret_key")); // Use the browser to access the returned URL system.out.println ("http://192.168.80.3:8000/" + remoteFileName + "? token=" + token + "&ts="+ ts); }}Copy the code

If an anti-theft link image is displayed during the access, there may be a gap between the test client and the server. The difference between the two hosts cannot be minute. You can use the following method to synchronize the server:

Install the client of the time synchronization server. Windows also needs to synchronize with this server
# Control Panel -> Clock and Region -> Set Date and time ->Internet Time -> Change Settings
yum install ntpdate
ntpdate time7.aliyun.comCopy the code

Integrate FastDHT for data de-duplication

FastDHT is a distributed hash system (DHT) that uses BerkeleyDB for data storage and libevent for network IO processing. Depends on the libfastCommon component.

download

Go to the Oracle website to download Berkeley DB database. Go to the Fastdfs GitHub page to download FastDHT source package. Since linEvent and libFastCommon have been installed before, you only need to install the database and FastDHT.

Install and configure the database with FastDHT

The tar XVF db - 6.2.23. Tar. GzcdThe db - 6.2.23 / build_unix /.. / dist/configure -- prefix = / applications/db - 6.2.23cd. /.. / rm -rf db-6.2.23 unzip fastdht-master.zipcd fastdht-master
vim make.shCopy the code

Modify line 27:

CFLAGS='-Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -I /applications/db-6.2.23/include/ -L /applications/db-6.2.23/lib/'Copy the code

-i: specifies the header file directory provided by the database

-l: specifies the library file directory provided by the database

./make.sh
./make.sh install
cd ../
rm -rf fastdht-masterCopy the code

configuration

mkdir /data/fdhtCopy the code

/etc/fdht/fdht_client.conf

base_path=/data/fdht
#include /etc/fdht/fdht_servers.confCopy the code

/etc/fdht/fdht_servers.conf

Group0 = 192.168.80.3:11411Copy the code

/etc/fdht/fdhtd.conf

base_path=/data/fdht
#include /etc/fdht/fdht_servers.confCopy the code

Modify /etc/ff/storage. conf:

check_file_duplicate=1
key_namespace=FastDFS
keep_alive=1
#include /etc/fdht/fdht_servers.confCopy the code

The server configuration file must be included.

ln -s/ applications/db - 6.2.23 / lib/libdb - 6.2 so/usr/lib/libdb - 6.2 so ln-s/ applications/db - 6.2.23 / lib/libdb - 6.2 so/usr/lib64 / libdb - 6.2. SoCopy the code

Start and test

fdhtd /etc/fdht/fdhtd.confCopy the code

Restart to use:

fdhtd /etc/fdht/fdhtd.conf restartCopy the code

View the results:

Netstat tunlap | grep: TCP 11411 0 0 0.0.0.0:0.0.0.0:11411 * 20605 / FDHTD LISTENCopy the code

Since FastDFS was turned off when you installed FastDHT, you need to start FastDFS

./start-serviceCopy the code

Modify the previous start-service script to add the following before starting tracker and storage:

fdhtd /etc/fdht/fdhtd.confCopy the code

Add before viewing tracker and nGIxN network status:

netstat -tunlap | grep :11411Copy the code

Testing:

fdfs_upload_file /etc/fdfs/client.conf /source/ db - 6.2.23. Tar. Gz group1 / M00/00/00 / wKhQA1yu2L6APTk - AqQOLABfhaQ. Tar. Gz fdfs_upload_file/etc/FDFS/client. The conf /source/ db - 6.2.23. Tar. Gz group1 / M00/00/00 / wKhQA1yu2MKAOmIiAqQOLHUWXfw tar. Gz ll/data/FDFS/storage/data / 00/00 / total 45268 -rw-r--r-- 1 root root 44305964 Apr 11 14:03 wKhQA1yu2L6AM0aiAqQOLKFBFuc.tar.gz lrwxrwxrwx 1 root root 64 Apr 11 14:03 wKhQA1yu2L6APTk-AqQOLABfhaQ.tar.gz -> /data/fdfs/storage/data/00/00/wKhQA1yu2L6AM0aiAqQOLKFBFuc.tar.gz lrwxrwxrwx 1 root  root 64 Apr 11 14:03 wKhQA1yu2MKAOmIiAqQOLHUWXfw.tar.gz -> /data/fdfs/storage/data/00/00/wKhQA1yu2L6AM0aiAqQOLKFBFuc.tar.gzCopy the code

Custom fastdfs – spring – the boot – the starter

To quickly build a SpringBoot project, we can customize a scenario launcher. Details: github.com/bluemiaomia… .

Let’s build a sample project using SpringBoot:

Add dependencies:

<dependency> <groupId>com.bluemiaomiao</groupId> <artifactId>fastdfs-spring-boot-starter</artifactId> < version > 1.0 - the SNAPSHOT < / version > < / dependency >Copy the code

Add annotations to the main configuration class:

@EnableFastdfsClient @SpringBootApplication public class DemoApplication { @Autowired private FastdfsClientService fastdfsClientService; public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); }}Copy the code

The global client is automatically initialized.

Interested in the article friends can pay attention to xiaobian nuggets good, there will be more quality article output.

FastDFS Distributed file clustering for Java Advanced Architecture

FastDFS profile

Comparison of storage systems

FastDFS client – server interaction principle

FastDFS + Nginx integration

Token-based anti-theft chain implementation

Integrate FastDHT for data de-duplication

Custom fastdfs – spring – the boot – the starter

Related Posts

Use raspberry PI and idle hard disk, build up the personal network disk

Spring Boot log collection and link tracing

How to make a reliable message queue with Redis