Distributed cluster architecture

Double the Tracker
2 set of Group,
Polling storage policies
Keepalived + Nginx high availability
Nginx cache
Four storage nodes

1. Cluster Planning List

1. Installation list

The name of the software	version	Baidu cloud disk storage name
FastDFS	5.11	Fastdfs – 5.11. Zip
FastDFS-Nginx-module	There is no	fastdfs-nginx-module-master.zip
LibFastCommon	1.0.36	Libfastcommon – 1.0.36. Zip
nginx	1.10.3	Nginx – 1.10.3. Tar. Gz
nginx-pure-cache	2.3	Ngx_cache_purge – 2.3. Tar. Gz

Installation required files are uploaded to Baidu cloud disk, location: FastDFS Baidu cloud disk

2. Plan the cluster

The virtual machine	IP	instructions
Keepalived+Nginx1[Master]	192.168.43.101	Nginx Server 01
Keeepalived+Nginx[Backup]	192.168.43.102	Nginx Server 02
VIP	192.168.43.150	Virtual drift IP
Tracker01	192.168.43.70	Tracker01 server
Tracker02	192.168.43.71	Tracker02 server
Storage01	192.168.43.72	Storage01 server [group1]
Storage02	192.168.43.73	Storage02 server [group1]
Storage03	192.168.43.74	Storage03 Server [group2]
Storage04	192.168.43.75	Storage04 server group2

The overall architecture is shown as follows:

2. Cluster installation

The following operations are performed before disabling firewalls on all nodes. Enable ports or disable firewalls based on your personal situation

1. Install the LibFastCommon/FastDFS module

Run nodes Tracker01, Tracker02, Storage01, Storage03, and Storage04

[root@localhost fastDFS]# unzip libfastcommon - 1.0.36. Zip
Copy the code

The decompressed directory is as follows:

[root @ localhost fastdfs 5.11]# ll[root @ localhost libfastcommon - 1.0.36]# ll32 drwxr-xr-x. 2 root root 117 4月 5 2017 doc-rw-r --r--. 1 root root 8005 4月 5 2017 history-rw-r --r-- 566 4月 5 2017 INSTALL -rw-r--r--. 1 root root 1606 4月 5 2017 libfastcommon.spec -rwxr-xr-x. 1 root root 3099 4月 5 2017 Make. Sh drwxr-xr-x. 2 root root 191 4月 5 2017 php-fastcommon-rw-r --r-- 1 root root 2763 4月 5 2017 README drwxr-xr-x Root root 4096 January 17 11:21 SRCCopy the code

Install the C compiler GCC

[root @ localhost fastdfs 5.11]# yum -y install gcc-c++
Copy the code

Installed with LibFastCommon

Compile and install the libfastcommon software

[root @ localhost fastdfs 5.11]# ./make.sh && ./make.sh install
Copy the code

Create a soft link for libcommon to the /usr/local/lib directory

[root @ localhost fastdfs 5.11]# ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so[root @ localhost fastdfs 5.11]# ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so[root @ localhost fastdfs 5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so[root @ localhost fastdfs 5.11]# ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
Copy the code

Install FastDFS and decompress the installation package

[root@localhost fastDFS]# unzip fastdfs - 5.11. Zip
Copy the code

Go to the decompressed directory and compile and install the software

[root@localhost fastDFS]# CD fastdfs - 5.11 /[root @ localhost fastdfs 5.11]# ./make.sh && ./make.sh install
Copy the code

After the FastDFS installation is successful, the FastDFS will be installed in /etc/fdfs:

[root @ localhost fastdfs 5.11]# ll /etc/fdfs/Total usage 76 -rw-r--r--. 1 root root 316 January 17 11:47 client.conf -rw-r--r--. 1 root root 1461 January 17 11:25 client.conf.sample 76 -rw-r--r-- -rw-r--r-- 1 root root 955 1月 17 13:20 http.conf -rw-r--r-- 1 root root 31172 1月 17 13:21 MIME. Types -rw-r--r-- 1 Root root 3716 1月 17 12:57 mod_fastdfs.conf -rw-r--r--. 1 root root 1278 1月 17 11:40 storage.conf -rw-r--r-- Root 7927 January 17 11:25 storage.conf.sample -rw-r--r-- 1 root root 105 January 17 11:25 storage_ids.conf.sample -rw-r--r-- 1 Root root 1356 1月 17 11:34 tracker.conf -rw-r--r--. 1 root root 7389 1月 17 11:25 tracker.conf.sampleCopy the code

We need to make a copy of these three sample files and remove the. Sample

[root@localhost fdfs]# cp client.conf.sample client.conf
[root@localhost fdfs]# cp storage.conf.sample storage.conf
[root@localhost fdfs]# cp tracker.conf.sample tracker.conf
Copy the code

The FastDFS installation is complete

2. Install the Tracker and configure the node information

Run Tracker01 and Tracker02

Create the tracker working directory

This directory is used to store data and logs for the tracker

[root@localhost fdfs]# mkdir /opt/fastdfs_tracker
Copy the code

Configure the tracker

Configure tracker.conf in the /etc/fdfs directory to implement the following five configurations:

1.disabled=false 
2.port=22122 # Default port number
3.base_path=/opt/fastdfs_tracker # directory I just created
4.http.server_port=8080 The default port is 8080
5.store_lookup=0  # Use polling policy for storage, 0 polling 1: always directed to a group 2: load for storing files
Copy the code

The complete tracker.conf file contains the following information:

disabled=falseBind_addr = 0.0.0.0 port=22122 CONNECt_TIMEOUT =30 network_timeout=60 base_path=/opt/ fastDFS_tracker max_connections=512 accept_threads=1 work_threads=4 min_buff_size = 8KB max_buff_size = 128KB store_lookup=0 store_group=group2 store_server=0 store_path=0 download_server=0 reserved_storage_space = 10% log_level=info run_by_group= run_by_user= allow_hosts=* sync_log_buff_interval = 10 check_active_interval = 120 thread_stack_size = 64KB storage_ip_changed_auto_adjust =true

storage_sync_file_max_delay = 86400

storage_sync_file_max_time = 300

use_trunk_file = false 

slot_min_size = 256

slot_max_size = 16MB

trunk_file_size = 64MB

trunk_create_file_advance = false

trunk_create_file_time_base = 02:00

trunk_create_file_interval = 86400

trunk_create_file_space_threshold = 20G

trunk_init_check_occupying = false

trunk_init_reload_from_binlog = false

trunk_compress_binlog_min_interval = 0

use_storage_id = false

storage_ids_filename = storage_ids.conf

id_type_in_filename = ip

store_slave_file_use_link = false

rotate_error_log = false

error_log_rotate_time=00:00

rotate_error_log_size = 0

log_file_keep_days = 0

use_connection_pool = false

connection_pool_max_idle_time = 3600

http.server_port=8080

http.check_alive_interval=30

http.check_alive_type=tcp

http.check_alive_uri=/status.html
Copy the code

Create a soft reference after saving the modification

[root@localhost fdfs]# ln -s /usr/bin/fdfs_storaged /usr/local/bin
Copy the code

Start the tracker and add the boot option

[root@localhost fdfs]# service fdfs_trackerd start
Copy the code

Add tracker to the boot option

[root@localhost fdfs]# echo "service fdfs_trackerd start" |tee -a /etc/rc.d/rc.local
Copy the code

3. Install and configure the Storage module

Run Storage01, Storage02, Storage03, and Storage04

Creating a Storage directory

Two directories fastDFs_storage_data and fastDFs_storage are created on each storage node

[root@localhost opt]# mkdir fastdfs_storage
[root@localhost opt]# mkdir fastdfs_storage_data
[root@localhost opt]# llDrwxr-xr-x. 4 Root root 30 January 17 11:45 fastDFs_storage drwxr-xr-x. 3 Root root 18 January 17 11:45 fastDFs_storage_data Drwxr-xr-x. 4 root root 30 1月 17 11:35 fastdfs_trackerCopy the code

Modify the /etc/ff/storage. conf configuration information in the storage node directory as follows:

disabled=false Enable the configuration file
group_name=group1 Group name = group1; group name = group2;
port=23000 The storage ports of the same group must be the same
base_path=/opt/fastdfs_storage Set storage data file and log directory
store_path0=/opt/fastdfs_storage_data The actual file storage path
store_path_count=1 Number of storage paths, which must match store_path numberTracker_server = 192.168.43.70:22122The IP address and port of the tracker serverTracker_server = 192.168.43.70:22122Add multiple configurations directly to multiple trackers
http.server_port=8888 Set the HTTP port number
Copy the code

Complete configuration information is as follows:

disabled=false
group_name=group1
bind_addr=
client_bind=true
port=23000
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
base_path=/opt/fastdfs_storage
max_connections=256
buff_size = 256KB
accept_threads=1
work_threads=4
disk_rw_separated = truedisk_reader_threads = 1 disk_writer_threads = 1 sync_wait_msec=50 sync_interval=0 sync_start_time=00:00 sync_end_time=23:59 write_mark_file_freq=500 store_path_count=1 store_path0=/opt/fastdfs_storage_data Subdir_count_per_path = 256 tracker_server = 192.168.43.70:22122 tracker_server = 192.168.43.71:22122 log_level = info run_by_group= run_by_user= allow_hosts=* file_distribute_path_mode=0 file_distribute_rotate_count=100 fsync_after_written_bytes=0 sync_log_buff_interval=10 sync_binlog_buff_interval=10 sync_stat_file_interval=300 thread_stack_size=512KB upload_priority=10 if_alias_prefix= check_file_duplicate=0 file_signature_method=hash
key_namespace=FastDFS
keep_alive=0
use_access_log = false
rotate_access_log = false
access_log_rotate_time=00:00
rotate_error_log = false
error_log_rotate_time=00:00
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record=false
use_connection_pool = false
connection_pool_max_idle_time = 3600
http.domain_name=
http.server_port=8888
Copy the code

Start Storage After each node has been configured, start the Storage

[root@localhost fdfs]# service fdfs_storaged start
Copy the code

View logs after startup

[root@localhost fdfs]# fdfs_monitor /etc/fdfs/storage.conf
[2018-01-20 16:56:48] DEBUG - base_path=/opt/fastdfs_storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=2, server_index=0

tracker server is 192.168.43.70:22122

group count: 2

Group 1:
group name = group1
disk total space = 47073 MB
disk free space = 35162 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

	Storage 1:
		id = 192.168.43.72
		ip_addr = 192.168.43.72 (localhost.localdomain)  ACTIVE
		http domain = 
		version = 5.11
		join time = 2018-01-19 13:59:30
		up time = 2018-01-20 12:37:18
		total storage = 47073 MB
		free storage = 35162 MB
		upload priority = 10
		store_path_count = 1
		subdir_count_per_path = 256
		storage_port = 23000
		storage_http_port = 8888
		current_write_path = 0
		sourcestorage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 3 success_upload_count = 3 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 791904 success_upload_bytes = 791904 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 775234 success_sync_in_bytes = 775234 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 4 success_file_open_count = 4 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 8 success_file_write_count = 8 last_heart_beat_time = 2018-01-20 16:56:18 last_source_update = 2018-01-19 19:34:55 last_sync_update = 2018-01-19 15:28:56 last_synced_timestamp = 2018-01-19 15:28:48 (0s delay) Storage 2: Id = 192.168.43.73 IP_ADDR = 192.168.43.73 ACTIVE HTTP domain = version = 5.11 Join time = 2018-01-19 14:00:21 Up time =  2018-01-20 12:37:42 total storage = 47073 MB free storage = 35166 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0sourceStorage ID = 192.168.43.72 IF_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 1 success_upload_count = 1 total_append_count = 0 success_append_count = 0  total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count  = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 775234 success_upload_bytes = 775234 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 791904 success_sync_in_bytes = 791904 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 4 success_file_open_count = 4 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 8 success_file_write_count = 8 last_heart_beat_time = 2018-01-20 16:56:42 last_source_update = 2018-01-19 15:28:48 last_sync_update = 2018-01-19 19:34:59 last_synced_timestamp = 2018-01-19 19:34:55 (0s delay) Group 2: group name = group2 disk total space = 47073 MB disk free space = 35165 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: Id = 192.168.43.74 IP_ADDR = 192.168.43.74 ACTIVE HTTP domain = version = 5.11 Join time = 2018-01-19 14:01:05 Up time =  2018-01-20 12:38:00 total storage = 47073 MB free storage = 35165 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0sourcestorage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 4 success_upload_count = 4 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 2107770 success_upload_bytes = 2107770 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 1550468 success_sync_in_bytes = 1550468 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 6 success_file_open_count = 6 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 15 success_file_write_count = 15 last_heart_beat_time = 2018-01-20 16:56:38 last_source_update = 2018-01-19 19:35:40 last_sync_update = 2018-01-19 15:28:53 last_synced_timestamp = 2018-01-19 15:28:50 (-1s delay) Storage 2: Id = 192.168.43.75 IP_addr = 192.168.43.75 ACTIVE HTTP domain = version = 5.11 Join time = 2018-01-19 14:01:27 Up time =  2018-01-20 12:38:20 total storage = 47073 MB free storage = 35165 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0sourceStorage ID = 192.168.43.74 IF_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 2 success_upload_count = 2 total_append_count = 0 success_append_count = 0  total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count  = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 1550468 success_upload_bytes = 1550468 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 2107770 success_sync_in_bytes = 2107770 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 6 success_file_open_count = 6 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 15 success_file_write_count = 15 last_heart_beat_time = 2018-01-20 16:56:23 last_source_update = 2018-01-19 15:28:49 last_sync_update = 2018-01-19 19:35:46 last_synced_timestamp = 2018-01-19 19:35:40 (0s delay)Copy the code

If two groups of Storage information are displayed, the configuration information is successfully configured and registered in the Tracker to view the log startup status

[root@localhost fdfs]# tail -f /opt/fastdfs_storage/logs/storaged.log [2018-01-20 12:37:18] info-fastdfs v5.11, base_path=/opt/ fastDFS_storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2018-01-20 12:37:18] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0
[2018-01-20 12:37:18] INFO - file: storage_func.c, line: 257, tracker_client_ip: 192.168.43.72, my_server_id_str: 192.168.43.72, g_server_id_in_filename: 1210820800
[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.43.71:22122, as a tracker client, my ip is 192.168.43.72
[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 1947, tracker server: #0. 192.168.43.70:22122, my_report_status: -1[2018-01-20 12:37:18] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.43.70:22122, as a tracker client, My IP is 192.168.43.72 [2018-01-20 12:37:18] info-file: tracker_client_thread.c, line: 1947, tracker server:#0. 192.168.43.70:22122, my_report_status: -1[2018-01-20 12:37:48] info-file: tracker_client_thread.c, line: 1263, tracker server 192.168.43.71:22122,setTracker leader: 192.168.43.71:22122 [2018-01-20 12:37:48] info-file: storage_sync.c, line: 2732, successfully connect to storage server 192.168.43.73:23000Copy the code

192.168.43.71 acts as the Leader of the Tracker.

Set the startup of the Storage

[root@localhost fdfs]# echo "service fdfs_storaged start" |tee -a /etc/rc.d/rc.local
Copy the code

Install fastdfs-nginx-module and nginx modules

Dependencies required to install Nginx modules

[root@localhost fdfs]# yum -y install pcre pcre-devel  
[root@localhost fdfs]# yum -y install zlib zlib-devel  
[root@localhost fdfs]# yum -y install openssl openssl-devel
Copy the code

Unzip nginx and fastdfs-nginx-module

[root@localhost fdfs]# tar - ZXVF nginx - 1.10.3. Tar. Gz
[root@localhost fdfs]# unzip fastdfs-nginx-module-master.zip
Copy the code

Go to the Nginx decompression directory to compile and install

[root @ localhost nginx - 1.10.3]# ./configure --prefix=/usr/local/nginx --add-module=/home/zhangyongliang/apps/fastdfs-nginx-module-master/src # fastdfs-nginx-module location after decompression
Copy the code

Nginx is installed in /usr/local/nginx

[root@localhost src]# ll /usr/local/nginx/Total usage 8 DRWX ------. 2 nobody root 6 1月 17 13:23 client_body_temp drwxr-xr-x. 2 root root 4096 1月 17 13:17 conf DRWX ------ 2 Nobody root 6 January 17 13:23 fastcgi_temp drwxr-xr-x. 2 Root root 40 January 17 13:17 HTML drwxr-xr-x. 2 root root 58 January 17 13:49 logs-rw-r --r--. 1 root root 1156 1月 17 13:29 nginx.conf DRWX ------. 2 nobody root 6 1月 17 13:23 proxy_temp 2 root root 19 January 17 13:17 sbin DRWX ------. 2 Nobody root 6 January 17 13:23 scgi_temp DRWX ------. 2 nobody root 6 January 17 13:23 uWSGI_tempCopy the code

The nginx folder does not have temporary folders, such as fastcgi_temp, when nginx is not yet running. Copy the fastdfs-nginx-module configuration file to /etc/fdfs and modify it

[root@localhost src]# cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/  
[root@localhost src]# vi /etc/fdfs/mod_fastdfs.conf  
Copy the code

Mod_fastdfs. conf for group 1 Storage: 10 base_path connect_timeout = = / opt/fastdfs_storage tracker_server = 192.168.1.131:22122 tracker_server = 192.168.1.132:22122  storage_server_port=23000 group_name=group1 url_have_group_name =truestore_path0=/opt/fastdfs_storage_data group_count = 2 [group1] group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/opt/fastdfs_storage_data [group2] group_name=group2 storage_server_port=23000 Store_path_count =1 store_path0=/opt/ fastDFs_storage_data (2) The mod_fastdfs.conf configuration of group 2 Storage is different from that of group 1 only with group_name:  group_name=group2Copy the code

The full information is as follows:

# connect timeout in seconds
# default value is 30s
connect_timeout=2

# network recv and send timeout in seconds
# default value is 30s
network_timeout=30

# the base path to store log files
base_path=/opt/fastdfs_storage

# if load FastDFS parameters from tracker server
# since V1.12
# default value is false
load_fdfs_parameters_from_tracker=true

# storage sync file max delay seconds
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V1.12
# default value is 86400 seconds (one day)
storage_sync_file_max_delay = 86400

# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V1.13
use_storage_id = false

# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V1.13
storage_ids_filename = storage_ids.conf

# FastDFS tracker_server can ocur more than once, and tracker_server format is
# "host:port", host can be hostname or ip address
# valid only when load_fdfs_parameters_from_tracker is trueTracker_server = 192.168.43.71 tracker_server = 192.168.43.70:22122:22122# the port of the local storage server
# the default value is 23000
storage_server_port=23000

# the group name of the local storage server
group_name=group1

# if the url / uri including the group name
# set to false when uri like /M00/00/00/xxx
# set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx
# default value is false
url_have_group_name = true

# path(disk or mount point) count, default value is 1
# must same as storage.conf
store_path_count=1

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
# must same as storage.conf
store_path0=/opt/fastdfs_storage_data
#store_path1=/home/yuqing/fastdfs1

# standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info

# set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log
# empty for output to stderr (apache and nginx error_log file)
log_filename=

# response mode when the file not exist in the local file system
## proxy: get the content from other storage server, then send to client
## redirect: redirect to the original storage server (HTTP Header is Location)
response_mode=proxy

# the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a
# multi aliases split by comma. empty value means auto set by OS type
# this paramter used to get all ip address of the local host
# default values is empty
if_alias_prefix=

# use "#include" directive to include HTTP config file
# NOTE: #include is an include directive, do NOT remove the # before include
#include http.conf


# if support flv
# default value is false
# since v1.15
flv_support = true

# flv file extension name
# default value is flv
# since v1.15
flv_extension = flv


# set the group count
# set to none zero to support multi-group on this storage server
# set to 0 for single group only
# groups settings section as [group1], [group2], ... , [groupN]
# default value is 0
# since v1.14
group_count = 2

# group settings for group #1
# since v1.14
# when support multi-group on this storage server, uncomment following section
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs_storage_data
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs_storage_data
# group settings for group #2
# since v1.14
# when support multi-group, uncomment following section as neccessary
#[group2]
#group_name=group2
#storage_server_port=23000
#store_path_count=1
#store_path0=/home/yuqing/fastdfs
Copy the code

Copy some configuration files from the FastDFS installation directory to the /etc/fdfs directory

[root@localhost conf]# pwd/ home/zhangyongliang/apps/fastdfs - 5.11 / conf/root @ localhost conf]# cp http.conf mime.types /etc/fdfs/
Copy the code

Create symbolic link from M00 to storage directory:

ln  -s  /opt/fastdfs_storage_data/data/ /opt/fastdfs_storage_data/data/M00
Copy the code

Configure Nginx, concise version of the Nginx configuration example

# vi /usr/local/nginx/conf/nginx.conf  
user root;  
worker_processes 1;  
events {  
    worker_connections 1024;  
}  
http {  
    include mime.types;  
    default_type application/octet-stream;  
    sendfile on;  
    keepalive_timeout 65;  
    server {  
        listen 8888;  
        server_name localhost;  
        location ~/group([0-9])/M00 {  
            #alias /fastdfs/storage/data;  ngx_fastdfs_module; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; }}}Copy the code

Server_port =8888 in /etc/fdfs/storage.conf. The default value of http.server_port is 8888. If you want to change it to 80, change it accordingly. Restart the Nginx service on each node

[root@localhost conf]#/usr/local/nginx/sbin/nginx -s reload
Copy the code

4. File upload test

Run Tracker01 and Tracker02 to modify the client configuration file in the Tracker server

# vi /etc/fdfs/client.conf  Base_path = / fastdfs/tracker tracker_server = 192.168.43.70:22122 tracker_server = 192.168.43.71:22122Copy the code

Run the following command to upload files

[root@localhost zhangyongliang]# fdfs_upload_file /etc/fdfs/client.conf P71022-205803.jpg 
group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg
[root@localhost zhangyongliang]# fdfs_upload_file /etc/fdfs/client.conf P71022-205803.jpg 
group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg
Copy the code

After two uploads, the files are evenly divided into two groups.

5.Tracker Install the Nginx and ngx_cache_purge modules

Install the dependency packages required to compile Nginx

[root@localhost zhangyongliang]# yum install gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel openssl openssl-devel  
Copy the code

Unzip the Nginx and ngx_cache_pure modules

[root@localhost apps]# tar ngx_cache_purge - 2.3. Tar. Gz
[root@localhost apps]# tar nginx - 1.10.3. Tar. Gz
Copy the code

Compile and install Nginx (add ngx_cache_purge module)

[root@localhost apps]# CD nginx - 1.13.0[root @ localhost nginx - 1.13.0#. / configure -- prefix = / usr/local/nginx - add - the module = / usr/local/SRC/ngx_cache_purge - 2.3[root @ localhost nginx - 1.13.0]# make && make install 
Copy the code

Configure Nginx, set up load balancing and caching

# vi /usr/local/nginx/conf/nginx.conf  

#user nobody;
worker_processes  1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    # '$status $body_bytes_sent "$http_referer" '
    # '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log logs/access.log main;
    sendfile        on;  
    tcp_nopush      on;  
    #tcp_nopush on;
    #keepalive_timeout 0;
    keepalive_timeout  65;

    #gzip on;
	# set cache
    server_names_hash_bucket_size 128;  
    client_header_buffer_size 32k;  
    large_client_header_buffers 4 32k;  
    client_max_body_size 300m;  
  
    proxy_redirect off;  
    proxy_set_header Host $http_host;  
    proxy_set_header X-Real-IP $remote_addr;  
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
    proxy_connect_timeout 90;  
    proxy_send_timeout 90;  
    proxy_read_timeout 90;  
    proxy_buffer_size 16k;  
    proxy_buffers 4 64k;  
    proxy_busy_buffers_size 128k;  
    proxy_temp_file_write_size 128k;
	Set cache storage path, storage mode, respectively memory size, maximum disk space, cache duration
    proxy_cache_path /opt/fastdfs_tracker/proxy_cache levels=1:2 
	keys_zone=http-cache:200m max_size=1g inactive=30d;
    proxy_temp_path /opt/fastdfs_tracker/tmp;
    #group1 service SettingsUpstream =1 max_fails=2 fail_timeout=30s; Server 192.168.43.73:8888 weight=1 max_fails=2 fail_timeout=30s; }# Group2 service SettingsUpstream =1 max_fails=2 fail_timeout=30s; Server 192.168.43.75:8888 weight=1 max_fails=2 fail_timeout=30s; } server { listen 8000; server_name localhost;#charset koi8-r;

        #access_log logs/host.access.log main;
		# load balancing configuration for group1
        location /group1/M00 {  
            proxy_next_upstream http_502 http_504 error timeout invalid_header;  
            proxy_cache http-cache;  
            proxy_cache_valid 200 304 12h;  
            proxy_cache_key $uri$is_args$args;  
            # Service Settings for group1
            proxy_pass http://fdfs_group1;  
            expires 30d;  
        }
        location /group2/M00 {  
            proxy_next_upstream http_502 http_504 error timeout invalid_header;  
            proxy_cache http-cache;  
            proxy_cache_valid 200 304 12h;  
            proxy_cache_key $uri$is_args$args;  
            # Service Settings for group2proxy_pass http://fdfs_group2; expires 30d; } location ~/purge(/.*) {allow 127.0.0.1; Allow 192.168.43.0/24; deny all; proxy_cache_purge http-cacheThe $1$is_args$args;  
        } 		 
        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page 404 /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        # proxy_pass http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        # root html;
        # fastcgi_pass 127.0.0.1:9000;
        # fastcgi_index index.php;
        # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
        # include fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        # deny all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    # listen 8000;
    # listen somename:8080;
    # server_name somename alias another.alias;

    # location / {
    # root html;
    # index index.html index.htm;
    #}
    #}


    # HTTPS server
    #
    #server {
    # listen 443 ssl;
    # server_name localhost;

    # ssl_certificate cert.pem;
    # ssl_certificate_key cert.key;

    # ssl_session_cache shared:SSL:1m;
    # ssl_session_timeout 5m;

    # ssl_ciphers HIGH:! aNULL:! MD5;
    # ssl_prefer_server_ciphers on;

    # location / {
    # root html;
    # index index.html index.htm;
    #}
    #}

}
Copy the code

Create folders in the corresponding directories according to the Nginx configuration

[root@localhost fastdfs_tracker]# mkdir proxy_cache tmp
[root@localhost fastdfs_tracker]# ll2 Root root 178 1月 20 12:37 data drwxr-xr-x. 2 root root 26 1月 19 12:01 logs drwxr-xr-x. 7 Nobody root 51 1月 19 19:35 proxy_cache drwxr-xr-x. 2 Nobody root 6 1月 19 19:35 TMPCopy the code

Restart Nginx for access testing

Restart Nginx [root@localhost fastdfs_tracker]# /usr/local/nginx/sbin/nginx -s reload 
Copy the code

Directly in front of the access Storage nodes in Nginx file http://192.168.43.72:8888/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg] http://192.168.43.74:8888/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg can now access through the Nginx Tracker (1) The Nginx Tracker1 to visit http://192.168.43.70:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg http://192.168.43.70:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg (2) to access through the Nginx Tracker2 http://192.168.43.71:8000/group1/M00/00/00/wKgrSFpjC26AH1g2AAvUQrxXbkA557.jpg http://192.168.50.71:8000/group2/M00/00/00/wKgrSlpjC3aAARrXAAvUQrxXbkA048.jpg

6. Construct Keepalive+Nginx to implement virtual IP proxy

Keepalived+Nginx+Tomcat to implement high availability Web clustering is not described in this article

Keepalvie+nginx Master
Start the Keepalvie+nginx BackUp standby node [192.168.43.102] modify the nginx.conf configuration file of the nginx directory on the two nodes. Add the following information: 2 trackers are added to the Keepalive+nginx agent

Upstream fastdfs_tracker {server 192.168.43.70:8000 weight=1 max_fails=2 fail_timeout=30s; Server 192.168.43.71:8000 weight=1 max_fails=2 fail_timeout=30s; }Copy the code

The second change is to add a location and the matching rule is fastdfs in the path

location /fastdfs {  
           root html;  
           index index.html index.htm;  
           proxy_pass http://fastdfs_tracker/;  
           proxy_set_header Host $http_host;  
           proxy_set_header Cookie $http_cookie;  
           proxy_set_header X-Real-IP $remote_addr;  
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
           proxy_set_header X-Forwarded-Proto $scheme;  
           client_max_body_size 300m;  
        }  
Copy the code

The complete configuration code is as follows:


#user nobody;
worker_processes  1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    # '$status $body_bytes_sent "$http_referer" '
    # '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log logs/access.log main;

    sendfile        on;
    #tcp_nopush on;

    #keepalive_timeout 0;
    keepalive_timeout  65;

    #gzip on;Upstream fastdfs_tracker {server 192.168.43.70:8000 weight=1 max_fails=2 fail_timeout=30s; Server 192.168.43.71:8000 weight=1 max_fails=2 fail_timeout=30s; Upstream {server 192.168.43.103:8080 weight=1; Server 192.168.43.104:8080 weight = 1; } server { listen 80; server_name localhost;#charset koi8-r;

        #access_log logs/host.access.log main;

        location / {
		    proxy_pass http://tomcat;
			proxy_set_header X-NGINX "NGINX-1";
            root   html;
            index  index.html index.htm;
        }

        #error_page 404 /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
		location /fastdfs {  
           root html;  
           index index.html index.htm;  
           proxy_pass http://fastdfs_tracker/;  
           proxy_set_header Host $http_host;  
           proxy_set_header Cookie $http_cookie;  
           proxy_set_header X-Real-IP $remote_addr;  
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
           proxy_set_header X-Forwarded-Proto $scheme;  
           client_max_body_size 300m;  
        }  

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        # proxy_pass http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        # root html;
        # fastcgi_pass 127.0.0.1:9000;
        # fastcgi_index index.php;
        # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
        # include fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        # deny all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    # listen 8000;
    # listen somename:8080;
    # server_name somename alias another.alias;

    # location / {
    # root html;
    # index index.html index.htm;
    #}
    #}


    # HTTPS server
    #
    #server {
    # listen 443 ssl;
    # server_name localhost;

    # ssl_certificate cert.pem;
    # ssl_certificate_key cert.key;

    # ssl_session_cache shared:SSL:1m;
    # ssl_session_timeout 5m;

    # ssl_ciphers HIGH:! aNULL:! MD5;
    # ssl_prefer_server_ciphers on;

    # location / {
    # root html;
    # index index.html index.htm;
    #}
    #}

}
Copy the code

After the modification, restart the active and standby Keepalive+Nginx2 nodes.

[root@nginx1 conf]# /usr/local/nginx/sbin/nginx -s reload
Copy the code

We will now use virtual IP192.168.43.150 to access the image we just uploaded, just remember to type fastdfs in the address bar (as specified in the location /fastdfs{} rule in our nginx.conf file). As shown in the figure below, we found that we can access the pictures we uploaded through the virtual IP. The advantage of this is that the user only needs to access the virtual IP, regardless of the forwarding mechanism within the FastDFS cluster.

Supplementary notes:

If the Tracker, Storage, or Nginx service does not start automatically after startup, perform the following operations and restart the service

[root@localhost ~]# chkconfig --add fdfs_trackerd
[root@localhost ~]# chkconfig --add fdfs_storaged
Copy the code

Edit /etc/rc.d/rc.local as follows:

#! /bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
/usr/local/nginx/sbin/nginx
Copy the code

Nginx is mainly added to start, after file authorization, effective file, restart the system

[root@localhost ~]# chmod +x /etc/rc.d/rc.local
[root@localhost ~]# source /etc/rc.d/rc.local 
[root@localhost ~]# reboot
Copy the code

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

CentOS 7 2 Set up the FastDFS distributed file system to implement the HA cluster

Distributed cluster architecture

1. Cluster Planning List

1. Installation list

2. Plan the cluster

2. Cluster installation

1. Install the LibFastCommon/FastDFS module

2. Install the Tracker and configure the node information

3. Install and configure the Storage module

4. File upload test

5.Tracker Install the Nginx and ngx_cache_purge modules

6. Construct Keepalive+Nginx to implement virtual IP proxy

Supplementary notes:

CentOS 7 2 Set up the FastDFS distributed file system to implement the HA cluster

Distributed cluster architecture

1. Cluster Planning List

1. Installation list

2. Plan the cluster

2. Cluster installation

1. Install the LibFastCommon/FastDFS module

2. Install the Tracker and configure the node information

3. Install and configure the Storage module

4. File upload test

5.Tracker Install the Nginx and ngx_cache_purge modules

6. Construct Keepalive+Nginx to implement virtual IP proxy

Supplementary notes:

Related Posts

Restful API learning summary

Java Stream API (Part 2)

Use OpenCV to realize face recognition