I haven’t seen you for a long time. I have been busy preparing for my new job for a long time over the weekend, so I didn’t spend any time writing articles. I have a schedule today, but I still want to spare some time to write an article summarizing the technical stack of this project.
Before the article begins, I’d like to say a few words about my idea of finding a job. With the influence of the open source community and some Youtube programmers UP, for my next job, it should be a job THAT I enjoy and have an English working environment. Happy coding can make a really popular program: -d and you won’t feel tired. I look forward to one day having an open source project as popular as Redis.
The following content is the project construction and configuration of the production environment of real enterprises. Due to the special population of the project, the system usage is not high and the concurrency is not large, so the complex design is not adopted. The default configuration also meets the pressure test requirements, so the configuration is not optimized too much.
System Architecture Diagram
Overall request flow diagram
Detailed request flow diagram
After the reader has a general understanding of the system framework through the above 3 figures, the following is attached with the steps of building all the components, a project named IURC.
Component deployment and configuration steps
1. Build a basic server environment
1.1 Creating robin users for All Servers
Assume that the IP addresses of nine servers are 10.210.111.12 to 10.210.111.20. The real environment needs to be logged in through fortress computer, assuming that you can directly log in to the server.
Log in to server 10.210.111.12 using SSH as user root and create the init_server.sh script
vim init_server.sh
#Create user Robin
adduser -m robin
#Changing a User password
passwd robin
#Install the telent
yum -y install telnet
Copy the codeCreate the send_all_Servers. sh script to distribute init_server.sh to all servers
vim send_all_servers.sh
#Write the distribution command to the script (this step is to save time creating users, but you still need to log into each server to execute the script)SCP $1 [email protected]:/root/ SCP $1 [email protected]:/root/ SCP $1 [email protected]:/root/ SCP $1 [email protected]:/root/ SCP $1 [email protected]:/root/ SCP $1 [email protected]:/root/ SCP $1 [email protected]: / root/SCP $1 [email protected]: / root /Copy the codeExecute the send_all_Servers. sh script (the password of the target server needs to be entered after the script is executed.)
chmod +x ./send_all_servers.sh
bash ./send_all_servers.sh ./init_server.sh
Copy the codeRun the init_server.sh script on all servers
chmod +x ./init_server.sh
bash ./init_server.sh
Copy the code1.2 All Servers Authorize Robin users
Example Add write permission to the sudoers file
chmod u+w /etc/sudoers
Copy the codeModify the sudoers file
vim /etc/sudoers
Copy the codeFind the figure below in the sudoers file
Add the following below the root line
robin ALL=(ALL) ALL
Copy the codeRevoke the write permission on sudoers files
chmod u-w /etc/sudoers
Copy the code⚠️ All ECS require login to perform authorization operations
1.3 Restoring the Server host name
Example Change the 10.210.111.12 server host name
hostnamectl set-hostname nginx_01
reboot
Copy the codeYou are advised to change all host names to services provided by the server to facilitate subsequent o&M. I set the host name as follows:
| The IP address | system | The host name |
|---|---|---|
| 10.210.111.12 | CentOS7.x | nginx_01 |
| 10.210.111.13 | CentOS7.x | nginx_02 |
| 10.210.111.14 | CentOS7.x | redis_01 |
| 10.210.111.15 | CentOS7.x | redis_02 |
| 10.210.111.16 | CentOS7.x | redis_03 |
| 10.210.111.17 | CentOS7.x | db_01 |
| 10.210.111.18 | CentOS7.x | db_02 |
| 10.210.111.19 | CentOS7.x | app_02 |
| 10.210.111.20 | CentOS7.x | app_02 |
2. Nginx installation and configuration
Server Configuration
| The IP address | system | function |
|---|---|---|
| 10.210.111.12 | CentOS7.x | Nginx |
| 10.210.111.13 | CentOS7.x | Nginx |
2.1 install Nginx
Log in to the server as Robin user nginx_0110.210.111.12 using SSH, download Nginx, and unzip it.
CD/home/robin wget http://nginx.org/download/nginx-1.20.1.tar.gz tar - ZXVF nginx - 1.20.1. Tar. Gz && CD nginx - 1.20.1Copy the codeInstall the precompiled environment
sudo yum update
sudo yum -y install gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel
Copy the codeCompile the installation are replaced with the following configuration/usr/local/nginx/conf/nginx. The content in the conf
CD /home/robin/ nginx-1.20.1. /configure --prefix=/usr/local/nginx make sudo make installCopy the code2.2 configure Nginx
Replace with the following configuration/usr/local/nginx/conf/nginx. The content of the conf
worker_processes 2; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location @router{ rewrite ^.*$ /iurc/index.html break; } location ^~/iurc { root /usr/local/nginx/html; try_files $uri $uri/ @router; } location ^~/iurc/api { proxy_pass http://iurcProxy/iurc/api/channel/http.do; proxy_send_timeout 1800; proxy_read_timeout 1800; proxy_connect_timeout 1800; client_max_body_size 2048m; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location @router_admin{ rewrite ^.*$ /iurc-admin/index.html break; } location ^~/iurc-admin {allow 192.168.10.0/24; Allow 192.168.12.0/24; Allow 192.168.30.0/24; Allow 192.168.160.0/24; deny all; root /usr/local/nginx/html; try_files $uri $uri/ @router_admin; } location ^~/iurc-admin/ API {allow 192.168.10.0/24; Allow 192.168.12.0/24; Allow 192.168.30.0/24; Allow 192.168.160.0/24; deny all; proxy_pass http://iurcAdminProxy/iurc/admin/api/channel/http.do; proxy_send_timeout 1800; proxy_read_timeout 1800; proxy_connect_timeout 1800; client_max_body_size 2048m; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location = /50x.html { root html; Upstream {server 10.210.111.19:30068; Server 10.210.111.20:30068; } upstream iurcAdminProxy {server 10.210.111.19:30070; Server 10.210.111.20:30070; }}Copy the code2.3 start the Nginx
sudo /usr/local/nginx/sbin/nginx
Copy the code⚠️ Install and configure Nginx on the nginx_0210.111.13 server by following steps 2.1-2.3
2.4 Creating an HTML Directory
sudo mkdir /usr/local/nginx/html/iurc
sudo mkdir /usr/local/nginx/html/iurc-admin
Copy the codeIf changed the/usr/local/nginx/conf/nginx. The content in the conf, can use the following command to restart the nginx, make the new configuration take effect.
sudo /usr/local/nginx/sbin/nginx -s reload
Copy the code3. Redis cluster installation and configuration (Sentinel mode)
Server Configuration
| The IP address | system | function |
|---|---|---|
| 10.210.111.14 | CentOS7.x | Redis, sentry |
| 10.210.111.15 | CentOS7.x | Redis, sentry |
| 10.210.111.16 | CentOS7.x | Redis, sentry |
3.1 installation Redis
Log in to server redis_0110.210.111.14 as robin user, download Redis and unzip it.
CD/home/robin wget https://download.redis.io/releases/redis-5.0.9.tar.gz tar - ZXVF redis - 5.0.9. Tar. Gz && CD redis - 5.0.9Copy the codeInstall the precompiled environment and install
sudo yum update
sudo yum -y install gcc
make
sudo make install PREFIX=/usr/local/redis
Copy the code⚠️ Install and configure Redis on the redis_0210.111.15 server by following Step 3.1
⚠️ Install and configure Redis on the redis_0310.210.111.16 server by following Step 3.1
3.2 configure Redis
In 10.210.111.14, copy redis.conf from the redis source directory to the redis installation directory
Sudo cp /home/robin/redis-5.0.9/redis.conf /usr/local/redis/bin/&& CD /usr/local/redis/bin/Copy the codeIn 10.210.111.14, modify the redis.conf file:
sudo vim redis.conf
Copy the codeChange daemonize no to daemonize yes
Comment out the default loopback address. By default, it cannot be accessed by other machines
To disable the protection mode, change the value from yes to no
Distribute the redis. Conf file modified in 10.210.111.14 to 10.210.111.15 and 10.210.111.16
SCP/usr/local/redis/bin/redis. Conf [email protected]: / usr/local/redis/bin /#The root password of 10.210.111.15 is requiredSCP/usr/local/redis/bin/redis. Conf [email protected]: / usr/local/redis/bin /#The root password of 10.210.111.16 is required
Copy the codeIn 10.210.111.15, edit redis.conf
sudo vim /usr/local/redis/bin/redis.conf
#Add the following on the last lineReplicaof 10.210.111.14 6379Copy the codeIn 10.210.111.16, edit redis.conf
sudo vim /usr/local/redis/bin/redis.conf
#Add the following on the last lineReplicaof 10.210.111.14 6379Copy the code3.3 Configuring Redis Sentinel
In 10.210.111.14, install redis-sentinel
sudo mkdir -p /usr/local/redis-sentinel
sudo cp -r /usr/local/redis/* /usr/local/redis-sentinel
Copy the codeIn 10.210.111.14, copy sentinel.conf
Sudo cp/home/robin/redis - 5.0.9 / sentinel. Conf/usr/local/redis - sentinel/binCopy the codeIn 10.210.111.14, modify sentinel.conf
sudo vim /usr/local/redis-sentinel/sentinel.conf
#Change daemonize from no to yes
daemonize yes
#Specifies the number of milliseconds after the primary node does not reply to sentinel. At this point, sentinel subjectively considers the primary node to be offline. The default value is 30000 milliseconds, which is changed to 3000 milliseconds
sentinel down-after-milliseconds mymaster 3000
Copy the codeCopy sentinel 10.210.111.14 to 10.210.111.15 and 10.210.111.16
SCP -r /usr/local/redis-sentinel/ [email protected]:/usr/local scp-r /usr/local/redis-sentinel/ [email protected]: / usr/localCopy the codeIn 10.210.111.15, modify the redis.conf file
vim /usr/local/redis-sentinel/bin/sentinel.conf
#Change the IP address of the primary Redis node monitored by Sentinel to 10.210.111.14Sentinel Monitor MyMaster 10.210.111.14 6379 2Copy the codeIn 10.210.111.16, modify the redis.conf file
sudo vim /usr/local/redis-sentinel/bin/sentinel.conf
#Change the IP address of the primary Redis node monitored by Sentinel to 10.210.111.14Sentinel Monitor MyMaster 10.210.111.14 6379 2Copy the code3.4 Enabling Redis and Redis Sentinel upon Startup
In 10.210.111.14, add the Redis startup service
sudo vim /etc/systemd/system/redis.service
#Copy and paste the following
[Unit]
Description=redis-server
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/bin/redis.conf
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Copy the codeIn 10.210.111.14, added the Redis Sentinel startup service
sudo vim /etc/systemd/system/redis-sentinel.service
[Unit]
Description=redis-sentinel
After=network.target
#Copy and paste the following
[Service]
Type=forking
ExecStart=/usr/local/redis-sentinel/bin/redis-sentinel /usr/local/redis-sentinel/bin/sentinel.conf
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Copy the codeThe configuration is explained as follows:
#[Unit]: indicates the service Unit
#Description: Describes the service
#After: Describes the service category
#[Serivce]: indicates the setting of service running parameters
#Type=forking: is the form of background running
#ExecStart: specifies the running command of the service
#ExecReload: restart command of the service
#ExecStop: Stop the service
#PrivateTmp=True: Independent temporary space is allocated to the service
#⚠️ Note: The start, restart, and stop commands of [Service] require absolute paths
#[Install]: sets the operating level for service installation. You can set it to multiple users, that is, set the operating level to 3
Copy the code⚠️ repeat the above steps in 10.210.111.15 and 10.210.111.16 to add Redis Startup service and Redis Sentinel startup service.
3.5 Starting Redis and Redis Sentinel
In 10.210.111.14, start Redis and add boot autoboot
sudo systemctl daemon-reload
sudo systemctl start redis.service
sudo systemctl enable redis.service
Copy the codeIn 10.210.111.14, start Redis Sentinel and add boot autostart
sudo systemctl start redis-sentinel.service
sudo systemctl enable redis-sentinel.service
Copy the code⚠️ repeat the above steps in 10.210.111.15 and 10.210.111.16 to start Redis and add startup startup and start Redis Sentinel and add startup startup steps.
3.6 test
Go to the Redis console on main Redis10.210.111.14
sudo /usr/local/redis/bin/redis-cli
Copy the codeType the command
info
Copy the codeIf the following results appear, the two slaves are connected, it means that the installation of Redis master-slave (Sentinel mode) is successful
connected_slavesIt doesn’t have to be 2 in real time. You can check it several times.
4. MySQL installation and configuration (master/slave mode)
Server Configuration
| The IP address | system | function |
|---|---|---|
| 10.210.111.17 | CentOS7.x | The master database |
| 10.210.111.18 | CentOS7.x | From the database |
Install MySQL 4.1
Log in to server db_0110.210.111.17 as robin user and install MySQL through YUM source.
The following links are detailed installation steps on the official website, readers can follow the steps in the installation of MySQL
Dev.mysql.com/doc/refman/…
4.2 Setting a Remote connection for MySQL
Set MySQL to only two APP SERVER10.210.111.19 and 10.210.111.20
#Enter the following command on the MYSQL command line
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.210.111.19' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.210.111.20' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.210.111.14' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.210.111.15' IDENTIFIED BY 'P@ssw0rd' WITH GRANT OPTION;
FLUSH PRIVILEGES;
Copy the code⚠️ MySQL 10.210.111.17 has been installed. Please log in to 10.210.111.18 and refer to steps [4.1-4.2] to install MySQL on 10.210.111.18.
4.3 Setting primary/secondary Synchronization
⚠️ set 10.210.111.17 to the primary database and 10.210.111.18 to the secondary database
mysql -uroot -p
#Enter the password
CREATE USER 'slave'@'%' IDENTIFIED BY 'slaveP@ssw0rd';
GRANT ALL PRIVILEGES ON *.* TO 'slave'@"%" IDENTIFIED BY "slaveP@ssw0rd";
Copy the codeModify the my.cnf configuration file in 10.210.111.17
sudo vim /etc/my.cnf
#Add two lines
log-bin=mysql-bin
server-id=1
Copy the code
instructions
#Enable binary logging, which is used to write log files at transaction commit time. The default size is 1 gb, followed by 001,002.
log-bin=mysql-bin
#Each instance of mysql is configured differently. This value defaults to 0, and if it is 0, the master server rejects any connections from the slave server.
server-id=1
#Other configurations (not required) :
#Specify which DATABASE is recorded in the mysql binlog. Configure multiple databases to be synchronized. If this parameter is not set, synchronize all databases.Binlog-do-db =db_001 (primary database configuration)
#You can configure multiple databases that are not synchronized.Binlog-ignore-db =mysql (primary database configuration)
#Set the binlog format
binlog_format=mixed
#Read-only 0 indicates no read-only, and 1 indicates read-only
read-only=0
#Used to set the ID conflict of the auto-increment column in the case of dual master, mainly used to set the auto-increment step
auto-increament-increment=10
#Indicates the serial number of the server. The value starts from 1 and does not exceed auto-increament-increment
auto-increment-offset=1
Copy the codeRestart MySQL at 10.210.111.17
sudo systemctl restart mysqld
Copy the codeModify the my.cnf configuration file in 10.210.111.18
sudo vim /etc/my.cnf
#Add two lines
server-id=2
relay_log=mysql-relay-bin
Copy the codeRestart MySQL at 10.210.111.18
systemctl restart mysqld.service
Copy the codeObtain the binlog information of MySQL 10.210.111.17
mysql -uroot -p
#Enter the password
show master status;
Copy the code
Enter MySQL Settings in 10.210.111.18 to synchronize with the primary database
mysql -uroot -p
#Enter the passwordchange master to Master_host = '10.210.111.17, master_user =' slave ', master_password = 'slaveP @ ssw0rd, master_log_file =' 000001 'mysql - bin. master_log_pos=154;Copy the code⚠️ Pay attention to the following instructions
Master_user and master_password are the user names slave and password created on the primary server. Master_log_file is the value of the File field in MySQL Binlog Information queried on the primary server. For example, mysql-bin.000001 master_log_pos is the Position field in mysql Binlog Information queried on the primary server. Like 154 in the figure aboveCopy the codeStart the MySQL slave thread 10.210.111.18
start slave;
Copy the codeCheck the slave status of MySQL 10.210.111.18
show slave status \G
Copy the code
4.4 Primary/secondary Verification
Perform database creation and table creation on 10.210.111.17 active MySQL to verify that the primary/secondary mode is successfully deployed
Mysql -uroot -p # create database test; use test; Create table user_test(id int comment' id ',name VARCHAR(20) comment' id ', Create_time TIMESTAMP DEFAULT now() comment' create time '); insert INTO user_test value(1,"A",NOW());Copy the codeCheck from MySQL on 10.210.111.18 to see if there are tables and data created in primary MySQL
Mysql -uroot -p database; use test; show tables; select * from user_test;Copy the codeThe result shows that the data has been synchronized from the primary MySQL to the secondary MySQL, indicating that the primary/secondary mode has been successfully deployed. Delete test data from 10.210.111.17 primary MySQL.
drop database test;
Copy the code5. Install static resources synchronously
This part of the content is for reference only, because now look at the design actually has some problems. Nginx allows static files to be accessed directly through Nginx and backup files to be accessed through the Static resource Synchronization service. This is fine for public static resources, but if static resources uploaded by users are not public, there is a great risk that an attacker could download users’ private information directly through Nginx.
To sum up, two suggestions can be made:
- For private files, OSS services can be used if necessary. If you can unconditionally refer to the following static resource synchronization service, I think there should be a better choice in addition to my following service, readers can find it by themselves;
- Static resources that can be exposed can be obtained directly through Nginx. For private files, remember not to configure Nginx directly.
Server Configuration
| The IP address | system | function |
|---|---|---|
| 10.210.111.13 | CentOS7.x | Rsync server |
| 10.210.111.12 | CentOS7.x | Rsync client |
5.1 Rsync Server
Install rsync at 10.210.111.13
cd /opt
sudo yum install -y rsync
Copy the codeConfigure the/etc/rsyncd. Conf
#Record file transfer logs
transfer logging = yes
#Specifying log files
log file = /var/log/rsyncd.log
#Pid file path
pid file = /var/run/rsyncd.pid
#Lock the path
lock file = /var/run/rsync.lock
#The user running the rsync daemon
uid = robin
#The group that runs the rsync daemon
gid = robin
#Whether to allow chroot to improve security. If the client is connected to the module, chroot must go to the directory specified in the module path parameter. If the value of chroot is yes, the root permission must be used and connection files outside the path cannot be backed up
use chroot = yes
#Ignore the error
ignore errors
#read-only
read only = no
#Set the timeout period, in seconds
timeout = 60
ignore nonreadable = yes
#Module, can configure more than one
[sync_file]
#Module root directory, synchronization directory, need to pay attention to permissions
path = /home/robin/static
#User name for module authentication. Multiple user names can be separated by Spaces or commas
auth users = sync
#Module authentication password file, which can be placed in global configuration
secrets file = /etc/rsyncd.secrets
#Set a white list, to specify the IP segment (172.18.50.1/255.255.255.0), the IP segment separated by SpacesHosts allow = 10.210.111.12 hosts deny = *#Whether module content is allowed to be listed
list = no
Copy the codeCreate /etc/rsyncd.secrets where robin is the user using the synchronization service
#Perform this operation as the root roleecho "robin:infoP@ssw0rd" > /etc/rsyncd.secrets
sudo -i
#Enter the password as root
echo "sync:123456" > /etc/rsyncd.secrets
#Switch back to normal user
su robin
Copy the codeSetting file Permissions
sudo chmod 600 /etc/rsyncd.secrets
Copy the codeStart the service
sudo systemctl restart rsyncd
Copy the code5.2 Rsync Client
Install rsync at 10.210.111.12
cd /opt
sudo yum install -y rsync
Copy the codeCreate /etc/rsync.passwd and run the following command:
#Perform this operation as the root roleecho "infoP@ssw0rd" > /etc/rsync.passwd
sudo -i
#Enter the password as root
echo "123456" > /etc/rsync.passwd
#Switch back to normal user
su robin
Copy the code⚠️ The rsync client requires only the password, which is the password of user Robin
Change the permissions
sudo chmod 600 /etc/rsync.passwd
Copy the codeThe installation way
The inotify-tools tool detects file additions, deletions, and modifications.
sudo yum install -y inotify-tools
Copy the codeWrite the startup script inotify_start.sh
cd /home/robin
Copy the codeThe script content is as follows:
#! /bin/bash
Path=/home/robin/static
#Set host to the IP address of the rsync serverServer=10.210.111.13 User=sync module=sync_file /usr/bin/inotifywait-mrq --format '%w%f' -e create,close_write,delete $Path | while read line do if [ -f $line ]; then rsync -az $line --delete ${User}@${Server}::${module} --password-file=/etc/rsync.passwd else cd $Path &&\ rsync -az ./ --delete ${User}@${Server}::${module} --password-file=/etc/rsync.passwd fi doneCopy the codeThe test command
Sudo rsync-avz --delete /home/robin/static/ [email protected]::sync_file --password-file=/etc/rsync.passwdCopy the code5.3 Writing a Service Startup Script
Write inotifyd.service and copy it to /usr/lib/systemd/system/ as follows
[Unit]
Description=inotify rsync script
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=root
Group=root
PIDFile=/var/run/inotify.pid
ExecStart=/usr/bin/nohup /home/robin/inotify_start.sh > /home/robin/inotify.log 2>&1 &
ExecReload=
ExecStop=/usr/bin/ps -ef | /usr/bin/grep inotify | /usr/bin/grep -v grep | /usr/bin/awk '{print $2}' | /usr/bin/xargs /usr/bin/kill -9
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Copy the codeStart the service
sudo systemctl start inotifyd.service
Copy the codeAllow auto start
sudo systemctl enable inotifyd.service
Copy the codeRe-read the restart script
sudo systemctl daemon-reload
Copy the code