Distribution methods: SCP, rsync, xsync-plus

“This is the 29th day of my participation in the Gwen Challenge in November. See details of the event: The Last Gwen Challenge in 2021”

The premise

Create three VMS

Please refer to my previous article: Environment: Virtualbox+Vagrant Installing Centos7 – Digging gold (juejin. Cn)

Changing the host Name

vim /etc/hostname

Each VM is K8S1, K8S2, and K8S3:

Configuring the hosts file

Viewing the Default NIC

ip route show

Query the DEFAULT NIC IP address of K8S1 (also for other VMS).

The three VMS ping this IP address

vim /etc/hosts

If the vm can be pinged, add the following information to the hosts file of each VM:

10.0.2.9k8s1 10.0.2.8k8s2 10.0.2.15k8s3Copy the code

K8s1 Test connection k8S2

[root@k8s1 test]# ssh k8s2

If the SSH connection is enabled, the hosts configuration is successful.

SSH Encryption-free login

Since the server is not configured with a password-free login, the following script also needs to enter the account password when running, so you must configure a password-free login. Because of space. Please refer to my other article:

Linux tips: Secure Shell (SSH) login and configure environment variables – digging gold (juejin. Cn)

Scp-secure copy

define

Secure copy (SCP) Is used to copy data between servers (servers are not necessarily themselves).

The basic grammar

scp -r $pdir/$fname $user@$host:$pdir/$fname

SCP: command
– r: recursion
$pdir/$fname: name of the file to be copiedIt can be a directory or a file)
$user@$host:$pdir/$fname: destination user@ host: destination path/name

Case of actual combat

Prerequisites: Folders in k8s1, k8S2, and k8S3 have been created and have permissions

Sudo chown owner: group name -r /app/test

I copied it to him on my own initiative

On k8s1, copy /app/test in k8s1 to /test in k8s2: /app/test/test.

[root@k8s1 .ssh]# mkdir /app/test
[root@k8s1 .ssh]# cd /app/test
[root@k8s1 test]# touch scp-test.txt
[root@k8s1 test]$ scp -r /app/test root@k8s2:/app/test 
Copy the code

I asked him to copy it for me

On k8s1, copy /app/test/pull. TXT from k8s2 to K8s1.

K8s2: Create a pull. TXT file

[root@k8s2 test]# touch pull.txt
[root@k8s2 test]# ll
total 0
-rw-r--r--. 1 root root  0 Nov 25 13:27 pull.txt
drwxr-xr-x. 2 root root 26 Nov 25 13:23 test
Copy the code

K8s1: pull the K8s2 file to the local PC

[root@k8s1 test]# scp -r root@k8s2:/app/test/pull.txt /app/test/
root@k8s2's password: TXT 100% 0 0.0KB/s 00:00 [root@k8s1 test]# ll total 0 -rw-r--r-- 1 root root 0 Nov 25 13:30 pull. TXT -rw-r--r-- 1  root root 0 Nov 25 13:06 scp-test.txtCopy the code

I want him to copy it to another him

Operating on K8s1, copy all directories under /app/test in K8S2 to K8S3.

[root@k8s1 test]# scp -r root@k8s2:/app/test/pull.txt /app/test/
root@k8s2's password: TXT 100% 0 0.0KB/s 00:00 [root@k8s1 test]# ll total 0 -rw-r--r-- 1 root root 0 Nov 25 13:30 pull. TXT -rw-r--r-- 1  root root 0 Nov 25 13:06 scp-test.txt [root@k8s1 test]# scp -r root@k8s2:/app/test/* root@k8s3:/app/test root@k8s2's password: 
The authenticity of host 'k8s3 (192.168.56.103)' can't be established. ECDSA key fingerprint is SHA256:yk/B9iEdgbi8h2XVM4klHaKjmobewr1/hFGZq6CqNSA. ECDSA key fingerprint is  MD5:b3:15:fc:c6:79:4f:10:df:e9:87:01:ab:5b:ab:ba:9c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'K8s3, 192.168.56.103' (ECDSA) to the list of known hosts.
root@k8s3'TXT 100% 0 0.0KB/s 00:00 scp-test. TXT 100% 0 0.0KB/s 00:00 Connection to K8S2 closed.Copy the code

Rsync – synchronous

Rsync is mainly used for backup and mirroring. It has the advantages of high speed, avoiding copying the same content and supporting symbolic links. Yum install -y rsync yum install -y rsync

Rsync and SCP differences:

Using rsync to copy files is faster than using SCPDifferences in the filedoupdate.
SCP is to copy all the files.

The basic grammar

rsync -av $pdir/$fname $user@$host:$pdir/$fname

Rsync: command
-av: option parameter
- -a: Archive copy
- -v: indicates that replication is performed
$pdir/$fname: path/name of the file to be copied
$user@$host:$pdir/$fname: destination user@ host: destination path/name

Case of actual combat

Synchronize data from /app/test/ in K8S1 to /app/test/ in K8S2

[root@k8s1 test]# rsync -av /app/test/ root@k8s2:/app/test/
root@k8s2's password: Sending incremental file list./ pull. TXT scp-test. TXT sent 178 bytes received 57 bytes 94.00 bytes/ SEC Total size is 0 The speedup is 0.00Copy the code

Xsync – plus – distribution

demand

Distribute multiple directories or files to all machines by default
- Command design:xsync-plus file1 file2 ...
- Command design:xsync-plus -d file1 file2 ...
Distribute a directory or file to the machine whose host name is k8sy increment y=[n,m] m>n, where n is an integer
- Command design:xsync-plus -f file server_pref start end
- Server_pref: prefix of the host name
- Start: indicates the start sequence number
- “End” : indicates the end sequence number
Distribute a directory or file to an enumerated machine
- Command design:xsync-plus -e file server1 server2 ...

Script implementation

#! /bin/bash

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # method area end # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

####################### The default operation is ######################
## ljw
Distribute multiple files to all machines by default
Xsync-plus file1 file2...
Xsync-plus -d file1 file2...
####################### The default operation is ######################

function defaultSend() {
for host in ${cluster[@]}
do
    #3. Traverse all directories and send them one by one
    for file in $@
    do
        #4. Determine if the file exists
        if [ $file! ="-d" ] && [ -e $file ]
            then
                $(dirname $file); $(dirname $file); Another command. PWD gets the parent directory
                pdir=$(cd -P $(dirname $file); pwd)

                #6. Obtain the name of the current file
                fname=$(basename $file)
                ssh $host "mkdir -p $pdir"
                echo run ---------- rsync -av $pdir/$fname $host:$pdir --------------------
                rsync -av $pdir/$fname $host:$pdir
            else
                if [ $file! ="-d" ]
                then
                        echo $fileFile or folder does not exist!exit;
                fi
        fi
    done
done
}

# # # # # # # # # # # # # # # # # # # # # # # - e operation # # # # # # # # # # # # # # # # # # # # # #
## ljw
## Distribute a file to the enumeration machine
Xsync-plus -e file server1 server2...
# # # # # # # # # # # # # # # # # # # # # # # - e operation # # # # # # # # # # # # # # # # # # # # # #
function eSend() {

#2. Determine the second parameter
if [ $action= ="-e" ] 
then 
        # # 3 parameters
        if [ $paramcount -lt 3 ]
        then
                echoInsufficient parameters!echo $0 -e file_name server1  server2 ...
                exit;
        else
                Get the file name
                file=$2
                
                #4. Determine if the file exists
        if [ -e $file ] 
                then 
                        Get the parent directory
                        pdir=$(cd -P $(dirname $file); pwd) 
                        Get the name of the current file
                        fname=$(basename $file) 
                                                
                        # Start number
                        start=3
                        # End number
                        end=$paramcount

                        # cycle
                        for((num=$start; num<=$end; num++)); do
				## assign intermediate variables first, this place is too bad
                                ser=$num
				Get $n n parameters for the input parameter.
				server=`eval echo '$'"$ser"`
				ssh $server "mkdir -p $pdir"
                                echo run ---------- rsync -av $pdir/$fname $server:$pdir ----------
                                rsync -av $pdir/$fname $server:$pdir
                        done
                        
                else 
                        echo $fileFile or folder does not exist!fi 
        
        fi 
fi

}


# # # # # # # # # # # # # # # # # # # # # # # - f operation # # # # # # # # # # # # # # # # # # # # # #
## ljw
Y =[n,m] m>n, where n is an integer
Xsync-plus -f file server_pref start end xsync-plus -f file server_pref start end
###server_pref: host name prefix
###start
###end
# # # # # # # # # # # # # # # # # # # # # # # - f operation # # # # # # # # # # # # # # # # # # # # # #
function fSend() {
if [ $action= ="-f" ] 
then 
        ## does not have 5 parameters
        if [ $paramcount -ne 5 ]
        then
				echoInsufficient parameters!echo $0 -f file server_pref start  end
                exit;
        else
                Get the file name
                file=$2
                #4. Determine if the file exists
				if [ -e $file ] 
				then 
						Get the parent directory
						pdir=$(cd -P $(dirname $file); pwd) 
						Get the name of the current file
						fname=$(basename $file) 
						Get hostname and start/stop number
						server_pref=$3
						# Start number
						start=$4
						# End number
						end=A $5
						
						# cycle
						for((host=$start; host<=$end; host++)); do
						        ssh $server_pref$host "mkdir -p $pdir"
								echo run ---------- rsync -av $pdir/$fname $server_pref$host:$pdir ----------								
								rsync -av $pdir/$fname $server_pref$host:$pdir
								done
						
				else 
						echo $fileFile or folder does not exist!fi 
        
        fi
        
fi
}

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # method area end # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #


####################### Global parameters ######################
#1. Determine the number of arguments after the command
paramcount=$#
params=$@

if [ $paramcount -lt 1 ] 
then 
    echoInsufficient parameters!exit; 
fi

Parameter action
action=The $1
## My machine
cluster=('k8s1' 'k8s2' 'k8s3')


# # # # # # # # # # # # # # # # # # # # # # # - the default action # # # # # # # # # # # # # # # # # # # # # #
#2. Determine the second parameter
if [ $action= ="-d" ] 
then 
    echo $0 -d file1 file2 ...
         defaultSend $@;
elif [ $action= ="-f" ] 
then
	 echo $0 -f file server_pref start  end
         fSend $@;
elif [ $action= ="-e" ] 
then
	 echo $0 -e file server1  server2 ...
         eSend $@ $params;
else	
    echo default:$0 file1 file2 ...
         defaultSend $@;
fi
    echoTransfer complete!!Copy the code

test

Default distribution /app/test/ folder data toallThe server

Xsync-plus /app/test/ or xsync-plus -d /app/test/

Distribute /app/test/ folder data to host nameincreasingServer: K8S2 K8S3

xsync-plus -f /app/test/ k8s 2 3

Distribute /app/test/ folder data toThe enumerationThe host name

xsync-plus -e /app/test/ k8s2 k8s3

The problem

Script copied from Win to Linux format problem

Check whether the script file is in DOS or Unix format. The end of the line of the DOS file is ^M$, and the end of the line of the Unix file is a dollar sign

cat -A filename

If the dos2UNIX command is not available, install it

sudo yum install dos2unix

Convert a DOS file to a Unix file

dos2unix filename

Note: If a large piece of code is copied to Linux and there is an “I” at the beginning of a line, the text will be truncated and the system will assume that it is in “I” input mode.

Gap-user group-related commands

groups: Displays the group members of the current login user
groups test: Displays the group to which the test user belongs and its members
whoami: Displays the current login user name
cat /etc/group: Displays information about all groups
The cat/etc/group | grep group name: Searches for a user group
cat /etc/passwd: Displays information about all users
The cat/etc/passwd | grep user name: Searches for a user

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

All on the cluster, you are still uploading files manually! Distribution methods: SCP, rsync, xsync-plus

Distribution methods: SCP, rsync, xsync-plus

The premise

Create three VMS

Changing the host Name

Configuring the hosts file

K8s1 Test connection k8S2

SSH Encryption-free login

Scp-secure copy

define

The basic grammar

Case of actual combat

I copied it to him on my own initiative

I asked him to copy it for me

I want him to copy it to another him

Rsync – synchronous

The basic grammar

Case of actual combat

Xsync – plus – distribution

demand

Script implementation

test

The problem

Script copied from Win to Linux format problem

Gap-user group-related commands

All on the cluster, you are still uploading files manually! Distribution methods: SCP, rsync, xsync-plus

Distribution methods: SCP, rsync, xsync-plus

The premise

Create three VMS

Changing the host Name

Configuring the hosts file

K8s1 Test connection k8S2

SSH Encryption-free login

Scp-secure copy

define

The basic grammar

Case of actual combat

I copied it to him on my own initiative

I asked him to copy it for me

I want him to copy it to another him

Rsync – synchronous

The basic grammar

Case of actual combat

Xsync – plus – distribution

demand

Script implementation

test

The problem

Script copied from Win to Linux format problem

Gap-user group-related commands

Related Posts

Brief introduction to three common uses of this keyword

【Soul Gateway Exploration 】 SpringCloud plug-in principle

Priority queues and heaps