A swordsman

grep

Syntax format:

• The first form:grep [option] [pattern] [file1,file2...]
• The second form:command | grep [option] [pattern]

The grep parameters:

The sample

#Prepare file with the following contents
Python
Java
Python is good
Java is perfect
Php is the best lanuage

#Filter lines containing Python
grep Python file
->Python
->Python is good
grep -vi python file
->Java
->Java is perfect
->Php is the best lanuage

#According to the line Numbers
grep -n Python file
->1:Python
->3:Python is good

#Support for extended regular expressions (you can also use egrep instead)
grep 'python|Python' file
->
grep -E 'python|Python' file
->Python
->Python is good

Copy the code

Swordsman ii

sed

Sed is a Stream Editor. The stream editor is used to perform basic text conversion on an input stream (file or input from a pipe). Although sed is somewhat similar to an editor that allows rough editing (for example, Ed), it works by passing the input only once and is therefore more efficient. Sed’s ability to filter text in a pipe makes it particularly different from other types of editors. (Stream in JDK8 should be familiar to Java programmers.)

Syntax format:

Sed options

The sample

#Match the Python
#You'll see that the lines that don't match are also printed, and the lines that do match are printed twice
#By default sed prints the original line (p is for print)command)
sed '/Python/p' file
->Python
->Python
->Java
->Python is good
->Python is good
->Java is perfect
->Php is the best lanuage

#Print only the matching line, not the original line
sed -n '/Python/p' file
->Python
->Python is good

#More than one pattern
sed -n -e '/Python/p' -e '/Java/p' file
->Python
->Java
->Python is good
->Java is perfect

#Extended regular expressions
#Maybe you will put the above two the pattern together: / Python | Java/p. Unfortunately, sed does not support extended regular expressions by default, requiring -r
sed -n -r '/Python|Java/p' file
->Python
->Java
->Python is good
->Java is perfect

# 'batch'To deal with
#If we have a lot of actions to perform, we can write them in the file directly. Use -f to import the action file and create multsed.sed(not required. Sed).
/Python/p
/Java/p

sed -n -f 'multsed.sed' file
->Python
->Java
->Python is good
->Java is perfect
Copy the code

The pattern of sed

The sample

#Prints the contents of the specified line number
sed -n '2p' file
->Java

#Print the information between the start line number and the end line number.Sed - n '1, 3 p' file->Python
->Java
->Python is good

#Specify the starting line number and print the next n lines
sed -n '1,+2p' file
->Python
->Java
->Python is good

#Print the matching line
sed -n '/Python/p' file
->Python
->Python is good

#Prints the line between two matches
sed -n '/Python/,/good/p' file
->Python
->Java
->Python is good

#Starts at the specified start line and ends at the matching line
sed -n '2,/good/p' file
->Java
->Python is good

#From the start of the matching line to the end of the specified line (if the matching line is smaller than the specified end line, only the matching line is displayed)
sed -n '/good/,4p' file
->Python is good
->Java is perfect

sed -n '/good/,2p' file
->Python is good

Copy the code

Sed command

The sample

#Append after line 1 (use the -i option if you need to modify the file)
sed '1aScala' file
->Python
->Scala
->Java
->Python is good
->Java is perfect
->Php is the best lanuage

#Prefixes the matched line with the content
sed '/Python/iHello Python' file
->Hello Python
->Python
->Java
->Hello Python
->Python is good
->Java is perfect
->Php is the best lanuage

#Read in external files (appending data after each line without writing matching patterns)
sed 'rmultsed.sed' file
->Python
->/Python/p
->/Java/p
->Java
->/Python/p
->/Java/p
->Python is good
->/Python/p
->/Java/p
->Java is perfect
->/Python/p
->/Java/p
->Php is the best lanuage
->/Python/p
->/Java/p

#Writes the matched line information to an external file
sed '/Python/wpython.txt' file

#Replace the first match in the row
sed -n 's/Python/Java/p' file
->Java Python python
->Java is good Python is good

#Replace all matches in the row
sed -n 's/Python/Java/gp' file
->Java Java python
->Java is good Java is good

#Replaces all matches in the row (ignoring case)
sed -n 's/Python/Java/igp' file
->Java Java Java
->Java is good Java is good

#Replace all old after the third match (ignore case)
sed -n 's/Python/Java/3igp' file
->Python Python Java

#Displays the matching line number
sed -n '/Python/=' file
->1
->3

#If pattern uses the value of a variable, double quotation marks must be used unless you also put single quotation marks around the variable
var1=Python
sed -n '/$var1/=' file
->
sed -n "/$var1/ =" file
->1
->3

sed -n '/'$var1'/=' file
->1
->3

Copy the code

If pattern uses the value of a variable, double quotation marks must be used unless you also put single quotation marks around the variable.

backreferences

The act of referencing the entire string that pattern matches in sed is called backreferencing, as illustrated in the following example.

#Requirements: Modify file in Python->Pythoner,Java->Javaer,Php->Phper
#If we didn't know about the backreference we would use the modify command multiple times to modify it
#It's easy to just look up these words
sed -n -r '/Python|Java|Php/p' file
#There is one difficulty if substitution is required: we cannot give a unified new character belowSed - n - r 's new character er/Python | | Php/Java/p' file
#By back reference
sed -n -r 's/Python|Java|Php/&er/p' file
->Pythoner Python python
->Javaer java
->Pythoner is good Python is good
->Javaer is perfect
->Phper is the best lanuage

#In addition to &, you can use \1,\2,\3, etc. (remember the parentheses), which can extract a single group
sed -n -r 's/(Python|Java|Php)/\1er/p' file
->Pythoner Python python
->Javaer java
->Pythoner is good Python is good
->Javaer is perfect
->Phper is the best lanuage

echo -e "Python Java Php" | sed -e "s/\(Python\) \(Java\) \(Php\)/\1er \2s \3best/g"
->Pythoner Javas Phpbest
Copy the code

A profound

Requirement description: Process a text similar to MySQL configuration file my.cnf, as shown in the following example, write a script to achieve the following functions: Output file has several sections, and for each section can count the total number of parameters.

The contents of the my.cnf file are as follows:

# this is read by the standalone daemon and embedded servers [client] port=3306 socket=/tmp/mysql.socket #ThisSegmentForserver [server] innodb_buffer_pool_size=91750M innodb_buffer_pool_instances=8 innodb_buffer_pool_load_at_startup=1 innodb_buffer_pool_dump_at_shutdown=1 innodb_data_file_path=ibdata1:1G:autoextend innodb_flush_log_at_trx_commit=1 innodb_log_buffer_size=32M innodb_log_file_size=2G innodb_log_files_in_group=2 innodb_max_undo_log_size=4G innodb_undo_directory=undolog innodb_undo_tablespaces=95 #thisisonlyforthemysqldstandalonedaemon [mysqld] port=3306 socket=/tmp/mysql.sock basedir=/usr/local/mysql Datadir =/data/mysql pid-file=/data/mysql/mysql.pid user=mysql bind-address=0.0.0.0 sort_buffer_size=16M join_buffer_size=16M thread_cache_size=3000 interactive_timeout=600 wait_timeout=600 #ThisSegmentFormysqld_safe [mysqld_safe] log-error=/var/log/mariadb/mariadb.log pid-file=/var/run/mariadb/mariadb.pid max_connections=1000 open_files_limit=65535 thread_stack=512K external-locking=FALSE max_allowed_packet=32M #thisisonlyforembeddedserver [embedded] gtid_mode=on enforce_gtid_consistency=1 log_slave_updates slave-rows-search-algorithms='INDEX_SCAN,HASH_SCAN'  binlog_format=row binlog_checksum=1 relay_log_recovery=1 relay-log-purge=1 # usethisgroupforoptionsthatolderserversdon 'tunderstand [mysqld - 5.5] key_buffer_size read_buffer_size = 32 m = 8 m read_rnd_buffer_size=16M bulk_insert_buffer_size=64M myisam_sort_buffer_size=128M myisam_max_sort_file_size=10G myisam_repair_threads=1 lock_wait_timeout=3600 explicit_defaults_for_timestamp=1 innodb_file_per_table=1Copy the code

#! /bin/bash

FILE_NAME="my.cnf"


function get_all_segment
{
	echo "`sed -n '/\[.*\]/p' $FILE_NAME | sed -r 's/(\[|\])//g'`"
}

#Count the number of configuration items in each segment
function count_items_in_segment
{
	#Section namesegname=$1 itemcount=`sed -n "/\[$segname\]/,/\[.*\]/p" $FILE_NAME | grep -v ^# | grep -v ^$ | grep -v "\[.*\]" | grep -c "" 'echo $itemcount} sum=1 for segname in' get_all_segment 'do echo" $segname `count_items_in_segment $segname`" sum=`expr $sum + 1` doneCopy the code

Swordsman three

awk

AWK is a language for processing text files and is a powerful text analysis tool. AWK is called AWK because it takes the first character of the Family Name of Alfred Aho, Peter Weinberger, and Brian Kernighan.

Syntax format:

Syntax:

Awk built-in variables

The sample

All the following sample files are /etc/passwd. Please make a copy of them for use

#Print the entire line
awk '{print $0}' passwd

#use":"The number is used as the delimiter to print the first field
awk 'BEGIN{FS=":"}{print $1}' passwd
->root
->bin
->daemon
->adm
->lp
->sync
->shutdown
->.

#Output line number
awk '{print NR}' passwd

#Multiple file line numbers are counted separately
awk '{print FNR}' passwd file2

#Specifies the line separator
echo "hello-world-hello-linux-hello-java" | awk 'BEGIN{RS="-"}{print $0}'
->hello
->world
->hello
->linux
->hello
->java
->

#Specifies the output field separatorawk 'BEGIN{FS=":"; OFS=":"}{print NR,$1}' passwd->1:root
->2:bin
->3:daemon
->4:adm
->5:lp
->.

Copy the code

Printf formats the output

The sample

#Print the first field as a string with":"As a separator
awk 'BEGIN{FS=":"}{printf "%s\t",$1}' passwd
->root	bin	daemon	adm	lp ...

#Print the first field and the corresponding line number as a string"Line number: Field contents"
awk 'BEGIN{FS=":"}{printf "%d:%s\n",NR,$1}' passwd
->1:root
->2:bin
->3:daemon
->4:adm
->5:lp
->.

#The left
#Default left alignment if no digit is specified, right alignment if digit is specified (digit must be specified)
awk 'BEGIN{FS=":"}{printf "%10d:%s\n",NR,$1}' passwd
->         1:root
->         2:bin
->         3:daemon
->         4:adm
->         5:lp
->.

awk 'BEGIN{FS=":"}{printf "%-10d:%s\n",NR,$1}' passwd
->1 :root
->2 :bin
->3 :daemon
->4 :adm
->5 :lp
->.
Copy the code

Two matching patterns

Relational operators:

• < : less than
• >.
• <= : Less than or equal to
• >= : The value is greater than or equal to
• = = : equal to zero
• ! = : Does not equal
• ~ : Matches the regular expression
• ! ~ : Does not match the regular expression

Boolean operators:

• | | : or
• && : with
• ! :

#Matches all lines in the file line that contain the root string
awk 'BEGIN{FS=":"}/root/{print $0}' passwd
->root:x:0:0:root:/root:/bin/bash
->operator:x:11:0:operator:/root:/sbin/nologin

#Matches all rows whose third field is less than 50
awk 'BEGIN{FS=":"}$3<50{print $0}' passwd

#The matching file contains all lines of information for mail or FTP
awk 'BEGIN{FS=":"}/mail/||/ftp/{print $0}' passwd
->mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
->ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

#Matches all lines in the file whose third field is less than 50 and whose fourth field is greater than 50
awk 'BEGIN{FS=":"}$3<50 && $4 > 50{print $0}' passwd
->games:x:12:100:games:/usr/games:/sbin/nologin

#The third field in the match file is less than 50 and the seventh field matches all line information for /bin/bash
awk 'BEGIN{FS=":"}$3<50 && $7 ~ /\/bin\/bash/{print $0}' passwd
->root:x:0:0:root:/root:/bin/bash

Copy the code

Awk expression

#Count blank rows
awk '/^$/{sum++}END{print sum}' /etc/services
->17

#Statistic student achievement total score peace average cent, report form is displayedName Chinese Math English physics Zhang SAN 80 60 85 90 Li Si 85 65 80 75 Wang Wu 70 60 85 90 Li Hua 65 80 84 91 Wang Ba 90 90 95 90 awk 'BEGIN{printf "% % - 8 - s - 5 s % % % - 5 s - 5 s to 5 s % % - 5 s to 8 s \ n", "name", "Chinese", "mathematics", "English", "physical", "total", "average"} {total = $2 + $3 + $4 + $5. avg=total/4; Printf "%-8s%-8d%-6d%-8d%-7d%-5d%0.2f\n",$1,$2,$3,$4,$5,total,avg}' stu. TXT 78.75 Li Si 85 65 80 75 305 76.25 Wang Wu 70 60 85 90 305 76.25 Li Hua 65 80 84 91 320 80.00 Wang Ba 90 90 95 90 365 91.25Copy the code

Conditional statements and loop statements

#Conditional statementsIf (conditional expression 1){action1}else if (conditional expression 2){action2}else{action3}
# 以":"Is a delimiter and prints less than 50 if the third field is less than 50, equals 50 if it is equal to 50, and greater than 50 otherwiseAwk 'BEGIN {FS = ":"} {the if ($3 < 50) {print "less than 50"} else if ($3 = = 50) {print "is equal to 50"} else {print "more than 50"}}' passwd
#We can save commands in a file (eg:oper.awk, suffix is not mandatory!). To use the -f option
awk -f oper.awk passwd

#Looping statementswhileWhile (conditional expression){action}
#Looping statementsdo whileDo {action}while(conditional expression)
#Looping statementsforFor (initialize the counter; Test counter; Change counter){action}
#Calculation of 1 + 2 + 3 +... And + 100.awk 'BEGIN{do{i++; sum+=i; }while(i<100)print sum}' awk 'BEGIN{while(i<100){i++; sum+=i; }print sum}' awk 'BEGIN{for(i=0; i<=100; i++){sum+=i; }print sum}'Copy the code

String function

The sample

#Returns the length of each field in each line of the file, delimited by:awk 'BEGIN{FS=":"}{for(i=1; i<=NF; i++){if(i==NF){printf "%d",length($i)}else{printf "%d:",length($i)}}print ""}' passwd
#Search string"I am a student"The position of the student substring in
echo "I am a student" | awk '{print index($0,"student")}'
->8

Copy the code

Awk common options

#Use -v to reference external variables
num1=100
num2=200
awk -v var1="$num1" -v var2="$num2" 'BEGIN{print var1+var2}'

Copy the code

An array of

Awk arrays are enclosed in parentheses, with Spaces separating each item, for example :arr=(“one” “two” “three” “four” “five”). In awK, array subscripts start at 1. Awk can use data structures such as associative arrays, whose indexes can be numbers or strings. The AWK associative number group also does not need to be declared in advance because it can be automatically increased or decreased at run time.

#Collect statistics on all TCP packets on hosts and classify them by TCP status
netstat -an | grep tcp | awk '{array[$6]++}END{for(item in array){print item,array[item]}}'
->LISTEN 8
->ESTABLISHED 26
Copy the code