Pipe related commands

The target

  • cut
  • sort
  • wc
  • uniq
  • tee
  • tr
  • split
  • awk
  • sed
  • grep
  • The preparatory work

    zhangsan 68 99 26
    lisi 98 66 96
    wangwu 38 33 86
    zhaoliu 78 44 36
    maq 88 22 66
    zhouba 98 44 46
    Copy the code
  • The above is the score sheet information

  • Using commas, the first column is your name, the second column is your Chinese score, the third column is your math score, and the fourth column is your English score

The preparatory work

vim 1.txt

111:aaa:bbb:ccc
222:ddd:eee:fff
333:ggg:hhh
444:iii
Copy the code

1 cut

1.1 the target

  • cutFrom command results according to conditionsextractCorresponding to the content

1.2 implementation

Step 1: extract the 5th character from the first 2 lines of the 1.txt file

The command meaning
Cut action file Intercepts content from the specified file
  • parameter
parameter English meaning
-c characters Select content by character
head -2 1.txt | cut -c 5
Copy the code

Step 2: intercept the first two lines in 1.txt file with “:” to separate the content of the first 1,2 paragraphs

parameter English meaning
-d 'separator' delimiter Specifying delimiters
-f n1,n2 fields After the segmentation shows the number of paragraphs of content, use.segmentation

Scope of control

The scope of meaning
n Only the NTH item is displayed
n- Display the NTH entry all the way to the end of the line
n-m Display entries from NTH to MTH (including m)
The head - 2 1. TXT | the cut - d ':' - 1, 2 fCopy the code
head -2 1.txt | cut -d ':' -f 1-2
Copy the code

1.3 summary

  • throughCut Action target fileCorresponding content can be extracted according to conditions
  • The preparatory work

    vim score.txt

    zhangsan 68 99 26
    lisi 98 66 96
    wangwu 38 33 86
    zhaoliu 78 44 36
    maq 88 22 66
    zhouba 98 44 46
    Copy the code

2 sort

2.1 the target

  • Sort sorts the content of a text file in behavioral units.

2.2 the path

  • Step 1: Sort the string
  • Step 2: Reorder
  • Step 3: Sort by value
  • Step 4: Rank your grades

2.3 implementation

Step 1: Sort the string

[root@node01 tmp]# cat 2.txt
banana
apple
pear
orange
pear

[root@node01 tmp]# sort 2.txt 
apple
banana
orange
pear
pear
Copy the code

Step 2: Reorder

parameter English meaning
-u unique unrepetitive

It simply removes duplicate lines from the output line.

[root@node01 tmp]# sort -u 2.txt 
apple
banana
orange
pear
Copy the code

Step 3: Sort by value

parameter English meaning
-n numeric-sort Sort by numerical size
-r reverse Reverse the order
  • To prepare data

    [root@node01 tmp]# cat 3.txt 
    1
    3
    5
    7
    11
    2
    4
    6
    10
    8
    9
    Copy the code
  • Sort by string by default

    [root@node01 tmp]# sort 2.txt 
    1
    10
    11
    2
    3
    4
    5
    6
    7
    8
    9
    Copy the code
  • ascending

    [root@node01 tmp]# sort -n 2.txt
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    Copy the code
  • Reverse order

    [root@node01 tmp]# sort -n -r 2.txt
    11
    10
    9
    8
    7
    6
    5
    4
    3
    2
    1
    Copy the code
  • merge

    [root@node01 tmp]# sort -nr 2.txt  
    11
    10
    9
    8
    7
    6
    5
    4
    3
    2
    1
    Copy the code

Step 4: Rank your grades

parameter English meaning
-t field-separator Specify field delimiters
-k key Sort by that column

‘ ‘

Sort -t ',' -k2nr score.txt 'sort -t ',' -k2nr score.txtCopy the code

3 wc command

3.1 the target

  • Displays information about the number of bytes, words, and lines of the specified file.

3.2 the path

  • Step 1: Display the number of bytes, words, and lines of the specified file.
  • Step 2: Display only the number of lines of the file
  • Step 3: count the number of lines words bytes of multiple files
  • Step 4: Check/etcHow many subcontents are in the directory

3.3 implementation

Step 1: Show the designationfile Bytes, words, linesInformation.

The command meaning
Wc filename According to specifiedfile Bytes, words, linesinformation
[root@hadoop01 export]# cat 4.txt
111
222 bbb
333 aaa bbb 
444 aaa bbb ccc
555 aaa bbb ccc ddd
666 aaa bbb ccc ddd eee

[root@hadoop01 export]# wc 4.txt 
 6 21 85 4.txt
Copy the code

Step 2: Display only the number of lines of the file

parameter English meaning
-c bytes The number of bytes
-w words Number of words
-l lines The number of rows
[root@hadoop01 export]# wc 4.txt 
 6 21 85 3.txt
Copy the code

Step 3: count the number of lines words bytes of multiple files

[root@hadoop01 export]# wc 1.txt 2.txt 3.txt 4 4 52 1.txt 11 11 24 2.txt 6 21 85 3.txt 21 36 161 total amount [root@hadoop01 Export # wc *.txt 4 4 52 1. TXT 11 11 24 2. TXT 6 21 85 3Copy the code

Step 4: Check/etcHow many subcontents are in the directory

[root@hadoop01 export]# ls /etc | wc -w
240
Copy the code

3.4 summary

  • throughWc fileYou canstatisticalOf the fileNumber of bytes, words, lines.

4 uniq

The uniq command is used to check and delete repeated lines in a text file. It is generally used together with the sort command.

4.1 the target

  • The uniq command is used to check and delete repeated lines in a text file. It is generally used together with the sort command.

4.2 the path

  • Step 1: Implement de-duplication effect
  • The second step: not only to the weight, but also statistics of the number of occurrences

4.3 implementation

Step 1: Implement de-duplication effect

The command English meaning
Uniq [parameter] file The only unique Remove duplicate rows
[root@hadoop01 export]# cat 5.txt Zhang SAN 98 Li Si 100 Wang Wu 90 Zhao Liu 95 Ma Qi 70 Li Si 100 Wang Wu 90 Zhao Liu 95 MA Qi 70 # sort [root@hadoop01 Export] # cat 5. TXT | sort li si 100 70 ma qi and li si 100 ma fifty and fifty and 90 90 70 98 zhang zhao six 95 zhao 95 # to heavy [root @ hadoop01 export] # cat 5. TXT | sort | uniq seven 70 fifty and 90 zhang SAN li si 100 ma zhao 6 95 98Copy the code

The second step: not only to the weight, but also statistics of the number of occurrences

parameter English meaning
-c count Count the number of occurrences in each line
[root @ hadoop01 export] # cat 5. TXT | sort | uniq -c 2 li si seven 2 fifty and 90 1 70 100 2 ma zhang zhao 6 95 98Copy the code

4.4 summary

  • throughUniq [options] fileCan be completed to repeat and count times

5 tee

5.1 the target

  • throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the

5.2 implementation

The command meaning
Command result | 1 file 2 3 tee file throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the
  • Place the results of de-statistics in a.tuck, B.tuck, and C.tuck files

    cat 5.txt | sort | uniq -c | tee a.txt b.txt c.txt
    Copy the code

5.3 summary

  • throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the

6 tr

6.1 the target

  • throughtrCommand is used toreplacedeleteCharacters in a file.

6.2 the path

  • Step 1: Implement the substitution effect
  • Step 2: Implement the delete effect
  • Step 3: Complete the word count case

6.3 implementation

Step 1: Implement the substitution effect

The command English meaning
Command result is replaced | tr characters New characters translate Implement the substitution effect
# replace lowercase I with uppercase I # convert itheima to uppercase # Convert HELLO to lowercaseCopy the code
# will be lowercase I replace uppercase I echo "itheima | tr" 'I' 'I' # put itheima converted to uppercase echo "itheima" | tr '[a-z]' '[a-z] # convert HELLO to lowercase echo "HELLO" |tr '[A-Z]' '[a-z]'Copy the code

Step 2: Implement the delete effect

The command English meaning
Command result | tr -d deleted characters delete Deletes the specified character
  • Requirement: Delete the number in abc1d4e5f
echo 'abc1d4e5f' | tr -d '[0-9]'
Copy the code

Step 3: Word count

The preparatory work
[root@hadoop01 export]# cat words.txt 
hello,world,hadoop
hive,sqoop,flume,hello
kitty,tom,jerry,world
hadoop
Copy the code

One replaces nu with a newline

2 the sorting

3 to heavy

4 count

[root @ # statistics the number of occurrences of each word hadoop01 export] # cat words. TXT | tr ', ' '\ n' | sort | uniq -c 1 flume hadoop 2 hello 1 hive jerry 1 kitty 1 sqoop 1 tom 2 worldCopy the code
  • The preparatory work

    Cat -n /etc/*. Conf >> /export/v.txt add the command result to /export/v.txtCopy the code

7 split

7.1 the target

  • throughsplitCommand willA large file Cut intoA number ofSmall file

7.2 the path

  • Step 1: Cut the large file into smaller files by byte
  • Step 2: Divide the large file into smaller files by the number of lines

7.3 implementation

Step 1: PressbyteCut large files into smaller ones

The command English meaning
Split-b 10k file byte Cut large files into pieces10KBThe small file

Step 2: PressThe number of rowsCut large files into smaller ones

The command English meaning
Split-l 1000 file lines Cut large files into piecesLine 1000The small file

7.4 summary

  • Split a large file into smaller files with the split option filename command

  • Preparation 1:

    vim score.txt

    zhangsan 68 99 26
    lisi 98 66 96
    wangwu 38 33 86
    zhaoliu 78 44 36
    maq 88 22 66
    zhouba 98 44 46
    Copy the code

8 awk

8.1 the target

  • throughawkimplementationFuzzy query.Extract fields as needed, can also be carried outjudgeAnd simpleoperationAnd so on.

Step 8.2

  • Step 1: Fuzzy query

  • Step 2: Specify the separator and display the content according to the subscript

  • Step 3: Specify a separator for the output field

  • Step 4: Call the functions provided by AWK

  • Step 5: Use the if statement to determine whether $4 passes

  • Step 6: Sum the contents of a segment

8.3 implementation

Step 1: Search for Zhangsan and Lisi’s results

The command meaning
awk ‘/zhangsan|lisi/’ score.txt Fuzzy query

Step 2: Specify the separator and display the content according to the subscript

The command meaning
awk -F ‘,’ ‘{print
1 . 1,
2, $3}’ 1.txt
TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

options

options English meaning
-F ',' field-separator useThe specified charactersegmentation
$+ digital To obtainHow many piececontent
$0 To obtainThe current linecontent
NF field Represents the number of fields in the current row
$NF Represents the last field
$(NF-1) Represents the penultimate field
NR The r is dealing with which row

Step 3: Specify a separator to display content by subscript

The command meaning
awk -F ‘ ‘ ‘{OFS=”===”}{print
1 . 1,
2, $3}’ 1.txt
TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

options

options English meaning
OFS = "character" output field separator Segment split string when output outward

Step 4: Call the functions provided by AWK

The command meaning
awk -F ‘,’ ‘{print toupper($2)}’ 1.txt TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

Common functions are as follows:

The function name meaning role
toupper() upper The character is converted to uppercase
tolower() lower Characters are converted to lowercase
length() length Return character length

Step 5: Query the information of students who pass the if statement

The command meaning
awk -F ‘,’ ‘{if(
4 > 60 ) p r i n t 4>60) print
1, $4 }’ score.txt
If you pass, you show it
1 . 1,
4
awk -F ‘,’ ‘{if(
4 > 60 ) p r i n t 4>60) print
1,
4 . Pass the ; e l s e p r i n t 4. pass the exam. else print
1, $4, “fail “}’ score.txt
Show name, $4, pass or not

options

parameter meaning
if(
0   a a ) p r i n t 0 ~ “aa”) print
0
If the line contains “AA “, the line is printed
if(
1   a a ) p r i n t 1 ~ “aa”) print
0
If the first paragraph of ** contains “aa”, print that line
if(
1 = = l i s i ) p r i n t 1 == “lisi”) print
0
ifThe first paragraph is equal to“Lisi “, just print this line

Step 6: Find the average score of the subject

The command meaning
Awk ‘BEGIN{initialize operation}{each line is executed} END{operation at END}’ filename BEGIN{this contains the statement before execution}

{here are the statements to execute when processing each line}

END {this is the statement to execute after all the lines are processed}
awk -F ',' 'BEGIN{}{total=total+$4}END{print total, NR, (total/NR)}' score.txt
Copy the code
  • The preparatory work

    vim 1.txt

    aaa java root
    bbb hello
    ccc rt
    ddd root nologin
    eee rtt
    fff ROOT nologin
    ggg rttt
    Copy the code

9 sed

9.1 the target

  • You can use SED to implement filtering and replacement.

9.2 the path

  • Step 1: implement the query function
  • Step 2: Implement the delete function
  • Step 3: Implement the modification function
  • Step 4: Implement replacement functionality
  • Step 5: Operate on the original file
  • Step 6: Comprehensive practice

9.3 implementation

Step 1: implement the query function

The command meaning
Sed optional target file To the target fileFilter queryreplace

Optional parameters

optional English meaning
p print print
$ That’s the last line
-n Only the processed results are displayed
-e expression Process according to the expression
  • Exercise 1: List rows 1 to 5 in 1.txt

Sed-n-e '1,5p' 1.txtCopy the code
  • Exercise 2 List all the data in 01.txt

sed -n -e '1,$p' 1.txt 
Copy the code
  • Exercise 3 List all data in 01.txt and display line numbers

optional meaning
= Prints the current line number
Sed -n -e '1,$=' -e '1,$p' 1.txt sed -n -e '1,$=' -e '1,$p' 1.txtCopy the code
  • Exercise 4: Find 01. TXT containing the root line

The answer:

sed -n -e '/root/p' 1.txt
Copy the code
  • Exercise 5 List the contents in 01.txt that contain root, which is case insensitive, and display the line number

optional English meaning
I ignore Ignore case

The answer:

nl 1.txt | sed -n -e '/root/Ip'

nl 01.txt | grep -i root

cat -n 01.txt | grep -i root
Copy the code
  • Exercise 6 Find the letters in 1. TXTrIt is followed by rows of multiple t’s and displays the line number

optional English meaning
-r regexp-extended Identification of a regular

The answer:

nl 01.txt | sed -nr -e '/r+t/p'
Copy the code

or

sed -nr -e '/r+t/p' -e '/r+t/=' 01.txt
Copy the code

Step 2: Implement the delete function

  • Exercise 1 Delete the first three lines in 01.txt and display the line number

optional English meaning
d delete Delete specified content

The answer:

Nl 01. TXT | sed -e '1, 3 d'Copy the code
  • Exercise 2 Retain the first 4 lines in 1.txt and display the line numbers

The answer:

Nl 01. TXT | sed -e '5, $d' nl 1. TXT | sed - n - e '1, 4 p'Copy the code

Step 3: Implement the modification function

  • Exercise 1: Add aaAAA after the second line of 01.txt and display the line number

parameter English meaning
i insert The targetIn front of theInsert content
a append The targetbehindAdditional content

The answer:

nl 01.txt | sed -e '2a aaaaa'
Copy the code
  • Exercise 2 add BBBBB before line 1 of 1.txt and display line number

The answer:

nl 01.txt | sed -e '1i bbbbb'
Copy the code

Step 4: Implement replacement functionality

  • Exercise 1 Replace nologin in 1.txt with Huawei and display the line number

English meaning
s/oldString/newString/ replace replace

The answer:

nl 1.txt | sed -e 's/nologin/huawei/'
Copy the code
  • Exercise 2 replace lines 1,2 in 01.txt with aaa and display line numbers

options English
2c New String replace Replace the selected line with a new string

The answer:

Nl passwd | sed -e '1, 2 c aaa'Copy the code

Step 5: Operate on the original file

  • Exercise 1 Replace Nologin with Huawei in 01.txt

parameter English meaning
-i in-place Replace the original file

The answer:

sed -i -e 's/nologin/huawei/' 01.txt
Copy the code
  • Exercise 2 is replaced with aaaaaa at lines 2 and 3 in 01.txt

The answer:

Sed - i-e '2,3c aaa' 01.txtCopy the code

Note: Before the operation, it is best to back up the data, put the operation wrong, the data cannot be recovered!

  • Exercise 3 Delete the first two lines in 01.txt and delete the data in the original file

The answer:

Sed -i -e '1,2d' 01.txt nl passwd view the dataCopy the code

Step 6: Comprehensive practice

  • Exercise 1 Obtaining an IP Address

The answer:

ifconfig eth0 | grep "inet addr" | sed -e 's/^.*inet addr://' | sed -e 's/Bcast:.*$//' 
Copy the code
  • Exercise 2 extracts data from 1.txt, matches content containing root, and replaces nologin with itheima

The answer:

Nl 01. TXT | grep 'root' | sed -e 's/nologin itheima/' or nl 01. TXT | sed - n - e'/root/p | sed -e 's/nologin itheima/' Or nl 01. TXT | sed - n - e '/ root / {s/nologin/itheima/p}' # show only replace the contentCopy the code
  • Exercise 3 extract the data from 1.txt, delete the first 2 lines, replace Nologin with itheima, and display the line number

The answer:

Nl 01. TXT | sed -e '1, 2 d | sed -e' s/nologin itheima/'Copy the code