Grep, sed, AWK efficient file processing 3 swordsmen

Grep, sed, awk, we called them the three musketeers, master they can better operations, improve work efficiency, if not operations, processing data is very convenient for us ~ in terms of a lot of data processing, programming is certainly also can deal with, but far from existing specific functions of command more efficient, we only need to commands. Through this article can explain the three musketeers some basic knowledge and practical, I hope you can start knocking, after all, their experience of the impression is more impressive, the back will continue to update…

grep

Introduction to the

Grep is a powerful text search tool that supports regular expressions.

Full name (Global Search Regular Expression (RE) and Print out the line)

Grammar: grep [option]… PATTERN [FILE]…

Commonly used:

usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
 [-e pattern] [-f file] [--binary-files=value] [--color=when]
 [--context[=num]] [--directories=action] [--label] [--line-buffered]
 [--null] [pattern] [file ...]
Copy the code

Common parameters:

Take anti - v-i ignores case-c Number of qualified rows-n Prints the line number with the output* * begins with **$ends with *^ $a blank lineCopy the code

The actual use

Prepare a short story TXT:

[root@iz2ze76ybn73dvwmdij06zz ~]# cat monkey
One day,a little monkey is playing by the well. One day, a little monkey is playing by the well.He looks inHe looked into the well and shouted:"Oh! My god! The moon has fallen into The well!" "Oh! My god! The moon has fallen into the well!"An older monkeys run over,takes a look,and says,"Goodness me! The moon is reallyinThe water!" "Bad! The moon has fallen into the well!"And olderly monkey comes over. The old monkey runs over.He is very surprised as well and cries out:"The moon isin"The moon is in the well."A group of monkeys run over to the well . A group of monkeys run over to the well.They look at the moon inThe well and shout: They see the moon in the well and shout:"The moon did fall into The well! Come on! Let 'get it out!""The moon has fallen into the well! Come on! Let's get it out!"Then,the oldest monkey hangs on the tree up side down ,with his feet on the branch . Then, the oldest monkey hangs on the tree upside down.And he has the next monkey's feet with his hands. He took the older monkey by the feet,All the other monkeys follow his suit.And they join each other one by one down to the moon in the well. One by one they end up in the well.Just before they reach the moon,the oldest monkey raises his head and happens to see the moon inJust as they reach the moon, the oldest monkey raises his head and sees the moon in the skyHe yells excitedly" Don't be so foolish! The moon is stillinThe sky!"He yells excitedly, "Don't be so stupid! The moon is still in the sky!Copy the code

Find the eligible rows directly

[root@iz2ze76ybn73dvwmdij06zz ~]# grep moon monkey
"Oh! My god! The moon has fallen into The well!" "Oh! My god! The moon has fallen into the well!""Goodness me! The moon is reallyinThe water!" "Bad! The moon has fallen into the well!""The moon isin"The moon is in the well."They look at the moon inThe well and shout: They see the moon in the well and shout:"The moon did fall into The well! Come on! Let 'get it out!"And they join each other one by one down to the moon in the well. Just before they reach the moon,the oldest monkey raises his head and happens to see the moon inJust as they reach the moon, the oldest monkey raises his head and sees the moon in the skyHe yells excitedly" Don't be so foolish! The moon is stillinThe sky!"Copy the code

Find rows that match the condition in reverse

[root@iz2ze76ybn73dvwmdij06zz ~]# grep -v moon monkey
One day,a little monkey is playing by the well. One day, a little monkey is playing by the well.He looks inHe looked into the well and shouted:An older monkeys run over,takes a look,and says,And olderly monkey comes over. The old monkey runs over.He is very surprised as well and cries out:A group of monkeys run over to the well . A group of monkeys run over to the well."The moon has fallen into the well! Come on! Let's get it out!"Then,the oldest monkey hangs on the tree up side down ,with his feet on the branch . Then, the oldest monkey hangs on the tree upside down.And he has the next monkey's feet with his hands. He took the older monkey by the feet,All the other monkeys follow his suit.One by one they end up in the well.He yells excitedly, "Don't be so stupid! The moon is still in the sky!"Copy the code

Find the number of rows that match the condition directly

[root@iz2ze76ybn73dvwmdij06zz ~]# grep -c moon monkey
8
Copy the code

Find the number of eligible rows regardless of case

Let’s look at the results of the direct lookup

[root@iz2ze76ybn73dvwmdij06zz ~]# grep my monkey
Copy the code

Ignore case view

[root@iz2ze76ybn73dvwmdij06zz ~]# grep -i my monkey
"Oh! My god! The moon has fallen into The well!" "Oh! My god! The moon has fallen into the well!"
Copy the code

Find the line that matches the criteria and print the line number

[root@iz2ze76ybn73dvwmdij06zz ~]# grep -n monkey monkey
1:One day,a little monkey is playing by the well. One day, a little monkey is playing by the well.4:An older monkeys run over,takes a look,and says,6:And olderly monkey comes over. The old monkey runs over.9:A group of monkeys run over to the well . A group of monkeys run over to the well.13:Then,the oldest monkey hangs on the tree up side down ,with his feet on the branch . 15:And he has the next monkey's feet with his hands. He took the older monkey by the feet,16:All the other monkeys follow his suit.19:Just before they reach the moon,the oldest monkey raises his head and happens to see the moon inJust as they reach the moon, the oldest monkey raises his head and sees the moon in the skyCopy the code

Find the row that starts with J

[root@iz2ze76ybn73dvwmdij06zz ~]# grep '^J' monkey
Just before they reach the moon,the oldest monkey raises his head and happens to see the moon inJust as they reach the moon, the oldest monkey raises his head and sees the moon in the skyCopy the code

Find the line that ends with a?

[root@iz2ze76ybn73dvwmdij06zz ~]# grep "$" monkey
Just before they reach the moon,the oldest monkey raises his head and happens to see the moon inJust as they reach the moon, the oldest monkey raises his head and sees the moon in the skyCopy the code

You can grep –help to see more related commands, which are not demonstrated here.

summary

With a strong network, a lot of things can be found on the Internet, but the foundation must be familiar with their own, just back when things do not panic.

sed

Sed is a stream editor. It is an excellent tool for processing text and can be used in conjunction with regular expressions.

Sed Execution Process

The sed command

Command: sed

Sed [选 词]… {command set} [input file]…

Common commands:

D Deletes the selected rowS findY to replaceI Inserts a line before the current lineA Inserts a row after the current rowP printed lineQ exit Replace the operator: Number: replace the numberG: Global replacement\1: substring matching mark, the preceding search can be used with meta character set \(.. \)&: Reserve the search knife character to replace other charactersCopy the code

Operation:

replace

View files:

➜  happy cat word
Twinkle, twinkle, little star
How I wonder what you are
Up above the world so high
Like a diamond in the sky
When the blazing sun is gone Copy the code

Replacement:

➜  happy sed 's/little/big/' word
Twinkle, twinkle, big star
How I wonder what you are
Up above the world so high
Like a diamond in the sky
When the blazing sun is gone Copy the code

View text:

➜  happy cat word1
Oh if there's one thing to be taught
it's dreams are made to be caught
and friends can never be bought
Doesn't matter how long it's been
I know you'll always jump in 'Cause we don't know how to quit Copy the code

Global replacement:

➜  happy sed 's/to/can/g' word1
Oh if there's one thing can be taught
it's dreams are made can be caught
and friends can never be bought
Doesn't matter how long it's been
I know you'll always jump in 'Cause we don't know how can quit Copy the code

Replace by line (replace 2 to last line)

➜  happy sed '2,$s/to/can/' word1
Oh if there's one thing to be taught
it's dreams are made can be caught
and friends can never be bought
Doesn't matter how long it's been
I know you'll always jump in 'Cause we don't know how can quit Copy the code

Delete:

View text:

➜  happy cat word
Twinkle, twinkle, little star
How I wonder what you are
Up above the world so high
Like a diamond in the sky
When the blazing sun is gone Copy the code

Delete:

➜  happy sed '2d' word
Twinkle, twinkle, little star
Up above the world so high
Like a diamond in the sky
When the blazing sun is gone
Copy the code

Display line number:

➜  happy sed '=; 2d' word
1
Twinkle, twinkle, little star
2
3
Up above the world so high 4 Like a diamond in the sky 5 When the blazing sun is gone Copy the code

Delete lines 2 through 4:

➜  happy sed '=; 2, 4 d ' word
1
Twinkle, twinkle, little star
2
3
4 5 When the blazing sun is gone Copy the code

Add the line:

Insert forward:

➜  happy echo "hello" | sed 'i\kitty'
kitty
hello
Copy the code

Insert backwards:

➜  happy echo "kitty" | sed 'i\hello'
hello
kitty
Copy the code

Modify the line:

Replace the second action with Hello Kitty

➜  happy sed '2c\hello kitty' word
Twinkle, twinkle, little star
hello kitty
Up above the world so high
Like a diamond in the sky
When the blazing sun is gone Copy the code

Replace the second to last line with Hello Kitty

➜  happy sed '2,$c\hello kitty' word
Twinkle, twinkle, little star
hello kitty
Copy the code

Written to the row

Write the line with star to c file,c is created ahead of time

➜ happy sed - n'/star/w c' word
➜  happy cat c
Twinkle, twinkle, little star
Copy the code

exit

After printing three lines, exit sed

➜  happy sed '3q' word
Twinkle, twinkle, little star
How I wonder what you are
Up above the world so high
Copy the code

awk

Name origin

The first letter of the surnames of founders Alfred Aho, Peter Weinberger and Brian Kernighan.

Powerful text processing tools

Awk is more than just a small tool compared to sed and grep. It is also a small programming language, with support for if judgment branches and while loops and built-in functions. It is a more powerful text-processing tool than grep and sed, but there is much more to learn.

Here are some of the basic concepts and operations of AWK.

grammar

The commonly used

Usage: awk [POSIX or GNU style options] -f progfile [–] file …

Usage: awk [POSIX or GNU style options] [–] ‘program’ file …

The domain

It’s kind of a database column concept, but it’s specified by ordinal number, like if I want the first column2, and so on. $0 is the output of the entire text. The default separator is space, although you can use -f to set the separator to suit your situation.

In advance, I compiled a section of data, students and student performance data table.

The number of columns	The name of the	describe
1	Name	The name
2	Math	mathematics
3	Chinese	Chinese language and literature
4	English	English
5	History	history
6	Sport	sports
8	Grade	The class

“Name Math Chinese English History Sport grade

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '{print $0}' students_store
Xiaoka          60   80    40    90   77  class-1
Yizhihua        70    66   50    80   90  class-1
kerwin          80    90   60    70   60  class-2
Fengzheng       90    78    62   40   62  class-2
Copy the code

Print the first column (name column)

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '{print $1}' students_store
Xiaoka
Yizhihua
kerwin
Fengzheng
Copy the code

Patterns & Actions

awk '{[pattern] action}' {filenames}  
Copy the code

model

The pattern can be

• Conditional statements
• regular

The two special fields of the schema BEGIN and END (the number of lines to match or print when not specified)

• BEGIN: Indicates the column name.

• END: Generally used to print a summary character.

action

The action is specified in {} and is usually used for printing or as a code snippet.

The sample

Add a header to the text above:

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print "Name Math Chinese English History Sport grade\n----------------------------------------------"} {print $0}' students_store

Name         Math  Chinese  English History  Sport  grade
----------------------------------------------------------
Xiaoka       60    80       40      90     77    class-1
Yizhihua 70 66 50 80 90 class-1 kerwin 80 90 60 70 60 class-2 Fengzheng 90 78 62 40 62 class-2 Copy the code

Just print your name, math grade, class information, and add a suffix (keep up the good work):

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print "Name Math grade\n---------------------"} {print $1 2 "\t" $7} END {print "continue to exert oneself"}' students_store

Name     Math  grade
---------------------
Xiaoka   60   class-1
Yizhihua 70 class-1 kerwin 80 class-2 Fengzheng 90 class-2 continue to exert oneself Copy the code

Combined with the regular

Grep and sed also support regular expressions. I won’t introduce regular expressions here, but I’ll give you a single article if you’re interested.

Usage:

The symbol ~ is followed by a regular expression

At this point we add a new student later, and there is no class division.

Let’s take a look at the current data

[root@iz2ze76ybn73dvwmdij06zz ~]# cat students_store
Xiaoka       60   80    40    90   77  class-1
Yizhihua     70    66   50    80   90  class-1
kerwin       80    90   60    70   60  class-2
Fengzheng    90    78   62    40   62  class-2
xman - - - - - - Copy the code

Fuzzy matching | query is divide into classes of students

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '$0 ~/class/' students_store
Xiaoka       60   80    40    90   77  class-1
Yizhihua     70    66   50    80   90  class-1
kerwin       80    90   60     70  60  class-2
Fengzheng    90    78   62     40  62  class-2
Copy the code

Precise matching | query the students of class 1

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '$7=="class-1" {print $0}' students_store
Xiaoka       60   80    40    90   77  class-1
Yizhihua     70    66   50    80   90  class-1
Copy the code

Reverse matching | query is not the students of class 1

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '$7!="class-1" {print $0}'  students_store
kerwin       80    90   60     70   60  class-2
Fengzheng    90    78    62     40  62 class-2
xman         -     -     -     -   -    -
Copy the code

Comparison operation

Select * from math > 80

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '$2>60 {print $0}' students_store
Yizhihua     70    66   50    80   90  class-1
kerwin       80    90   60     70   60  class-2
Fengzheng    90    78    62     40  62 class-2
Copy the code

Select * from math > English

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '$2 > $4 {print $0}' students_store
Xiaoka       60   80    40    90   77  class-1
Yizhihua     70    66   50    80   90  class-1
kerwin       80    90   60    70   60  class-2
Fengzheng    90    78    62   40   62  class-2
Copy the code

Matches any character in the specified character

Let’s add a list of majors. Let’s take a look at Han’s majors, and give the last new kid a class.

And then let’s look at the data at this point.

[root@iz2ze76ybn73dvwmdij06zz ~]# cat students_store
Xiaoka       60   80    40    90   77  class-1  Java
Yizhihua     70    66   50    80   90  class-1  java
kerwin       80    90   60     70   60  class-2 Java
Fengzheng    90    78    62     40  62 class-2  java
xman - - - - - class-3 php Copy the code

| or relationship matching query students of class 1 and class 3

root@iz2ze76ybn73dvwmdij06zz ~]# awk '$0 ~/(class-1|class-3)/' students_store
Xiaoka       60   80    40    90   77  class-1  Java
Yizhihua     70    66   50    80   90  class-1  java
xman         -     -     -     -   -   class-3 php
Copy the code

Match any character | name is the second letter

Character interpretation:

^ : The beginning of a field or record.

. : Any character.

root@iz2ze76ybn73dvwmdij06zz ~]# awk '$0 ~/(class-1|class-3)/' students_store
Xiaoka       60   80    40    90   77  class-1  Java
Yizhihua     70    66   50    80   90  class-1  java
xman         -     -     -     -   -    class-3 php
Copy the code

Compound expression

&& AND

The relationship, must meet at the same time oh ~

Select * from children where math > 60 and Chinese > 60;

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '{ if ($2 > 60 && $3 > 60) print $0}' students_store
Yizhihua     70    66   50    80   90  class-1  java
kerwin       80    90   60    70   60  class-2  Java
Fengzheng    90    78    62   40   62  class-2  java
Copy the code

|| OR

Select * from children where math > 80 or Chinese > 80.

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '{ if ($2 > 80 || $4 > 80) print $0}' students_store
Fengzheng    90    78    62     40  62 class-2  java
Copy the code

Printf formats the output

In addition to being functional, a nice format is also essential, so formatted output will look more comfortable

grammar

Printf ([format], parameter)

Printf %x(format) The specific argument x represents the specific format

symbol	instructions
–	The left
Width	Step length of the domain
.prec	Maximum string length or number of digits to the right of the decimal point

Format converter

It’s pretty much the same as any other language

Commonly used formats

symbol	describe
%c	ASCII
%d	The integer
%o	octal
%x	A hexadecimal number
%f	Floating point Numbers
%e	Floating point number (scientific notation)
% s	string
%g	Decide to use the floating-point conversion E/F

Operation Examples

ASCII码🐎

[root@iz2ze76ybn73dvwmdij06zz ~]# echo "66" | awk '{printf "%c\n",$0}'
B
Copy the code

Floating point Numbers

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {printf "%f\n",100}'
100.000000
Copy the code

Hexadecimal number

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {printf "%x",996}'
3e4
Copy the code

More operations, small partners interested can try one by one ~

Built-in variables

High frequency common built-in variable

NF: Records the number of browse fields. Set this parameter after records are read.

NR: indicates the number of read records.

FS: Sets the input field separator

A R G C: Indicates the number of command line parameters.

RS: Control record separator

FIlENAME: indicates the name of the file awk is currently reading

operation

Output the number of student transcripts and fields and the number of read records.

[root@iz2ze76ybn73dvwmdij06zz ~]# awk '{print $0, NF , NR}' students_store
Xiaoka       60   80    40    90   77  class-1  Java 8 1
Yizhihua     70    66   50    80   90  class-1  java 8 2
kerwin       80    90   60     70  60  class-2  Java 8 3
Fengzheng    90    78   62     40  62  class-2  java 8 4
xman - - - - - class-3 php 8 5 Copy the code

Built-in function

Commonly used functions

Length (s) returns s length

Index (s,t) returns the first occurrence of the string t in s

Match (s,r) s specifies whether r is contained in the string

Split (s,a,fs) Splits S into sequence A at fs

Gsub (r,s) with s instead of r, scope full text

Gsub (r, S,t) in range T,s replaces r

Substr (s,p) returns the string s starting at position P and the rest of it starting at 1.

Substr (s,p,n) returns the following n parts of the string s starting at position P

operation

length

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print length(" hello,im xiaoka")}'
16
Copy the code

index

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print index("xiaoka","ok")}'
4
Copy the code

match

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print match("Java ","va ")}'
3
Copy the code

gsub

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'gsub("Xiaoka","xk") {print $0}' students_store
xk       60   80    40    90   77  class-1  Java
Copy the code

substr(s,p)

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print substr("xiaoka",3)}'
aoka
Copy the code

substr(s,p,n)

[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN {print substr("xiaoka",3,2)}'
ao
Copy the code

split

[root@iz2ze76ybn73dvwmdij06zz ~]# str="java,xiao,ka,xiu"
[root@iz2ze76ybn73dvwmdij06zz ~]# awk 'BEGIN{split('"\"$str\""',ary,","); for(i in ary) {if(ary[i]>1) print ary[i]}}'
xiu
java
xiao
ka Copy the code

Awk script

As mentioned earlier, AWK is a small programming language, so to speak. If the command is short, you can execute it directly on the command line. If the command line is long, you can use scripts to process it, which is more readable than the command line. You can also add comments.

Write a complete AWK script and execute the steps

1. Create an AWK file

[root@iz2ze76ybn73dvwmdij06zz ~]# vim printname.awk
Copy the code

2. Specify the interpreter in the first line of the script

#! /usr/bin/awk -f
Copy the code

3. Write the script content and print the name

[root@iz2ze76ybn73dvwmdij06zz ~]# cat printname.awk
#! /usr/bin/awk -f
# you can add a comment, haha
BEGIN { print "My Name is Java"}
Copy the code

4. Since it is a script, the essential executable permission arrangement on ~

[root@iz2ze76ybn73dvwmdij06zz ~]# chmod +x printname.awk
[root@iz2ze76ybn73dvwmdij06zz ~]# ll printname.awk
-rwxr-xr-x 1 root root 60 7月   1 15:23 printname.awk
Copy the code

5. With executable permission, let’s execute and see the result

[root@iz2ze76ybn73dvwmdij06zz ~]# ./printname.awk
My name is Java xiaokxiuCopy the code

After understanding the steps of writing AWK scripts, you can write your own wave ~

There’s a lot more to it, but I’ll share it later

Behind will continue to improve my Linux actual combat command small manual, continue to update…

Thank you guys for your support

Another book arrived today, updated continuously. ⛽ ️

reference

Introduction to AWK

Linux Command Line and Shell Scripts

Birdman’s Linux Home Dish

Happy Command Line

Baidu encyclopedia

This article is formatted using MDNICE