Grep Utility page

This time about the external grep command on the bash command line.

history^[1]

Grep is a command line tool originally used on Unix operating systems. Given a list of files or standard input, grep searches for text that matches one or more regular expressions and outputs only lines or text that matches (or doesn’t).

Grep is an application originally written by Ken Thompson. Grep was originally an application program under Ed, and its name comes from G/RE/P (globally search a regular expression and print). In Ed, the command g/re/p will print out all strings matching the defined styles in behavior units.

In 1973, version 4 of Unix, grep first appeared in man pages.

The above is mainly from Wikipedia.

function

Grep uses regular expressions to search for text and print out matching lines. As input, text can come from standard input or from any number of wildcard files, and the new version of grep also supports regular matching and searching through all files against subfolders of the current directory.

Typical grep options include:

Mode selection and interpretation:

-e uses the template style as extended plain notation, meaning that extended regular expressions can be used. (Extended regular expression) -f treats the template style as a list of fixed strings. (Newline-Separated Strings) -g uses the template style as a basic re. (Basic regular expression) -p Use the template style as a Perl representation. (Perl regular expression) -e< template style > Specifies a string as the template style for finding file contents. -f< Template file > Specifies a template file with one or more template styles. Grep is asked to find the template content that meets the template conditions in the template style of each column. -i Ignores the difference of case and case. -w Displays only the columns that match the full word. -x Displays only the columns that match all columns.

Miscellaneous:

-v Reverse lookup. -s Indicates that no error information is displayed.

Output control:

-b Displays the byte offset of the output line from the beginning of the file. -c Calculates the number of columns that match the template style. -h Does not indicate the file name of the column before displaying the column that matches the template style. -h Indicates the file name of the column before displaying the column that matches the template style. -l Lists the names of files whose contents match the specified template style. -l Lists the names of files whose content does not match the specified template style. -n Indicates the column number before displaying the column that matches the template style. -o Displays only the matched parts of the file. -q Displays no information. -r /-r This parameter has the same effect as specifying the -d recurse parameter.

Content control:

-b < Show number of columns > Displays the content before the line except the one that matches the template style. -a < Show column number > Displays everything after the line except the one that matches the template style. -c < Show column number > or -< Show column number > displays the contents before and after the column except the column that matches the template style.

See the grep –help output for more options.

Usage: grep [OPTION]... PATTERN [FILE]...
Search for PATTERN in each FILE.
Example: grep -i 'hello world' menu.h main.c
Copy the code

For complete reference manuals, run the man grep and info grep commands.

Basic usage^[2]

Search for a word in the file and the command returns a line of text containing ** “match_pattern” ** :

grep match_pattern file_name
grep 'match_pattern' file_name
grep "match_pattern" file_name
Copy the code

The above three commands are equivalent to grep. The difference is that single quotes prevent Spaces in match_pattern and prohibit bash nested calculations (such as $var variable embedding), Double quotes, while having the effect of single quotes, also support bash variable expansion, bash command nesting, bash arithmetic expression evaluation, and expansion.

Find in multiple files:

grep "match_pattern" file_1 file_2 file_3 ...
Copy the code

Print all lines except the -v option:

grep -v "match_pattern" file_name
Copy the code

Again, such as

ps -auxef|grep java|grep -v grep
Copy the code

Grep -v grep here means to exclude instances with grep text from the previous result (all Java running instances). In effect, this is to rule out instances of grep Java so that we have a pure running instance of Java.

–color=auto option:

grep "match_pattern" file_name --color=auto
Copy the code

Use the regular expression -e option:

grep -E "[1-9] +"
# or
egrep "[1-9] +"
Copy the code

Egrep indicates Extended regular expression syntax.

Prints only the part of the file that matches the -o option:

echo this is a test line. | grep -o -E "[a-z]+\."
line.

echo this is a test line. | egrep -o "[a-z]+\."
line.
Copy the code

Count the number of lines in a file or text containing a matching string -c option:

grep -c "text" file_name
Copy the code

Print the number of lines containing the matching string -n option:

grep "text"- n file_name or cat file_name | grep"text" -n

# multiple files
grep "text" -n file_1 file_2
Copy the code

The print style matches the character or byte offset at which it is located:

echo gun is not unix | grep -b -o "not"
7:not

The value of a string in a # line is calculated from the first character of the line, starting with a value of 0. The -b-o option is always used in conjunction.
Copy the code

Search multiple files and find which files the matching text is in:

grep -l "text" file1 file2 file3...
Copy the code

Ignore character case in matching styles:

echo "hello world" | grep -i "HELLO"
hello
Copy the code

The -e option specifies multiple matching styles:

echo this is a text line | grep -e "is" -e "line" -o
is
line

# You can also use the -f option to match multiple styles, writing out the characters to be matched line by line in the style file.
cat patfile
aaa
bbb

echo aaa bbb ccc ddd eee | grep -f patfile -o
Copy the code

Grep recursively searches for files

Recursive search for text in a multi-level directory:

grep "text" . -r -n
#. Indicates the current directory.
Copy the code

Include or exclude specified files in grep search results:

# recursively search for "main()" in all.php and.html files in the directory
grep "main()" . -r --include *.{php,html}

# Exclude all README files from search results
grep "main()" . -r --exclude "README"

# exclude files in the filelist filelist from the search results
grep "main()" . -r --exclude-from filelist
Copy the code

Grep with xargs with the zero-value byte suffix:

# test file:
echo "aaa" > file1
echo "bbb" > file2
echo "aaa" > file3

grep "aaa" file* -lZ | xargs -0 rm
# delete file1 and file3 after execution, grep output uses the -z option to specify zero-valued byte as the terminator filename (\0), xargs-0 reads the input and separates the filename with a zero-valued byte terminator, then deletes the matching file, -z is usually used in conjunction with -l.
Copy the code

Grep Silent output:

grep -q "test" filename
The command returns 0 on success and non-0 on failure. Generally used for conditional testing.
Copy the code

Print the line before or after the matching text:

# display 3 lines after matching A result, using the -a option:
seq 10 | grep "5" -A 3
5
6
7
8

# display 3 lines before matching a result using the -b option:
seq 10 | grep "5" -B 3
2
3
4
5

# display the first three and last three lines matching a result, using the -c option:
seq 10 | grep "5" -C 3
2
3
4
5
6
7
8

# if there are more than one matching result, we will use "--" as the separator between each matching result:
echo -e "a\nb\nc\na\nb\nc" | grep a -A 1
a
b
--
a
b
Copy the code

Grep -p Enables The Perl syntax rule. At this point you can write rules using Perl regular syntax.

Perl regular syntax, also known as PCRE expressions, is available in the Wiki for the complete set of PCRE expressions.

Common usage

find text string recursively

In a folder, I do not know which files contain fantasy text description, you can find it like this:

grep -PHni 'fantasy' * -r
Copy the code

This command lists all files in the current folder that contain Fantasy, listing their file names and the lines that contain fantasy text and their line numbers.

If you also need to look at the context of the matching text, you can use:

grep -PHni 'fantasy' * -r -C 3
Copy the code

-p indicates the Perl regular syntax

-h displays the file name of the matching line

-n Displays the line number of a matching line

-i indicates that case is ignored

-c 3 Indicates that the first and second lines are listed.

-b 3 indicates that the first three lines are also enumerated.

-a 3 indicates that the next three lines are also listed.

find ip address

With the -o argument, grep is often used to extract the text content of a particular pattern, rather than output the entire matching line.

Such as:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 56:00:01:c6:ab:01 brd ff:ff:ff:ff:ff:ff
    inet 217.179.87.159/23 brd 217.179.87.255 scope global dynamic ens3
       valid_lft 63125sec preferred_lft 63125sec
3: ens7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5a:00:01:c6:ab:01 brd ff:ff:ff:ff:ff:ff

$ ip addr | grep -Po 'inet \d+\.\d+\.\d+\.\d+' | grep -v 'inet 127' | grep -Po '\d+.+'
217.179.87.159
Copy the code

Here the expression will extract two lines of ‘inet XXXXXX ‘in the form:

Inet 127.0.0.1 inet 217.179.87.159Copy the code

Expression 2 excludes line 127.0.0.1, expression 3 removes the inet prefix, and we have the DESIRED IP address.

A similar approach can be used to extract IPv6 addresses.

Awk ‘{print $2}’ awk ‘{print $2}’ This phrase splits the input text into n segments by space. $2 indicates the second segment, which is the DESIRED IP address.

ports

If you want to find the service listening to the port on the current host, you can use the output of the lsof command:

$ sudo lsof -Pni|grep LISTEN
sshd        858              root    3u  IPv4    19572      0t0  TCP *:22 (LISTEN)
sshd        858              root    4u  IPv6    19582      0t0  TCP *:22 (LISTEN)
nginx      6170              root    9u  IPv4 53951827      0t0  TCP *:443 (LISTEN)
nginx      6170              root   10u  IPv4 53951828      0t0  TCP *:8060 (LISTEN)
nginx      6170              root   11u  IPv4 53951829      0t0  TCP *:80 (LISTEN)
Copy the code

From this, we can write a common command function ports and put it in a.bashrc file so that we can easily see the port numbers. The function can be written like this:

ports () {
    local x=The $1
    if [ "$x"= ="" ]; then
        sudo lsof -Pni|grep -P 'LISTEN|UDP'
    else
        sudo lsof -Pni|grep -P 'LISTEN|UDP'|grep ":$x"
    fi
}
Copy the code

Then we can use it like this:

ports
ports 443
ports 22
Copy the code

Note that it is best to change your Linux account to password-free sudo, otherwise you may need to enter your own password to gain sudo identity when using Ports. Of course, if you only want to check the port number of the service you started, you can remove the sudo directive.

has-user, has-group

How do I check if a Linux account exists?

There is no generic command in Linux specifically for this detection. Often commands like useradd return a failure in the presence of the user, but this is not the proper way to detect it.

To do this, we have to interpret the /etc/passwd file ourselves. This file lists all the accounts in the system in something like this:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
...
Copy the code

Therefore, to determine whether a user exists, you only need to determine the first field.

Obviously, AWK is good for this:

$ cat /etc/passwd|awk -F: '{print $1}'
root
daemon
bin
sys
sync
Copy the code

In this article, grep is used to solve the problem:

has-user() {
    local name=${1:-root}
    cat /etc/passwd|grep -q "^$name"
}

has-user 'joe' && echo 'joe exists' || 'joe not exists'
Copy the code

Similarly, we can define a similar function has-group:

has-group () {
    local name=${1:-root}
    cat /etc/group|grep -q "^$name"
}

has-group staff && echo 'staff group exists' || echo 'staff group not exists'
Copy the code

More and more

Here are some practical examples:

function find_ip () { ip addr|grep -Poi "The inet ((192.168. \ d +. \ d +) | (172. \ d +. \ d +. \ d +) | (10. \ d +. \ d +. \ d +))"|grep -Poi "\d+.\d+.\d+.\d+"; }

function find_ip_uniq () { ip addr|grep -Poi "The inet ((192.168. \ d +. \ d +) | (172. \ d +. \ d +. \ d +) | (10. \ d +. \ d +. \ d +))"|grep -Poi "\d+.\d+.\d+.\d+"|grep -v '255 \.|head -n1; }

genpasswd(){ strings /dev/urandom|grep -oP '[[:alnum:]]|[\#\%\@\&\^]'|head -n "The ${16} 1: -"|tr -d '\n';echo; }Copy the code

conclusion

Grep and AWk, sed are the three tools of Linux, which to a large extent represent the design philosophy of Linux, that is, small, focused and combined. The biggest trick with a tool like grep is to decompose the target behavior: take the source text, filter the source text, and construct the resulting output.

This article has only covered the basics of usage, but it’s up to you to get your wits about you.

reference

---

1. Zh.wikipedia.org/wiki/Grep ↩ ︎

2. Man.linuxde.net/grep ↩ ︎