Original: Taste of Little Sister (wechat official ID: XjjDog), welcome to share, please reserve the source. Any reprint that does not retain this statement is plagiarism.

Xjjdog has been in bad shape lately. Maybe it was spring, or maybe it was the result of all the other things, and wang was in bad shape. Is my influence on him, let him in this virtual world of continuous YY and continuous destruction. Here, XjjDog would like to say to the virtual Wang: I’m sorry that I didn’t give you the ability to play five episodes of a Korean DRAMA.

But being long and smelly is a style, and Wang decided to study this girl as carefully as he studied Linux commands. If a girl were a piece of white paper, what would she write on it? If not, what could he change? But first, he needs to pick up a pen.

As you probably already know, the first column of the ll-l command displays the file type for Linux. Get a general idea of this, because many of the commands that follow will use this knowledge.

  • -Represents a common file
  • dRepresenting directory files
  • lRepresents linked files, such as shortcuts
  • sSocket file
  • cCharacter device files, for example/dev/Many of the files in
  • bRepresents block device files, such as some disks
  • pPipeline file

Files on Linux can be suffix-free, and you can create some counterintuitive files. For example, the suffix is PNG, but it is a compressed file (which is not usually done). When I was in college, there were clever students who hid small movies like this, and the effect was very good.

To see the specific type of file, you can use the file command, which is clever enough to recognize many file formats.

[root@localhost ~]# file /etc
/etc: directory
[root@localhost ~]# file /etc/group
/etc/group: ASCII text
[root@localhost ~]# file /dev/log
/dev/log: socket
[root@localhost ~]# file /dev/log
/dev/log: socket
[root@localhost ~]# file /bin
/bin: symbolic link to `usr/bin'
Copy the code

The operations in this section are for plain ASCII text files. Next, we will create some files. Then write something to the file for subsequent operations.

1. Create a file

1.1. Numeric sequences

Using redirects, you can generate files directly. Next, I’m going to generate 10 to 20 numbers, one line for each number, and write it to a file called Spring. Coincidentally, the seq command can do this.

seq 10 20 >> spring
Copy the code

We mentioned earlier that > means redirecting the output of the previous command to somewhere else. Here, we use two >, which still means redirection, but means append content to the original file.

That’s what w+ and A + stand for in programming languages.

1.2. View the content

n 

[root@localhost ~]# cat spring
10
11
12
13
14
15
16
17
18
19
20
[root@localhost ~]# cat -n spring
1	10
2	11
3	12
4	13
5	14
6	15
7	16
8	17
9	18
10	19
11	20
Copy the code

The cat command is often used for more than just viewing the contents of files. Only by uniting with other commands can it find meaning in its life.

Merge file A and file B into file C
cat a  b>> c

# pipe the contents of file A as input. We'll talk about that later
cat a | cmd

Write content to the specified file. Very common in shell scripts. And we're going to use this notation a couple of times
cat > index.html <<EOF
<html>
    <head><title></title></head>
    <body></body>
</html>
EOF
Copy the code

Since our files are small, the cat command does no harm. However, if the file is several gigabytes, it is much more dangerous to use cat, a little command called cat, which will output frantically on the terminal and you can terminate it by pressing CTRL + C multiple times.

2, look at the file peacefully

Since cat is not suitable for large files, there must be an alternative. Less and more. Less is generally used today because it is faster than more. Its main use is to page through the contents of files and provide some quick lookup methods. Less is an interactive command, and you need to use some keyboard shortcuts to control it.

This time we used SEQ to generate 10 million rows, a whopping 76MB in size, and opened it with less.

[root@localhost ~]# seq 10000000 > spring
[root@localhost ~]# du -h spring
76M	spring
[root@localhost ~]# less spring
Copy the code

The general operations for LESS are as follows:

  • The blank spaceScroll down to turn the page
  • bScroll up to turn the page
  • /Go into search mode, for example/ 1111The word 1111 will be looked for
  • qExit the less
  • gTo the beginning
  • GGo to the end
  • jScroll down
  • kScroll up. These two buttons are very similar to what VIm does

3. File header and end

Head displays the file header, and tail displays the file tail. They can both specify the number of rows with the argument -n.

[root@localhost ~]# head -n 3 spring
1
2
3
[root@localhost ~]# tail -n 3 spring
9999998
9999999
10000000
Copy the code

Tail -f is probably one of the most commonly used commands for some programmers. It can monitor file changes in real time at the control terminal, and look at some scrolling logs. Such as looking at nginx or Tomcat logs, etc. In normal cases, too fast log scrolling still causes some problems. You need to use the grep command to filter logs.

# Scroll to view system logs
tail -f /var/log/messages

Scroll to view the log information containing info
tail -f /var/log/messages | grep info
Copy the code

There is also an uppercase argument F for the tail command. This parameter can monitor the recreated file. For example, logs such as some log4j logs are rolled by the day and tail -f cannot detect this change.

4. Find files

Consider the following scenario. We need to find a file called decorator.py, which is a ghost and could be anywhere on the system. The find command is competent for this ghost hunt.

We use the find command to start from the root directory. Due to the number of files on the system, the following command may take a while.

[root@localhost site-packages]# find / -name decorator.py -type f/ usr/lib/python2.7 / site - packages/decorator. PyCopy the code

Using the time command, you can view the specific execution time. Execution is still pretty fast! Seconds!

[root@localhost site-packages]# time find / -name decorator.py -type f
/usr/lib/python2.7/site-packages/decorator.py

real	0m0.228s
user	0m0.098s
sys	0m0.111s
Copy the code

The find command finds a collection of paths. Usually after the query is queried, additional processing is performed, usually in conjunction with the Xargs command (xargs reads the input and processes it line by line). Forget it, it doesn’t work!

Delete all class files in the current directory
find . | grep .class$ | xargs rm -rvf

Find the file /root accessed the day before
find /root  -atime 1 -type f

Find files that have been updated within 10 minutes
find /root  -cmin  -10

Find the file belonging to user root
find /root -user root

Delete files that are larger than 1MB
find /root -size  +1024k  -type f | xargs rm -f
Copy the code

Find has a very large number of parameters. Except for common ones, you can run the man command to view all of them. The operation of man is also very similar to that of vi. Type /EXAMPLES to see various EXAMPLES. However, I think the commands listed above are more applicable.

4.1 Data sources

In the figure above, you see words like mtime,ctime,atime. Where do they come from? The next logical step is to look at the stat command.

[root@localhost ~]# stat springFile: 'Spring' Size: 78888897 Blocks: 154080 IO Block: 4096 Regular File Device: FD00h /64768d Inode: 8409203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:admin_home_t:s0 Access: 2019-11-04 18:01:46.698635718-0500 Modify: 2019-11-04 17:59:38.823458157-0500 Change: What will we do as a BirthCopy the code

Isn’t that a file property? Everything from the file size, to the file type, and even the last modification, access time, can be obtained from here. Linux file systems store information in blocks, and to find out where a file resides in storage space, each file is indexed by an I-node (inode), which you can think of as a pointer to a file.

  • The number of bytes of the file
  • File owner User
  • Group to which the file belongs
  • Read, write, and execute permissions on files
  • The timestamp of the file
    • Ctime indicates the time when the inode last changed
    • Mtime indicates the time when the file content was last changed
    • Atime Indicates the time when the file was last opened.
  • The number of links, i.e. how many file names point to the inode (ln command)
  • Location of file data block (specific data location)

Inode is a big topic and an important one. If you are interested, you can search for it yourself. We just need to know where the information came from.

4.2. Small exercises

If I just want to get Modify, I can use a combination of the commands I learned above. First get the last three rows, then get the first row. The effect is as follows:

[root@localhost ~]# stat spring | tail -n 3 | head -n 1The Modify: the 2019-11-04 17:59:38. 823458157-0500Copy the code

The following commands have the same effect as above and the output is exactly the same. As the saying goes, all roads lead to Rome. Next, let’s first introduce grep, which appears frequently. In addition, we in the above these commands, use | for many times, this is very important in Linux pipeline concept, the following will also be introduced.

stat spring | head -n 7 | tail -n 1
stat spring | grep Modify
stat spring | sed -n '7p'
stat spring | awk 'NR==7'
Copy the code

5. String matching

Grep is used to filter content. Color can be printed on supported terminals with the –color parameter. N is used to output the specific number of lines for fast location. This is a command that must be practiced.

For example, view POST requests in nginx logs.

grep -rn --color POST access.log
Copy the code

It is recommended to use this parameter every time.

If I want to see the context before and after an exception, I can use the ABC parameter. They are abbreviations of several words and are often used.

  • A after n lines after the content
  • A) before B) before
  • C N lines before and after the content

Something like this:

# look at the first 2 and last 10 lines of the Exception keyword
grep -rn --color Exception -A10 -B2   error.log

Find all import keywords in /usr/, the file they are in, and the number of lines
grep -rn --color import /usr/
Copy the code

6, pipes,

In the above command, we use many times to |, it seems can do some amazing things. | is the meaning of the pipe, it can relate multiple commands. In general, commands have the following associations:

  • ;Sequential execution, for examplemkdir a; rmdir a
  • &&Conditional execution, e.gmkdir a && rmdir a
  • ||Conditional execution, e.gmkdir a || rmdir aThe following commands will not be executed
  • |Pipes, the output of the previous command, will serve as the input of the following command

The first three types of command correlation are very simple and logical, and very easy to understand. Pipes, on the other hand, have their own characteristics.

Anyone who has worked with a programming language knows the concepts of STdin, STdout, and Stderr. Let’s reorganize the definition for pipes: the output of the previous command (stdin) will be the input of the following command (stdout).

Let’s take a line of command to illustrate.

seq 20 100 | head -n 50 | tail -n 1
Copy the code

The command above will print 69. 69 is the magic number. How did you get there? Let’s take a little picture, and it all falls into place.

Linux uses a number to abbreviate input, output and errors, which is often used in some scripts, and even in some installation files.

  • 0 indicates stDIN standard input
  • 1 indicates stDout standard output
  • 2 indicates stderr standard error

Error messages can be directed to standard output with syntax similar to 2>&1. Let’s prove it with a command.

Error message cannot be output to file
[root@localhost ~]# cat aaaaaaaaa > b
cat: aaaaaaaaa: No such file or directory
[root@localhost ~]# cat b

The error message was redirected
[root@localhost ~]# cat aaaaaaaaa > b 2>&1
[root@localhost ~]# cat b
cat: aaaaaaaaa: No such file or directory
Copy the code

7, sorting,

Now that you know how pipes work, you can introduce the sort command. It can usually be combined with uniq to perform sorting and de-weighting operations. Run the cat command to generate a file with the following content.

cat > sort.txt <<EOF
1 11
3 22
2 44
4 33
5 55
6 66
6 66
EOF
Copy the code

Let’s put these two commands on stage. Sort can use -t to specify delimiters and -k to specify columns to sort. But space, is not required to do these additions to the specified.

Sort by the first column in reverse order
[root@localhost ~]# cat sort.txt | sort -n -k1 -r
6 66
6 66
5 55
4 33
3 22
2 44
1 11

# count the number of occurrences of each row and sort it in reverse order
The number of rows has changed from 7 to 6
[root@localhost ~]# cat sort.txt | sort | uniq -c | sort -n -k1 -r2 6 66 1 5 55 1 4 33 1 3 22 1 2 44 11 11Copy the code

Note: the uniq command is usually used on sorted result sets. Therefore, in many cases, you need to use the sort command first and then the uniq command. Beginners often forget the first step, causing commands to fail to run properly.

8, summary

In this section, we start with file properties and look at some common commands for manipulating files. And incidentally introduced the concept of pipeline. Now, let’s practice.

Find all of them in the systemgrub.cfgFile, and prints its line count.

Analysis: First you need to use the find command to find these files. Xargs is then used for line-by-line processing. Finally, use the wc command to count the exact number of lines.

[root@localhost grub2]# find / | grep grub.cfg | xargs wc -l
141 /boot/grub2/grub.cfg
Copy the code

The system group list is displayed

cat /etc/group | awk -F ':' '{print $1}'
Copy the code

The following command outputs the IP of the nginx log and the PV of each IP, the top 10 pv

# The 2019-06-26 T10:01:57 + 08:00 | nginx001. Server. Ops. Pro. Dc | 100.116.222.80 | 10.31.150.232:41021 | | | | 0.000 0.011 0.014 200 | 200 | 273 | | / visit | sign = 91 cd1988ce8b313b8a0454a4bbe930df | | - | - | HTTP POST | 112.4.238.213

awk -F"|" '{print $3}' access.log | sort | uniq -c | sort -nk1 -r | head -n10
Copy the code

9. Think & expand

1, Linux terminal, is how to achieve color text? How do I output a green Hello World?

2. What is the difference between soft links and hard links?

3. Know a few commands that are biased but not too biased.

  • cutWith AWk, it’s almost impossible to use cut
  • tr
  • col
  • paste
  • join
  • split

End

Wet eyes, sad crotch.

Perhaps, conservative is the biggest pain of wang, is because is also the fruit, is the fruit hide however. This character determined him from the start, and by his 30s, he was still alone.

The world is all knowledge, human lian up to the article. In the next section, I will take Xiao Wang to understand: all roads lead to Rome, even the iron tree can blossom.

Xjjdog is a public account that doesn’t allow programmers to get sidetracked. Focus on infrastructure and Linux. Ten years architecture, ten billion daily flow, and you discuss the world of high concurrency, give you a different taste. My personal wechat xjjdog0, welcome to add friends, further communication.

4 sort 4 uniq 7 tail 5 less 3 more 3 stat 3 head 5 find 7 grep 6 cat 6 seq 2 tr 1 col 1 paste 1 join 1 split 1 awk 9