Original: Coding diary (wechat official ID: Codelogs), welcome to share, reprint please reserve the source.

Introduction to the

There are many auxiliary development of small tools on the Internet, such as Base64, MD5 and so on, but these small tools can be basically realized with Linux commands, that is convenient and efficient.

Viewing special Characters

I put this in the first place because it’s so important that programs often get bugged by special, invisible characters, so you should always look for special characters in unexpected string handling that should have performed correctly.

$ echo 'hello'|sed -n 'l'
hello$
$ echo 'hello'|cat -A
hello$
$ echo 'hello'|od -c
0000000   h   e   l   l   o  \n
0000006
Copy the code

Echo adds a newline character to the output.

Viewing bytes

This command can be useful when a database field is too long, such as oracle’s varchar2(64) type, which means 64 bytes at most, and the string inserted by the business is a mix of Chinese and English, which can be tricky to estimate.

$ echo -n 'hello, zhang SAN'|iconv -t utf-8|wc -c
12
Copy the code

Timestamp conversion

# date string to timestamp
$ date -d 'the 2020-06-08 00:13:28' +%s
1591546408
# timestamp to date string
$ date -d '@ 1591546408' +"%F %T %z"The 2020-06-08 00:13:28 + 0800Time zones are very important. Be aware of them at all times
TZ='GMT-8' date -d 'the 2020-06-08 00:13:28' +%s
TZ='GMT-8' date -d '@ 1591546408' +"%F %T %z"
Copy the code

CSV get json

# Use csvJSON in csvKit
csvjson -y0 -I data.csv
# using python
python -c "import csv,json; print(json.dumps(list(csv.reader(open('data.csv')))))"
Copy the code

File server

python3 -m http.server 8000
Copy the code

urlencode,base64,md5,sha256

We use these a lot in development.

# urlencode is recommended as an alias
$ alias urlencode='python3 -c "import sys; from urllib import request as rq; print(rq.quote(sys.argv[1],safe=\"\"))"'$urlenCode Hello %E4%BD%A0%E5%A5%BD# urldecode, which is recommended as an alias
$ alias urldecode='python3 -c "import sys; from urllib import request as rq; print(rq.unquote(sys.argv[1]))"'
$ urldecode '%E4%BD%A0%E5%A5%BD'helloBase64 codec
$ echo hello |base64 -w0
aGVsbG8K
$ echo aGVsbG8K|base64 -d
hello

# md5 digest
$ echo hello |md5sum
b1946ac92492d2347c6235b4d2611184  -

# sha256 in this paper,
$ echo hello | sha256sum
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  -

# hmacWithSha256 the
$ echo hello | hmac256 'secret'
171b5670f7b4037fb90bef773b022130e48100fdd40ea023730097da9a68f4ff

Copy the code

Json formatting

$ echo '{"id":1,"name":"lisi"}' | jq .
{
  "id": 1,
  "name": "lisi"
}
Copy the code

The characters escape and unescape

# change "to \"
$ echo 'hi,"lisi"' | jq -R 'tojson' -r
"hi,\"lisi\""
# convert \" to"
$ echo '"hi,\"lisi\""' | jq -R 'fromjson' -r
hi,"lisi"
#json property value is json content, I don't know if you have encountered this bad design
$ echo '{"code":200,"data":"{\"id\":1,\"name\":\"lisi\"}"}' | jq '.data=(.data|fromjson)'
{
  "code": 200,
  "data": {
    "id": 1,
    "name": "lisi"}}Copy the code

unicode

When developing, you often need to switch between Chinese characters and the \u4f60 form, so the following commands are useful.

# unicode decoding
$ echo -e '\u4f60\u597d'Hello $echo $'\u4f60\u597d'Hello $echo '\u4f60\u597d'|sed 's/\\u//g'| XXD - r - ps | iconv -f ucs - 2 be hello# unicode
$ echo -n 'hello'|iconv -t ucs-2be|xxd -ps|sed -E 's/.{4}/\\u&/g'
\u4f60\u597d

Convert the file to UTf8, which is also useful
enconv -L zh_CN -c -x UTF-8 file.txt
Copy the code

Generate random passwords

Generate random passwords
$ openssl rand -hex 16
fb9f47a7ebad6bd77be332d6b3a0bc0b

$ cat /dev/urandom |head -c 16|xxd -ps
eb297181cad546210a00118d543b78bf

$ cat /dev/urandom |tr -dc A-Za-z0-9  |head -c 32
V1NB2Oc1mCJ3mNfofDZCQB68dRde30Xz

Generate a random number up to 10
awk -v b=$(cat /dev/urandom|tr -dc 0-9|head -c 9) 'BEGIN{srand(b); print int(rand()*10)}'
cat /dev/urandom|head -c 4|od -An -t u4 --endian=big|awk '{print int($1/2^32*10)}'

# uuid generated
$ uuidgen
7b45c1c2-0533-45e5-9903-802ee58b6638
Copy the code

Change the IP address to a number

As we all know, an IP address is actually a 4-byte number, so you can save space if you store the IP as a number in the database.

# the IP Numbers
$ echo192.168.0.1 | tr.'\n'|xargs printf "%02X"|xxd -r -ps|od -An -t u4 --endian=big
 3232235521
# Digital to IP
$ printf "%08X" 3232235521|xxd -r -ps |od -An -t u1
 192 168   0   1
Copy the code

Hexadecimal conversion

#printf can convert decimal to hexadecimal
$ printf "%08X" 3232235521
C0A80001

# BC can implement interconversion between any base
$ echo C0A80001|sed 's/^/obase=10; ibase=16; /g'|bc
3232235521

$ echo C0A80001|sed 's/^/obase=2; ibase=16; /g'|bc
11000000101010000000000000000001
Copy the code

Check the ASCII

# Look directly at the MAN document
man ascii 
# Use od to check
printf "% 0.2 X"{0.. 127}| xxd -r -ps | od -t x1d1caCopy the code

Search content

Tac is often more useful than CAT when searching logs, such as the last 10 exceptions we generated.

tac app.log|grep -iw -B 20 -m 10 'exception'|tac 
Copy the code

In addition, use grep -rn to search first when you don’t know what you’re looking for in the file in the current directory, such as forgetting where tomcat’s port is configured.

grep -rn -w 8080 . 
Copy the code

Viewing Process Logs

When you enter an unfamiliar server environment and don’t know where the log files generated by Java processes are, it is better to find them yourself than to ask others.

# use lsof
pid=`pidof java`
lsof -p $pid|grep .log$
#lsof not available, try this
ls -l /proc/$pid/fd |grep .log$
Copy the code

Reverse shell

Sometimes you need help with troubleshooting, but you don’t want to give out your server password, you can give someone a reverse shell.

Get shell end
socat file:`tty`,rawer TCP-LISTEN:9999,bind= 0.0.0.0 reuseaddr, keepalive, keepcnt = 3, keepidle = 600, keepintvl = 600, pf = ip4# give the shell end
nohup socat system:'stty rows 63 columns 207; bash -li'Pty, stderr setsid, sigint, sane, ctty TCP: 192.168.0.1:9999 &Copy the code

The statistical number of rows

# count rows
wc -l
sed -n '$='
Group the number of rows
uniq -c
Copy the code

Generate continuous time slice

Generate continuous time slice, generally used in scripts, for example, to query the amount of data in a year. When the amount of data is very large, the direct query is unable to produce the results. At this time, you can split the year into 1 day, so that the query 1 day run. Mysql > create SQL (1 year, 1 day, 1 day);

generate_day_query_sql(){
    fmt='select count(*) from order where create_time >= "%s" and create_time < "%s"; \n';
    seq 0 The $1 \
        |xargs -i date -d "2021-01-01 00:00:00 {} days" +'%F %T' \
        |sed '1! {$! p}' \
        |paste - - \
        |awk -F'\t' -v fmt="$fmt" '{printf fmt,$1,$2}'
}
generate_day_query_sql 365 | mysql -vvv
Copy the code

Pv displays progress and speed control

The pv command is the result of a script that has been running for a long time and it can be very frustrating not to see progress. For example, the cp command does not display the progress function itself, and copies large files in a dazed manner. In this case, the pv command can be used instead.

pv file1 > file2 
Copy the code

In the preceding section, you only need to add the pv command before the mysql command to view the execution progress of the script for querying one-year data volume.

# -l counts the number of rows that flow through PV, which by default counts bytes
# -s specifies the total number of lines, so that each line of text flows through pv, and PV calculates the current progress
generate_day_query_sql 365 | pv -l -s 365 | mysql -vvv
Copy the code

Pv can also be used to control speed so that scripts don’t hang up on the database.

# -l 2 indicates that text flows at a speed of no more than 2 lines per second
generate_day_query_sql 365 | pv -l -s 365 -L 2 | mysql -vvv
Copy the code

Contrast table structure

In fact, it is difficult to keep the same table structure for databases in the same system and different environments. When it is necessary to find out their differences, try not to compare fields one by one, because it is too inefficient.

Export all table names in db1
echo 'show tables' | mysql --skip-column-names -D db1 > table_names.txt
Export the table structure of DB1 and DB2
cat table_names.txt | sed 's/.*/show create table &; / ' | tee >(mysql -D db1 > db1_tables.txt) >(mysql -D db2 > db2_tables.txt)
# restore line breaks
sed -i 's/\\n/\n/g' db1_tables.txt db2_tables.txt
Compare table structures
icdiff db1_tables.txt db2_tables.txt
Copy the code

conclusion

You can add these utility commands as Linux aliases, and once you get used to them, you’ll find you can’t live without them.

Content of the past

Linux parallel command parallel command parallel command parallel command parallel command