Build a productive work environment – Shell

Note: This paper is made up of
Mr Leven(Javaer/Emacser) and I edit together, so the copyright of the article belongs to Lei Jun and me, the rebearer must give the source and us two authors. The original article was first posted on the wechat public account of Kushell, and then I made some modifications and posted to the blog.

Programmers are lazy people who want code to do the work for them, and they are constantly automating things in their working lives with code, making society more and more efficient. Therefore, when programmers are ready to optimize the world, they must first optimize their working environment. It is said that “if you want to do good work, you must first sharpen your tools”.

It’s up to each of us to create an environment that makes us more productive. Even having one less command to type, one less key to press, and one less mouse to keyboard toggle makes the programmer’s job much more efficient. Therefore, programmers generally need a good performance, will not be opened too many web pages or programs will be jammed, but also equipped with multiple monitors, a monitor to write code, a document, a test results, without having to switch back and forth in a variety of Windows…… Switching between a large number of Windows is often confusing and error prone (confusing lines or test environments)…

In addition to hardware equipment, software is also where programmer productivity can be increased. One of the most important things to increase programmer productivity at the software level is the command line and script. Using the mouse and graphical interface can greatly reduce programmer productivity. Cool Shell has written a few before, such as Shells You May Not Know and Linux Tips you Should Know, but the Unix/Linux Shell is a treasure trove of things to write about, otherwise there would be no “Where is the Shell, there is a way”.

The command line

There are great command-line tools available on different operating systems, such as Iterm2 on Mac, native command-line on Linux, and if you’re working on Windows, it’s not a problem because WSL is now available on Windows. WSL provides a Linux-compatible kernel interface developed by Microsoft (without Linux kernel code) on which you can then run GNU user Spaces such as Ubuntu, openSUSE, SUSE Linux Enterprise Server, Debian and Kali Linux. Such user Spaces may include Bash shells and command languages, using native GNU/Linux command line tools (sed, AWk, etc.), programming language interpreters (Ruby, Python, etc.), and even graphical applications (using the X windowing system on the host side).

Using the command line, you can complete all the daily operations, such as new folder (mkdir), New file (touch), move (MV), copy (cp), delete (rm), and so on. And the best way to use the Linux/Unix command line is to use awk, sed, grep, xargs, find, sort, etc., and pipe them together to do what you want, especially some simple statistics. This is an incomparable advantage for the Linux command line. Such as:

• View the top10 IP addresses connecting to your server:

netstat -nat | awk '{print $5}' | awk -F ':' '{print $1}' | sort | uniq -c | sort -rn | head -n 10

• Check out your top 10 commands:

cat .bash_history | sort | uniq -c | sort -rn | head -n 10 (or cat .zhistory | sort | uniq -c | sort -rn | head -n 10

(Note: AWK and SED are two of the most powerful tools in the world, so I’ve written two previous articles about them: AWK Tutorial and SED Tutorial, which you can check out.)

Using alias on the command line can combine frequently used commands or complex commands into a single command, or modify native commands.

Here are a few commands you probably type every day. So, you should set it to Alias for efficiency

alias nis="npm install --save " alias svim='sudo vim' alias mkcd='foo(){ mkdir -p "$1"; cd "$1" }; foo ' alias install='sudo apt get install' alias update='sudo apt-get update; sudo apt-get upgrade' alias .. ="cd .." alias ... ="cd .. ; cd .." alias www='python -m SimpleHTTPServer 8000' alias sock5='ssh -D 8080 -q -C -N -f [email protected]'Copy the code

You can also refer to the following articles to see how others use aliases well

• 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X
• What are your favorite bash aliases?
• 23 Handy Bash Shell Aliases For Unix, Linux, and Mac OS X
• A few more of my favorite Bash aliases

In addition to native commands, there are many tools on the command line that can enhance the user experience. Here is a list of some great commands that make the native commands more powerful:

• fasdTo enhance thecdCommand.
• batTo enhance thecatCommand. If you want to have syntax highlightedcatYou can tryccatCommand.
• exaTo enhance thelsCommand, if you need to browse various files in many directories,rangerCommands can be compared tocd 和 catMore efficient and can even preview images in your terminal.
• fdIs a ratiofindSimpler and faster commands, it will also automatically ignore some of your configurations in.gitignoreFiles in, as well as.gitThe next file.
• fzfWill be a very useful document search artifact, its main is to search the current directory of the following files, can also be usedfzf --preview 'cat {}'Browse content while searching for files.
• grepIt’s an ancient artifact, however,ack,ag 和 rgIs better grep, and abovefdAlso, when a recursive directory matches, it will use your configuration in.gitignoreRules in.
• rmIs a dangerous command, especially of all kindsRm - rf..., so,trashIs a better delete command.
• manCommands are easy to read documentation commands, but man’s documentation is sometimes too long, so you can try ittldrCommand to show you some examples from the document.
• If you want a graphical representationpingYou can tryprettyping 。
• If you want to search for previously typed commands, instead of Ctrl +R, you can use the enhanced version of HSTR.
• Htop is an enhanced version of Top. However, there are also many kinds of top, such as IOTOP for viewing IO load, IFTOP for network load, and Atop, which integrates all of them together.
• Ncdu is much easier to use than DU. Another option is NNN.
• If you want to record your command line operations as an SVG GIF, you can try asciinema and SVG-trem.
• httpieIs one that can be used insteadcurl 和 wgetHTTP client,httpieJson and syntax highlighting are supported for HTTP access using simple syntax:http -v github.com.
• Tmux can be useful when you need to log in to a remote server frequently to work, maintain a remote login session, and view the status of multiple shells in a single window.
• Taskbook is a task manager that can be used entirely from the command line, with support for ToDo management and the ability to prioritize each task.
• SSHRC is a magic tool that allows you to log in to a remote server using the rc configuration of the native shell.
• Goaccess is a lightweight tool for analyzing statistical log files, mainly analyzing various access logs.

The main reference for these additional commands is from the following articles

1. 10 Tools To Power Up Your Command Line
2. 5 More Tools To Power Up Your Command Line (Part 2 Of Series)
3. Power Up Your Command Line, Part 3
4. Power Up Your Command Line
5. Hacker Tools

Shell and scripting

A shell is a text interface that can efficiently interact with a computer. The shell provides a set of interactive programming languages (scripts). There are many types of shells, such as sh, bash, ZSH, and so on.

Shell has a strong vitality. Nowadays, various high-level programming languages are popular, many tasks are still inseparable from shell. For example, you can use the shell to perform some compilation tasks, or do some batch tasks, initialize data, package programs, and so on.

A popular one is ZSH + Oh-my-Zsh + Zsh-autoSuggestions, which you can also try. ZSH and oh-my-zsh are routine operations, but zsh-AutoSuggestions is a super quick way to complete commands you type, making command line operations more efficient.

Fish is also another awesome shell. For example, command line autocomplete (based on history), command line commands are highlighted, and when you enter command line parameters, it automatically tells you what parameters are available… Fish is also fun to use in many places. It’s kind of on par with the oh-my-Zsh up there.

You may argue that scripting in Python or PHP is better and less buggy than Shell, but HERE’s my defense:

• For one thing, if you ever need to maintain an online machine or a bank user’s system (completely isolated from the outside world, and without Python/PHP or their advanced libraries installed on the server, then Shell is all you have to use).
• Second, and if you need to interact a lot with the command line, the Shell is the best choice. Imagine going to 100 remote machines and checking the Access.log for an error.

So, learn to use only traditional grep/awk/sed and other native POSIX default installation commands.

Of course, it’s not easy to write a script well, so here are some small templates for you to consider:

A sample of processing command line arguments

while [ "$1" != "" ]; do
    case $1 in
        -s  )   shift  
        SERVER=$1 ;;  
        -d  )   shift
        DATE=$1 ;;
    --paramter|p ) shift
        PARAMETER=$1;;
        -h|help  )   usage # function call
                exit ;;
        * )     usage # All other parameters
                exit 1
    esac
    shift
doneCopy the code

A sample command line menu

#! /bin/bash # Bash Menu Script Example PS3='Please enter your choice: ' options=("Option 1" "Option 2" "Option 3" "Quit") select opt in "${options[@]}" do case $opt in "Option 1") echo "you chose choice 1" ;; "Option 2") echo "you chose choice 2" ;; "Option 3") echo "you chose choice $REPLY which is $opt" ;; "Quit") break ;; *) echo "invalid option $REPLY";; esac doneCopy the code

${Blu}blue ${Red} Red ${RCol}etc…. Outputs colored text

RCol='\e[0m' # Text Reset # Regular Bold Underline High Intensity BoldHigh Intens Background High Intensity Backgrounds Bla='\e[0;30m'; BBla='\e[1;30m'; UBla='\e[4;30m'; IBla='\e[0;90m'; BIBla='\e[1;90m'; On_Bla='\e[40m'; On_IBla='\e[0; 100m'; Red='\e[0;31m'; BRed='\e[1;31m'; URed='\e[4;31m'; IRed='\e[0;91m'; BIRed='\e[1;91m'; On_Red='\e[41m';  On_IRed='\e[0;101m'; Gre='\e[0;32m'; BGre='\e[1;32m'; UGre='\e[4;32m'; IGre='\e[0;92m'; BIGre='\e[1;92m';  On_Gre='\e[42m'; On_IGre='\e[0;102m'; Yel='\e[0;33m'; BYel='\e[1;33m'; UYel='\e[4;33m'; IYel='\e[0;93m'; BIYel='\e[1; 93m'; On_Yel='\e[43m'; On_IYel='\e[0;103m'; Blu='\e[0;34m'; BBlu='\e[1;34m'; UBlu='\e[4;34m'; IBlu='\e[0;94m';  BIBlu='\e[1;94m'; On_Blu='\e[44m'; On_IBlu='\e[0;104m'; Pur='\e[0;35m'; BPur='\e[1;35m'; UPur='\e[4;35m'; IPur='\e[0; 95m'; BIPur='\e[1;95m'; On_Pur='\e[45m'; On_IPur='\e[0;105m'; Cya='\e[0;36m'; BCya='\e[1;36m'; UCya='\e[4;36m';  ICya='\e[0;96m'; BICya='\e[1;96m'; On_Cya='\e[46m'; On_ICya='\e[0;106m'; Whi='\e[0;37m'; BWhi='\e[1;37m'; UWhi='\e[4; 37m'; IWhi='\e[0;97m'; BIWhi='\e[1;97m'; On_Whi='\e[47m'; On_IWhi='\e[0;107m';Copy the code

Dirname $(readlink -f $0)

FILE="$0"
while [[ -h ${FILE} ]]; do
    FILE="`readlink "${FILE}"`"
done
pushd "`dirname "${FILE}"`" > /dev/null
DIR=`pwd -P`
popd > /dev/nullCopy the code

How do I run a local script on a remote server

SSH user@server 'bash-s' < local.script.sh # SSH user@server ARG1=" ARG1 "ARG2=" ARG2"' bash-s' < local_script.shCopy the code

How to check if a command exists, using which? It’s best not to use it, because many operating systems don’t have an exit status code so you don’t know if it exists. So, you should use the following method.

# POSIX compatible: command -v <the_command> Hash <the_command> type <the_command> # Example: gnudate() {if hash gdate 2> /dev/null; then gdate "$@" else date "$@" fi }Copy the code

Then, if you want to write more robust scripts, here are some tips:

• use-eParameters, such as:set -eor#! /bin/sh -e, so that your script will stop running in case of an error, this will prevent your script from continuing to scramble in case of an error.
• use-uParameters, such as:set -euThat means that if you have a variable in your code that is not defined, it will exit.
• For some variations, you can use the default values. Such as:${FOO:-'default'}
• Handle the exit code for your code. This makes it easy to integrate your script with other command lines or scripts.
• Try not to use;To execute multiple commands instead of using&&This will stop running subsequent commands in the event of an error.
• For some string variables, use quotation marks to avoid Spaces or other weird characters.
• If your foot has parameters, you need to check that the script runs with the parameters you want, or that your script can safely run without parameters.
• Set for your script-h 和 --helpTo display help information. Never use these two parameters as a function.
• useThe $()Instead of “to get the output from the command line, mainly because it’s easy to read.
• Beware of different platforms, especially MacOS and Linux cross-platform.
• forrm -rfFor high-risk operations like this, you need to check whether the following variable name is empty, for example:rm -rf $MYDIDR/*if$MYDIRIs empty, and the result is disastrous.
• Consider using “find/while” instead of “for/find”. Such as:for F in $(find . -type f) ; do echo $F; donewrittenfind . -type f | while read F ; do echo $F ; doneNot only does it tolerate whitespace, it’s faster.
• Defensive programming, checking for relevant things, such as the existence of files and directories, before executing commands.

Finally, some Shell and script references are recommended.

All kinds of interesting command assembly, a line of command to go:

• www.bashoneliners.com/
• www.shell-fu.org/
• www.commandlinefu.com/