hashub

Super Brother Introduction

Recommended Reading
https://effective-shell.com

1. Basics#

The role of shell is

• To interpret and execute user input commands or programs, etc.
• The user inputs a command, and the shell interprets one command at a time.
• The way Linux responds to keyboard input commands is called interactive.

The role of shell_1

The shell is a layer that wraps around the system core, positioned at the outermost layer of the operating system, directly interacting with the user. It interprets the user's input to the operating system and then processes the output results from the operating system, displaying the results on the screen for the user to see.

From the moment we log into Linux, inputting our username and password until we enter the Linux interactive interface, all operations are handed over to the shell for interpretation and execution.

The role of shell_2

1.1 What is a shell script#

When commands or program statements are written in a file, executing the file reads the code within, and this program file is called a shell script.

In a shell script, multiple Linux commands and loop control statements are defined, and then these Linux commands are executed all at once. The method of executing the script file is called non-interactive mode.

In Windows, there are *.bat batch processing scripts.

In Linux, commonly used *.sh script files.

Shell script rules

In the Linux system, shell scripts (or bash shell programs) are usually edited with vim and consist of Linux commands, bash shell instructions, logical control statements, and comment information.

Shebang#

In computer programs, a shebang refers to the first two characters #! that appear at the beginning of a text file.

In Unix systems, the program analyzes the content after the shebang as an interpreter directive, for example:

• A file starting with #! /bin/sh will call /bin/sh, which is the bash interpreter, when executed.
• A file starting with #!/usr/bin/python specifies that the python interpreter should be used for execution.
• A file starting with #!/usr/bin/env followed by the interpreter name is a way to correctly find the interpreter across different platforms.

Notes:

• If the script does not specify a shebang, the current shell will be used to interpret the script by default, i.e., $SHELL.
• If the shebang specifies an executable interpreter, such as /bin/bash or /usr/bin/python, the filename will be passed as an argument to the interpreter during execution.
• If the interpreter specified by #! does not have executable permissions, it will report an error "bad interpreter: Permission denied".
• If the interpreter specified by #! is not an executable file, the specified interpreter will be ignored, and the script will be executed by the current SHELL.
• If the interpreter specified by #! does not exist, it will report "bad interpreter: No such file or directory".
• The interpreter after #! must be written with its absolute path (e.g., #!/bin/bash); it will not automatically search for the interpreter in $PATH.
• If you use a command like bash test.sh to execute the script, the #! line will be ignored, and the interpreter will be the bash explicitly specified in the command line.

#! /bin/bash

# Date: xxx
# Author: xxx
# Blog: xxx

Shell scripting is well-suited for handling plain text data, and the philosophy of Linux is that everything is a file, such as logs, configuration files, text, web files, most of which are plain text types. Therefore, shell can conveniently process text, akin to the powerful Linux trio (grep, sed, awk).

Script Language#

Shell scripting is a weakly typed language that does not require variable type declaration; it can be defined and used directly.

Strongly typed languages must first define variable types, determining whether they are numbers, strings, etc., before assigning values of the same type.

# Default shell in Linux
[root@localhost ~]# echo $SHELL
/bin/bash

# Supported shells in Centos7
[root@localhost ~]# cat /etc/shells
/bin/sh
/bin/bash
/usr/bin/sh
/usr/bin/bash
/bin/tcsh
/bin/csh

Default sh interpreter

[root@localhost ~]# ll /usr/bin/sh
lrwxrwxrwx. 1 root root 4 Jul 18 19:54 /usr/bin/sh -> bash

1.2 bash#

History commands

[root@localhost ~]# echo $HISTSIZE
1000

[root@localhost ~]# echo $HISTFILE
/root/.bash_history

history
-c Clear history
-r $HISTFILE Restore history

[root@localhost ~]# history -c
[root@localhost ~]# history
    1  history
[root@localhost ~]# history -r ~/.bash_history

!History ID  # Quickly execute a history command
!!            # Execute the last command

Bash features

• File path tab completion
• Command completion
• Shortcut keys ctrl + a, e, u, k, l
• Wildcards
• Command history
• Command aliases
• Command line expansion

1.3 Shell Variables#

Variable definition and assignment; note that == there must be no spaces between the variable and the value ==

• Variable types: bash treats all variables as strings by default.
• Bash variables are weakly typed, requiring no prior declaration of type; declaration and assignment occur simultaneously.

Variable substitution/referencing

name="This is test!"
echo ${name}
echi $name # Braces can be omitted

Single-quoted variables do not recognize special syntax.

Double-quoted variables can recognize special symbols.

Variable name rules

• Names should be meaningful and follow rules; do not reference reserved keywords (help check reserved words).
• Can only contain numbers, letters, and underscores.
• Cannot start with a number.
• Cannot use punctuation.
• Variable names are case-sensitive.

Variable scope

• Local variables only apply to the current shell process.

pstree Check process tree

Environment variables: also known as global variables, apply to the current shell and any of its subprocesses. Environment variables can be divided into custom and built-in types.

Local variables: defined within shell functions or shell scripts.

Positional parameter variables: used for parameters passed to shell scripts.

Special variables: built-in special effect variables in shell.

• $?
  • 0 : Success
  • 1 - 255 : Error code

Custom variables

• Variable assignment: varName=value
• Variable referencing: ${varName}, $varName
• "", the variable name will be replaced with the variable value.

Single-quoted variables do not recognize special syntax.

Double-quoted variables can recognize special symbols.

1. Each time the bash/sh interpreter executes a script, a subshell is opened, so the current shell variables are not retained. Check the process tree with the pstree command.

2. Calling source or the dot symbol loads the script in the current shell environment, thus retaining variables.

# 1. Execution method that opens a subshell
cat make_vars.sh
name="test"

echo $name
aa

bash make_vars.sh

echo $name
aa

# 2. Execution method that does not open a subshell

`Linux command`
Backticks

echo dir=`ls`
echo $dir
xxxx  xxx 

Shell variable interview question

cat test.sh
user1=`whoami`

sh test.sh

echo $user1
B

A. Current user
B. Empty

1.4 Environment Variable Settings#

Environment variables generally refer to variables exported using the export built-in command, used to define the shell's runtime environment and ensure the correct execution of shell commands.

The shell determines the logged-in username, PATH, file system, and various applications through environment variables.

Environment variables can be temporarily created in the command line, but they will be lost when the user exits the shell terminal. To make them permanent, the environment variable configuration file needs to be modified.

• User personal configuration files: ~/.bash_profile, ~/.bashrc, specific files for remote login users.
• Global configuration files: /etc/profile, /etc/bashrc, and it is recommended to create them in /etc/profile.d/ rather than directly modifying the main file, as modifying the global configuration file affects all users logging into the system.

Commands to check system environment variables

• set: All variables in the current shell, including global variables, local variables, and functions.
• env: Only displays global variables.
• declare: Outputs all variables, similar to set.
• export: Displays and sets environment variable values.

Revoke environment variables

• unset variable_name, deletes the variable or function.

Set read-only variables

• readonly, read-only variables become invalid only when the shell ends.

System reserved environment variable keywords

Bash has many built-in environment variables used to define the working environment of bash.

export | awk -F '[ :=]' '{print $3}'

Bash multi-command execution

ls /data/;cd /tmp/;cd /home;cd /data

Environment variable initialization and loading order

After ssh logging into Linux, the system starts a bash shell.

 /etc/profile: Global environment variable file, and collects shell settings from configuration files in the /etc/profile.d directory.

Read scripts from the /etc/profile.d directory.

Run $HOME/.bash_profile (user environment variable file).

Run $HOME/.bashrc.

Run /etc/bashrc.

1.5 Special Variables#

Special parameter variables

Shell's special variables are used in scripts and functions to pass parameters, with the following special positional parameter variables:

$0    Get the shell script filename and script path.
$n    Get the nth parameter of the shell script, where n is between 1 and 9, e.g., $1, $2, $9; for n greater than 9, write as ${10}; parameters are space-separated.
$#    Get the total number of parameters after executing the shell script.
$*    Get all parameters of the shell script; without quotes, it is equivalent to $@; with quotes "$*", it receives all parameters as a single string, "$1 $2...."
$@    Without quotes, it has the same effect as above; with quotes, it receives all parameters as independent strings, like "$1" "$2" "$3" ..., preserving spaces.

for_variables.sh

#! /bin/bash

for var in "$*"
do
    echo "$var"
done

for var in "$@"
do
    echo "$var"
done

bash for_variables.sh test ha 1 2 3 4 

Special status variables

$?    The return value of the last command execution status, 0 for success, non-zero for failure.
$$    The process ID of the current shell script.
$!    The PID of the last background process.
$_    The last parameter of the previously executed command.

man bash Search Special Parameters

echo
-n    Do not output a newline.
-e    Parse special characters in the string.

\n Newline
\r Carriage return
\t Tab
\b Backspace

printf

eval
Execute multiple commands
eval ls;cd /tmp

exec
Does not create a child process; after executing subsequent commands, it automatically exits.

expr
expr function_name

1.6 Shell Substring#

${variable}                Returns the variable value.
${#variable}               Returns the length of the variable, character length.
${variable:Offset}         Returns the characters after the Offset value of the variable.
${variable:Offset:length}  Extracts characters limited by length after Offset.

# Deletion
${variable#word}          Deletes the shortest matching word substring from the beginning of the variable.
${variable##word}         Deletes the longest matching word from the beginning of the variable.
${variable%word}          Deletes the shortest word from the end of the variable.
${variable%%word}         Deletes the longest matching word from the end of the variable.

# Replacement
${variable/pattern/string} Replaces the first matching pattern with string.
${variable//pattern/string} Replaces all patterns with string.

# Specified character content extraction
a*c Matches strings that start with a, have any number of characters in between, and end with c (case-sensitive).

seq -s ":" 10

time for n in {1...10000};do char=`seq -s "testtt" 100`;echo ${#char} &>/dev/null;done

time for n in {1...10000};do char=`seq -s "testtt" 100`;echo ${char}|wc -L &>/dev/null;done

time for n in {1...10000};do char=`seq -s "testtt" 100`;expr length "${char}" &>/dev/null;done

time for n in {1...10000};do char=`seq -s "testtt" 100`;echo ${char}|awk "{print length($0)}" &>/dev/null;done

When programming in shell, try to use built-in Linux commands, built-in operations, and built-in functions for the highest efficiency.

Minimize the use of pipe operations.

Batch rename files

for file_name in `ls *.jpg`;do mv $file_name `echo ${file_name//_finished/}` done;

1.7 Shell Extended Variables#

${parameter:-word}    If the parameter variable value is empty, return the word string.

${parameter:=word}    If the parameter variable is empty, the word replaces the variable value and returns its value.

${parameter:?word}    If the parameter variable is empty, the word is output as stderr; otherwise, the variable value is output. Used to return error information when a variable is empty.

${parameter:+word}    If the parameter variable is empty, do nothing; otherwise, return the word.

stdout 1

stderr 2

Applications: Data backup, deleting expired data.

find xargs
# Delete data expired for more than 7 days
find path -name "" -type "file_type" -mtime +7 | xargs rm -f

dir_path="/data/my_data"
find ${dir_path} -name '*.tar.gz' -type f -mtime +7 | xargs rm -f

find ${dir_path:=/data/my_data} -name '*.tar.gz' -type f -mtime +7 | xargs rm -f

1.8 Parent and Child Shell#

Parent and Child Shell_1

pstree

ps -ef
-e Lists information about all processes, similar to the -A option.
-f Displays UID, PID, PPID.

ps -ef --forest

Parent and Child Shell_2

Creating a process list (creating a child shell)

# Shell process list
(cd ~; pwd; ls; cd /tmp; ls)

Check if in a child shell environment

# If it is 0, it is executed in the current shell environment; otherwise, it is opened in a child shell.
echo $BASH_SUBSHELL

(cd ~; echo $BASH_SUBSHELL)

Child shell nesting

(pwd; echo $BASH_SUBSHELL)

(pwd; (echo $BASH_SUBSHELL))

Using parentheses to open the concept of a child shell, in shell script development, child shells are often used to handle multi-process tasks to improve program concurrency efficiency.

1.9 Built-in Commands, External Commands#

Built-in commands: Loaded into memory at system startup, resident in memory, more efficient but consume resources.

External commands: Users need to read program files from the hard disk and then load them into memory.

type cd

type ls

External commands feature: They will definitely open a child process for execution.

ps -f --forest

Built-in commands are part of the shell and do not need to be loaded from a separate file; they are loaded into memory after the system starts.

# View all built-in commands
compgen -b

Summary#

echo "username: ${USER}"
echo "UID: $UID"
echo "User home: " $HOME

${}        Retrieve variable value.
$()        Execute commands in parentheses and get the execution result.
``        Same as above.
()        Open a child shell to execute.
$vars     Retrieve variable value.

2. Shell Development#

2.1 Mathematical Operations#

Shell arithmetic operators

Common six methods of mathematical calculation in Shell

2.1.1 Double Parentheses#

$(())    # $ symbol must be present

# In .sh scripts, echo can be omitted
a=10
b=2
echo $(($a - $b))
echo $(($a > $b))

2.1.2 Calculation Script#

#! /bin/bash

print_usage() {
    echo "Please enter a number!!!"
    exit 1
}

read -p "Please enter the first number: " firstnum
if [ -n "`echo` $firstnum | sed 's/[0-9]//g'" ]
  then
    print_usage
fi 

if [ "${operator}" != "+" ] && [ "${operator}" != "-" ] && [ "${operator}" != "*" ] && [ "${operator}" != "/" ]
  then
    echo "Only + - * / are allowed"
    exit 2
fi

read -p "Please enter the second number: " secondnum
if [ -n "`echo` $secondnum | sed 's/[0-9]//g'" ]
  then
    print_usage
fi

echo "${firstnum}${operator}${secondnum} Result is: $((${firstnum}${operator}${secondnum}))"

2.1.3 Check if nginx service is running#

while true;
  do
    wget --timeout=${timeout} --tries=1 http://xxx.com/ -q -O /dev/null
    if [ $? -ne 0 ]
      then
        let fails+=1
      else
        let success+=1
     fi
     
     if [ $success -ge 1 ]
       then
         echo "Normal"
         exit 0
     fi
     
     if [ ${fails} -ge 2 ];then
       echo "error"
       exit 2
     fi
done

-ne        Not equal
-ge        Greater than or equal to

In vim, locate another parenthesis with shift + %.

2.1.4 expr#

expr 1 + 2

# Pattern matching
expr aa.png ":" ".*"

expr aa.png ":" ".p"

# Check jpg file
expr bb.jpg ":" ".*\.jpg" # 6
expr bb.jpghhfff ":" ".*\.jpg" # Also matches successfully # 6

Check if the file name suffix is valid

#!/bin/bash

if expr "$1" ":" ".*\.jpg" &> /dev/null
  then
    echo "This is a jpg!"
else
    echo "not a jpg!"
fi

Find words with a length not exceeding 2

#!/bin/bash

for str1 in This is a str, you know?
    do
        if [ `expr length str1` -le 2 ]
            then
                echo $str1
        done

2.1.5 bc, awk, Square Bracket Calculation#

bc interactive calculation

Calculate the sum from 1 to 100

echo {1..100} | tr " " "+" | bc

seq -s "+" 100 | bc

echo $((`seq -s "+" 100`)) # Highest efficiency

seq -s "+" 100 | xargs expr

echo "2.2 1.1" | awk '{print ($1 + $2)}'

num=5
res=$[num+4]
echo $res

res=$[num*4]
echo $res

2.2 Shell Conditional Testing#

2.2.1 test Conditional Testing#

-e Check if the file exists (ordinary file, directory); returns true (0) if it exists, otherwise false (1).
echo $?

Combine with && and || for use.

File types

File permission checks

File comparison

-z If the string is empty, it returns true; otherwise, it returns false.
-n The opposite; if the string has content, it returns true; otherwise, it returns false.

2.2.2 [ ] Conditional Testing#

The function of test and [] is the same.

There must be spaces in [ ].

When using variables in conditional tests, double quotes "" must be used.

[ -f "${file1}" ]
[[ -f "${file1}" ]]

2.2.3 Variable Testing#

Use double quotes.

2.2.4 String Comparison Testing#

Variables must have double quotes.

There must be spaces around =.

2.2.5 Numeric Comparison Testing#

Generally, use [].

[[]] also supports -eq, etc.

2.2.6 Logical Operations#

read -p "Please input a char:" var1

[ "$var1" -eq "1" ] && {
    echo $var1
    exit 0
}

[ "$var2" = "2" ] && {
    echo $var1
    exit 0
}

[ "$var1" != "2" -a "$var1" != "1" ] && {
    echo "Please input 1 or 2!!"
    exit 1
}

# Regular expressions
[[ $num =~ [1-3] ]]

Summary#

Logical AND -a

Logical OR -o

[[]] cannot use >= and <=, only -ge or -le can be used.

Most commonly used []

2.3 Script Development#

Develop Memory Check Script#

# Second line, last value
free -m | awk 'NR==2 {print $NF}'

#! /bin/bash
FreeMem=`free -m | awk 'NR==2 {print $NF}'`
CHARS="current mem is $FreeMem"

if [ "$FreeMem" -lt "100" ]; then
    echo $CHARS|tee /tmp/message.txt
    # mail -s subject, recipient < content
    mail -s "`date +%F-%T`$CHARS" [email protected] < /tmp/message.txt
fi

MySQL Monitoring Script#

MySQL monitoring script flow

# Local
netstat -tunlp | grep mysql | wc -l
ss -tunlp | grep mysql | wc -l
lsof -i tcp:3380 | wc -l

# Remote
nmap 127.0.0.1 -p 3380 | grep open | wc -l

# telnet
echo -e "\n" | telnet 127.0.0.1 3380 2>/dev/null | grep Connected | wc -l

Rsync Start/Stop Script Development#

2.4 Functions#

function xx() {

}

function xx{

}

xx(){

}

Beautification related

lsb_functions="/lib/lsb/init-functions"
if test -f $lsb_functions;then
    . $lsb_functions
else
    init_functions="/etc/init.d/functions"
    if test -f $init_functions;then
        . $init_functions
    fi
    log_success_msg() {
        echo " SUCCESS! $@"
    }
    log_failure_msg() {
        echo " ERROR! $@"
    }
fi

log_success_msg echo "test"

Function Return Values#

Shell functions can only return integer values. If you want the function to return a string, there are generally two methods: assign the return value to a string variable or output the return value, assigning it to a variable at the function call.

https://www.cnblogs.com/duanxz/p/4661767.html

https://roc-wong.github.io/blog/2017/03/shell-%E5%87%86%E5%88%86%E5%8F%82%E6%95%B0%E8%BF%94%E5%9B%9E%E5%AD%97%E7%AC%A6%E4%B8%B2%E7%9A%84%E6%96%B9%E6%B3%95.html

1. Assign the return value to a string variable

get_project_version(){
  VERSION=`grep version $1/pom.xml | head -n 1 | sed -E 's:</?version>::g' | sed -E 's/^\t*//' | sed -E 's/^ *//' | sed -E 's/\r$//'`
}

2. Output the return value, assigning it to a variable at the function call

get_project_version(){
  echo `grep version $1/pom.xml | head -n 1 | sed -E 's:</?version>::g' | sed -E 's/^\t*//' | sed -E 's/^ *//' | sed -E 's/\r$//'`
}
version=`get_project_version $PROJECT_PATH`