# 範例與常用技巧 ##### 檔頭常用宣告 - [Use Bash Strict Mode](http://redsymbol.net/articles/unofficial-bash-strict-mode/) ```shell # quickly syntax set -euo pipefail # let script exit if a command fails set -o errexit # OR set -e # let script exit if an unsed variable is used set -o nounset # OR set -u # This setting prevents errors in a pipeline from being masked. set -o pipefail # for Debug set -x # setting IFS IFS=$'\n\t' ``` How `set -o pipefail` works ```shell $ grep some-string /non/existent/file | sort grep: /non/existent/file: No such file or directory $ echo $? 0 $ set -o pipefail $ grep some-string /non/existent/file | sort grep: /non/existent/file: No such file or directory $ echo $? 2 ``` How the `IFS` works ```shell #!/bin/bash names=( "Aaron Maxwell" "Wayne Gretzky" "David Beckham" "Anderson da Silva" ) echo "With default IFS value..." for name in ${names[@]}; do echo "$name" done echo "" echo "With strict-mode IFS value..." IFS=$'\n\t' for name in ${names[@]}; do echo "$name" done ############### Output ############# With default IFS value... Aaron Maxwell Wayne Gretzky David Beckham Anderson da Silva With strict-mode IFS value... Aaron Maxwell Wayne Gretzky David Beckham Anderson da Silva ``` ##### 檔案的目錄位置 ```shell # 指定的檔案 readlink -f # 目前檔案 WORKDIR=$(readlink -f "$0") ;目前檔案的絕對路經 WORKDIR=$( cd $( dirname "$0" ) && pwd ) ``` ##### Script 檔案名稱 ``` $ echo $0 ./test.sh $ echo `basename $0` test.sh ``` ##### 輸出多行的文字訊息 ```shell cat <&1 | tee -a $LOG ```

不適用在內容裡有 python 指令的訊息輸入,輸入等待的畫面會無法顯示。

Sample #2 ```shell temp=$(mktemp) exec &> ${temp} echo "All outputs will be saved into the file ${temp}." ``` Sample #3 ```shell #!/bin/bash set -eu exec 3>&1 4>&2 trap 'exec 2>&4 1>&3' 0 1 2 3 exec 1>/path/to/script.log 2>&1 # rest of the script below dnf -y in foo bar # firewall rules goes here .... ... ``` ##### 快速修改大量檔案的副檔名 ```editable ## *.old -> *.new for fname in $(ls *.old);do echo "mv $fname ->"; echo $(echo $fname |sed 's/.old/.new/');mv $fname $(echo $fname | sed 's/.old/.new/');done ``` ##### 快速新增或列出多個指定目錄 ```shell mkdir {AAA,BBB,CCC} ``` ##### 快速備份設定檔 ```shell cp my.cfg{,.bak} ``` ##### 計算檔案總數量 用 find ```bash # 計算目前目錄底下所有檔案數 find . -type f | wc -l # 計算第一層所有子目錄的檔案數並排序清單 find . -maxdepth 1 -type d -print0 | xargs -0 -I {} sh -c 'echo $(find {} -type f | wc -l) {}' | sort -n NOTE: 第一層子目錄名稱不可包含空格 ``` 用 rsync ``` rsync --stats --dry-run -ax /usr /tmp Number of files: 326,373 (reg: 211,698, dir: 24,284, link: 90,391) Number of created files: 326,373 (reg: 211,698, dir: 24,284, link: 90,391) Number of deleted files: 0 Number of regular files transferred: 211,698 Total file size: 7,180,685,730 bytes Total transferred file size: 7,178,524,818 bytes NOTE: 這指令實際不會作檔案複製,/tmp 只是一個假目錄,回傳結果是 reg: 211698 就是檔案總數 ``` 用 tree ``` tree /mydir -a | tail -n 1 5 directories, 56 files NOTE: 當目錄底下有包含 symbolic link 也會被計算 ``` ##### 執行外部 SHELL 或指令 1\. 使用 pipe line ```shell echo"md5sum $X > $X.sum "| bash ``` 2\. 使用 eval ```shell get_arch="uname -p" if [ "`eval "$get_arch"`" = "i686" ]; then .... fi ``` ##### 提示字元的路徑名稱太長 加上這變數 ```shell PROMPT_DIRTRIM=2 ``` ##### 處理 CSV 檔 - [Doing a database join with CSV files](https://www.johndcook.com/blog/2019/12/31/sql-join-csv-files/) - [tvs-utils](https://github.com/eBay/tsv-utils) - eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more. - [How To Parse CSV Files In Bash Scripts In Linux](https://ostechnix.com/parse-csv-files-in-bash-scripts/) - [How to convert JSON to CSV using Linux / Unix shell](https://www.cyberciti.biz/faq/how-to-convert-json-to-csv-using-linux-unix-shell/) ```shell #!/bin/bash INPUT=data.cvs OLDIFS=$IFS IFS=',' [ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; } while read flname dob ssn tel status do echo "Name : $flname" echo "DOB : $dob" echo "SSN : $ssn" echo "Telephone : $tel" echo "Status : $status" done < $INPUT IFS=$OLDIFS ``` CSV and JSON ```shell # JSON to CSV cat df.json | jq -r '.[]| join(",")' cat bingbot.json | jq -r '.prefixes[] | {cidr: .ipv4Prefix, comment: "BingBot"} | join(",")' > bingbot.csv ``` ##### JSON 檔 - \[GitHub\] [gron - Make JSON greppable!](https://github.com/TomNomNom/gron) ##### 建立暫存檔 ```shell tmpfile1=$(mktemp) tmpfile2="/tmp/$(basename $0).$$.tmp" ``` ##### Get my public IP ```bash curl ifconfig.me curl ifconfig.me/ip curl ifconfig.co curl checkip.amazonaws.com curl icanhazip.com curl ipecho.net/plain dig +short myip.opendns.com @resolver1.opendns.com dig TXT +short o-o.myaddr.l.google.com @ns1.google.com dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{print $2}' ``` ##### 主機 IP ```bash # On Linux hostname -I hostip=$(/sbin/ip a | awk '/eth[012]:|ens192:|bond0:/,/^$/' | grep -E "inet [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | head -1 | awk -F " " '{print $2}' | cut -d"/" -f1) # On AIX hostip=$( ifconfig -a | grep inet | awk '{print $2}' | head -1 ) hostip=$( ifconfig -a | grep inet | awk '{print $2}' | sed -n '/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/p' | head -1 ) ``` ##### 找出最大的目錄或檔案 ```bash # Way 1 du -a /var | sort -n -r | head -n 10 # Way 2 cd /path/to/some/where du -hsx * | sort -rh | head -10 du -hsx -- * | sort -rh | head -10 # Way 3 find /path/to/dir/ -printf '%s %p\n'| sort -nr | head -10 find . -printf '%s %p\n'| sort -nr | head -10 ## Skip directories and only display files find /path/to/search/ -type f -printf '%s %p\n'| sort -nr | head -10 # Create a shell alias ## shell alias ## alias ducks='du -cks * | sort -rn | head' ## deal with special files names ## alias ducks='du -cks -- * | sort -rn | head' ``` ##### 檔名移除空白字元 ```bash # With tr for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done # With find find . -type f -name "* *.xml" -exec bash -c 'mv "$0" "${0// /_}"' {} \; ``` ##### shuf: 隨機排序 ``` curl -s https://www.imdb.com/list/ls020046354/export | cut -d ',' -f 6 | shuf ``` ##### tree: 顯示專案目錄的檔案樹 ```shell tree --dirsfirst --filelimit 10 --sort=name # Display size of files tree -s # Display permissions of files tree -p # Display directory only tree -d # Display till a certain level/depth tree -L 1 # List only those files that match pattern given tree -P *screenshot* ``` ##### sort : 資料排序 ```shell sort -t ',' -k5,5 -k1,1 -k9,9 -k3,3 -k11,11 my.csv ``` - -t 分隔符號 - -k5,5 排序第 5 欄,以字串類型排列 - 欄位排序先後依序為第 5, 1, 9, 3, 11 欄 - 欄位排序若要以數值方式來排,改成 -k5,5n ##### timeout : 自動停止執行 ```shell timeout 10 tail -f /var/log/httpd/access.log timeout 5m ping 8.8.8.8 timeout 300 tcpdump -n -w data.pcap # Sending specific signal # To get a list of all available signals, use the command kill -l . timeout -s SIGKILL ping 8.8.8.8 ``` ##### variables : 變數
變數說明
$0 腳本檔名
$1 第1個參數
$2 第2個參數
${10} 第10個參數 #10
$# 參數的個數
$\* 顯示所有參數 (作為一個字串)
$@ 顯示所有參數 (每個為一個獨立字串)
${$\*} 傳遞到腳本中的參數的個數
${$@} 傳遞到腳本中的參數的個數
$? 腳本結束後的傳回值
$$ 腳本的程序 ID
$\_ 前個命令的最後一個參數
$! 最後一個執行程序的 ID
u=${1:-root} 如果 $1 未指定,就賦予值 root
u=${USER:-foo} 如果 $USER 未指定,就賦予值 foo
len=${#var} 計算 $var 字串的長度
如果 $2 未指定或空值,輸出錯誤訊息 ```shell ${varName?Error varName is not defined} ${varName:?Error varName is not defined or is empty} ``` - [Introduction to Bash Shell Parameter Expansions](https://linuxconfig.org/introduction-to-bash-shell-parameter-expansions) 建立目錄並且切換至目錄 ```bash mkdir my-dir && cd $_ ``` ##### cut: 切割文字 ```shell # AAA = BBB, 取出 BBB cut -d= -f2 # 111 2222 33 444444 555, 取出 33 以後的所有內容 cut -d ' ' -f3- # 檢視超長的文字 head -n 1 data.csv | wc # 計算行的字元數 head data.csv | cut -c -30 # 列出前幾行的前 30 個字元, 如果要顯示後 30 個字元,可以用 30- ``` ##### tr: 置換 ```bash # aaa bbb ccc # 換成 # aaa # bbb # ccc echo "aaa bbb ccc" | tr " " "\n" ``` ##### xargs 接入(Pipe) 不支援 stdin 指令的替代方法 - [How to Use the Powerful Xargs Command in Linux](https://linuxhandbook.com/xargs-command/) ##### printf: 格式化輸出 - %s 字串 - %d 整數 ``` printf "%-40s ..................%s\n" "Disable the service $1" "[$2]" Disable the service apmd ..................[OK] Disable the service bluetooth ..................[OK] Disable the service hidd ..................[OK] Disable the service cups ..................[OK] Disable the service firstboot ..................[OK] Disable the service readahead_early ..................[OK] printf "%40s ..................%s\n" "Disable the service $1" "[$2]" Disable the service apmd ..................[OK] Disable the service bluetooth ..................[OK] Disable the service hidd ..................[OK] Disable the service cups ..................[OK] Disable the service firstboot ..................[OK] Disable the service readahead_early ..................[OK] ``` ##### 印出一個長符號 ``` printf -- '-%.0s' {1..80} printf -- '=%.0s' {1..80} ``` ##### ping: 掃描一個 IP 範圍 ```shell { for p in {1..254}; do ping -c1 -w1 10.22.9.$p & done } | grep "64 bytes" ``` ##### nice: Reduce CPU and Disk load of backup scripts ```shell # Reduce the I/O priority of the /usr/local/bin/backup.sh script so that it does not interfere with other processes # The -n parameter must be between 0 and 7, where lower numbers mean higher priority /usr/bin/ionice -c2 -n7 /usr/local/bin/backup.sh # To reduce the CPU priority, use the command nice # The -n parameter can range from -20 to 19, where lower numbers mean higher priority /usr/bin/nice -n 19 /usr/local/bin/backup.sh # Nice and ionice can also be combined, to run a script at low I/O and CPU priority /usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/bin/backup.sh ``` ##### Hex to ASCII ```shell # hex = 54657374696e672031203220330 # ascii = Testing 1 2 3 # xxd echo 54657374696e672031203220330 | xxd -r -p && echo '' # printf printf '\x54\x65\x73\x74\x69\x6e\x67\x20\x31\x20\x32\x20\x33\x0' && echo '' # sed echo -n 54657374696e67203120322033 | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf && echo '' ``` ##### 隨機密碼生成 ```shell genpasswd() { local l=$1 [ "$l" == "" ] && l=16 tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs } ``` ```shell tr -dc A-Za-z0-9_ < /dev/urandom | head -c 16 | xargs # Generate more than one tr -dc A-Za-z0-9_ < /dev/urandom | fold -16 | head -5 # echo FooBar$RANDOM | md5sum | base64 | cut -c 1-12 ``` ##### ls: 進階技巧 ```shell # Find the biggest zip file ls -lSrh ~/Downloads/*.zip ``` ##### stat: 檔案屬性 ```bash ❯ stat --printf='Name: %n\nPermissions: %a\n' my.log Name: my.log Permissions: 777 ❯ stat --format="%F" my.log regular file # Symlink ❯ stat ~/bin/FoxitReader File: /home/alang/bin/FoxitReader -> /home/alang/opt/foxitsoftware/foxitreader/FoxitReader.sh Size: 56 Blocks: 0 IO Block: 4096 symbolic link Device: 10302h/66306d Inode: 787474 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 1000/ alang) Gid: ( 1000/ alang) Access: 2023-07-16 10:26:13.412193581 +0800 Modify: 2023-02-26 12:13:50.234374171 +0800 Change: 2023-02-26 12:13:50.234374171 +0800 Birth: 2023-02-26 12:13:50.234374171 +0800 ❯ stat -L ~/bin/FoxitReader File: /home/alang/bin/FoxitReader Size: 120 Blocks: 8 IO Block: 4096 regular file Device: 10302h/66306d Inode: 1457222 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 1000/ alang) Gid: ( 1000/ alang) Access: 2023-07-02 14:34:09.957428285 +0800 Modify: 2023-02-26 12:13:50.246374250 +0800 Change: 2023-02-26 12:13:50.246374250 +0800 Birth: 2023-02-26 12:13:48.530362986 +0800 ```