# 範例與常用技巧

##### 檔頭常用宣告

- [Use Bash Strict Mode](http://redsymbol.net/articles/unofficial-bash-strict-mode/)

```shell
# quickly syntax
set -euo pipefail

# let script exit if a command fails
set -o errexit 
# OR
set -e

# let script exit if an unsed variable is used
set -o nounset
# OR
set -u

# This setting prevents errors in a pipeline from being masked.
set -o pipefail

# for Debug
set -x

# setting IFS
IFS=$'\n\t'
```

How `set -o pipefail` works

```shell
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
$ echo $?
0

$ set -o pipefail
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
$ echo $?
2
```

How the `IFS` works

```shell
#!/bin/bash
names=(
  "Aaron Maxwell"
  "Wayne Gretzky"
  "David Beckham"
  "Anderson da Silva"
)

echo "With default IFS value..."
for name in ${names[@]}; do
  echo "$name"
done

echo ""
echo "With strict-mode IFS value..."
IFS=$'\n\t'
for name in ${names[@]}; do
  echo "$name"
done

############### Output #############
With default IFS value...
Aaron
Maxwell
Wayne
Gretzky
David
Beckham
Anderson
da
Silva

With strict-mode IFS value...
Aaron Maxwell
Wayne Gretzky
David Beckham
Anderson da Silva
```

##### 檔案的目錄位置

```shell
# 指定的檔案
readlink -f <file.name>

# 目前檔案
WORKDIR=$(readlink -f "$0") ;目前檔案的絕對路經
WORKDIR=$( cd $( dirname "$0" ) && pwd )
```

##### Script 檔案名稱

```
$ echo $0
./test.sh

$ echo `basename $0`
test.sh 
```

##### 輸出多行的文字訊息

```shell
cat <<EOF
Welcome .....
	
Here are the messages that you want to show up
		
EOF
```

##### 所有輸出訊息導入一個檔案

Sample #1

```shell
#!/bin/sh
LOG="my.log"
(
....
) 2>&1 | tee -a $LOG 
```

<p class="callout info">不適用在內容裡有 python 指令的訊息輸入，輸入等待的畫面會無法顯示。</p>

Sample #2

```shell
temp=$(mktemp)
exec &> ${temp}

echo "All outputs will be saved into the file ${temp}."
```

Sample #3

```shell
#!/bin/bash
set -eu 
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>/path/to/script.log 2>&1

# rest of the script below
dnf -y in foo bar
# firewall rules goes here 
....
...
```

##### 快速修改大量檔案的副檔名

```editable
## *.old -> *.new
for fname in $(ls *.old);do echo "mv $fname ->"; echo $(echo $fname |sed 's/.old/.new/');mv $fname $(echo $fname | sed 's/.old/.new/');done
```

##### 快速新增或列出多個指定目錄

```shell
mkdir {AAA,BBB,CCC}
```

##### 快速備份設定檔

```shell
cp my.cfg{,.bak}
```

##### 計算檔案總數量

用 find

```bash
# 計算目前目錄底下所有檔案數
find . -type f | wc -l 

# 計算第一層所有子目錄的檔案數並排序清單
find . -maxdepth 1 -type d -print0 | xargs -0 -I {} sh -c 'echo $(find {} -type f | wc -l) {}' | sort -n
NOTE: 第一層子目錄名稱不可包含空格 
```

用 rsync

```
rsync --stats --dry-run -ax /usr /tmp

Number of files: 326,373 (reg: 211,698, dir: 24,284, link: 90,391)
Number of created files: 326,373 (reg: 211,698, dir: 24,284, link: 90,391)
Number of deleted files: 0
Number of regular files transferred: 211,698
Total file size: 7,180,685,730 bytes
Total transferred file size: 7,178,524,818 bytes

NOTE: 這指令實際不會作檔案複製，/tmp 只是一個假目錄，回傳結果是 reg: 211698 就是檔案總數
```

用 tree

```
tree /mydir -a | tail -n 1

5 directories, 56 files

NOTE: 當目錄底下有包含 symbolic link 也會被計算
```

##### 執行外部 SHELL 或指令

1\. 使用 pipe line

```shell
echo"md5sum $X > $X.sum "| bash
```

2\. 使用 eval

```shell
get_arch="uname -p"
if [ "`eval "$get_arch"`" = "i686" ]; then
....
fi 
```

##### 線上執行 Shell

```bash
bash <(curl -fsSL https://raw.githubusercontent.com/IT-BAER/proxmorph/main/install.sh) install
```

##### 提示字元的路徑名稱太長

加上這變數

```shell
PROMPT_DIRTRIM=2
```

##### 處理 CSV 檔

- [Doing a database join with CSV files](https://www.johndcook.com/blog/2019/12/31/sql-join-csv-files/)
- [tvs-utils](https://github.com/eBay/tsv-utils) - eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
- [How To Parse CSV Files In Bash Scripts In Linux](https://ostechnix.com/parse-csv-files-in-bash-scripts/)
- [How to convert JSON to CSV using Linux / Unix shell](https://www.cyberciti.biz/faq/how-to-convert-json-to-csv-using-linux-unix-shell/)

```shell
#!/bin/bash
INPUT=data.cvs
OLDIFS=$IFS
IFS=','
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read flname dob ssn tel status
do
	echo "Name : $flname"
	echo "DOB : $dob"
	echo "SSN : $ssn"
	echo "Telephone : $tel"
	echo "Status : $status"
done < $INPUT
IFS=$OLDIFS
```

CSV and JSON

```shell
# JSON to CSV
cat df.json | jq -r '.[]| join(",")'
cat bingbot.json | jq -r '.prefixes[] | {cidr: .ipv4Prefix, comment: "BingBot"} | join(",")' > bingbot.csv

```

##### JSON 檔

- \[GitHub\] [gron - Make JSON greppable!](https://github.com/TomNomNom/gron)

##### 建立暫存檔

```shell
tmpfile1=$(mktemp)
tmpfile2="/tmp/$(basename $0).$$.tmp"
```

##### Get my public IP

```bash
curl ifconfig.me
curl ifconfig.me/ip
curl ifconfig.co
curl checkip.amazonaws.com
curl icanhazip.com
curl ipecho.net/plain

dig +short myip.opendns.com @resolver1.opendns.com
dig TXT +short o-o.myaddr.l.google.com @ns1.google.com
dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{print $2}'
```

##### 主機 IP

```bash
# On Linux
hostname -I
hostip=$(/sbin/ip a | awk '/eth[012]:|ens192:|bond0:/,/^$/' | grep -E "inet [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | head -1 | awk -F " " '{print $2}' | cut -d"/" -f1)

# On AIX
hostip=$( ifconfig -a | grep inet | awk '{print $2}' | head -1 )
hostip=$( ifconfig -a | grep inet | awk '{print $2}' | sed -n '/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/p' | head -1 )
```

##### 找出最大的目錄或檔案

```bash
# Way 1
du -a /var | sort -n -r | head -n 10

# Way 2
cd /path/to/some/where
du -hsx * | sort -rh | head -10
du -hsx -- * | sort -rh | head -10

# Way 3
find /path/to/dir/ -printf '%s %p\n'| sort -nr | head -10
find . -printf '%s %p\n'| sort -nr | head -10
## Skip directories and only display files
find /path/to/search/ -type f -printf '%s %p\n'| sort -nr | head -10

# Create a shell alias
## shell alias ##  
alias ducks='du -cks * | sort -rn | head'
## deal with special files names ##
alias ducks='du -cks -- * | sort -rn | head'
```

##### 檔名移除空白字元

```bash
# With tr
for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done

# With find
find . -type f -name "* *.xml" -exec bash -c 'mv "$0" "${0// /_}"' {} \;
```

##### read

輸入密碼

```bash
echo -n "MySQL username: " ; read username
echo -n "MySQL password: " ; stty -echo ; read password ; stty echo ; echo
```

Yes/No

```bash
while true; do
echo "The Server IP is $serverip"
read -p "Are you sure that you want to continue? (y/N): " input
input=${input:-n}
    case "$input" in
        y|Y)
            echo
            break
            ;;
        n|N)
            echo "Exit"
            exit 1
            ;;
        *) echo "Please answer Y or N.";;
    esac
done
```

##### shuf: 隨機排序

```
curl -s https://www.imdb.com/list/ls020046354/export | cut -d ',' -f 6  | shuf
```

##### tree: 顯示專案目錄的檔案樹

```shell
tree --dirsfirst --filelimit 10 --sort=name

# Display size of files
tree -s

# Display permissions of files
tree -p

# Display directory only
tree -d

# Display till a certain level/depth
tree -L 1

# List only those files that match pattern given
tree -P *screenshot*
```

##### sort : 資料排序

```shell
sort -t ',' -k5,5 -k1,1 -k9,9 -k3,3 -k11,11 my.csv
```

- -t 分隔符號
- -k5,5 排序第 5 欄，以字串類型排列
- 欄位排序先後依序為第 5, 1, 9, 3, 11 欄
- 欄位排序若要以數值方式來排，改成 -k5,5n

##### timeout : 自動停止執行

```shell
timeout 10 tail -f /var/log/httpd/access.log
timeout 5m ping 8.8.8.8
timeout 300 tcpdump -n -w data.pcap

# Sending specific signal
# To get a list of all available signals, use the command kill -l .
timeout -s SIGKILL ping 8.8.8.8
```

##### variables : 變數

<table border="1" cellpadding="1" cellspacing="1" id="bkmrk-%E8%AE%8A%E6%95%B8-%E8%AA%AA%E6%98%8E-%C2%A0%240-%C2%A0%E8%85%B3%E6%9C%AC%E6%AA%94%E5%90%8D-%C2%A0%241-"><tbody><tr><td>變數</td><td>說明</td></tr><tr><td> $0</td><td> 腳本檔名</td></tr><tr><td> $1</td><td> 第1個參數</td></tr><tr><td> $2</td><td> 第2個參數</td></tr><tr><td> ${10}</td><td> 第10個參數 #10</td></tr><tr><td> $#</td><td> 參數的個數</td></tr><tr><td> $\*</td><td> 顯示所有參數 (作為一個字串)</td></tr><tr><td> $@</td><td> 顯示所有參數 (每個為一個獨立字串)</td></tr><tr><td> ${$\*}</td><td> 傳遞到腳本中的參數的個數</td></tr><tr><td> ${$@}</td><td> 傳遞到腳本中的參數的個數</td></tr><tr><td> $?</td><td> 腳本結束後的傳回值</td></tr><tr><td> $$</td><td> 腳本的程序 ID</td></tr><tr><td> $\_</td><td> 前個命令的最後一個參數</td></tr><tr><td> $!</td><td> 最後一個執行程序的 ID</td></tr><tr><td>u=${1:-root}</td><td> 如果 $1 未指定，就賦予值 root</td></tr><tr><td>u=${USER:-foo}</td><td> 如果 $USER 未指定，就賦予值 foo</td></tr><tr><td>len=${#var}</td><td> 計算 $var 字串的長度</td></tr></tbody></table>

如果 $2 未指定或空值，輸出錯誤訊息

```shell
${varName?Error varName is not defined}
${varName:?Error varName is not defined or is empty}
```

- [Introduction to Bash Shell Parameter Expansions](https://linuxconfig.org/introduction-to-bash-shell-parameter-expansions)

建立目錄並且切換至目錄

```bash
mkdir my-dir && cd $_
```

##### cut: 切割文字

```shell
# AAA = BBB, 取出 BBB
cut -d= -f2

# 111 2222 33 444444 555, 取出 33 以後的所有內容
cut -d ' ' -f3-

# 檢視超長的文字
head -n 1 data.csv | wc  # 計算行的字元數
head data.csv | cut -c -30  # 列出前幾行的前 30 個字元, 如果要顯示後 30 個字元，可以用 30-
```

##### tr: 置換

```bash
# aaa bbb ccc
# 換成
# aaa
# bbb
# ccc
echo "aaa bbb ccc" | tr " " "\n"
```

##### xargs

接入(Pipe) 不支援 stdin 指令的替代方法

- [How to Use the Powerful Xargs Command in Linux](https://linuxhandbook.com/xargs-command/)

##### printf: 格式化輸出

- %s 字串
- %d 整數

```
printf "%-40s ..................%s\n" "Disable the service $1" "[$2]"

Disable the service apmd                 ..................[OK]
Disable the service bluetooth            ..................[OK]
Disable the service hidd                 ..................[OK]
Disable the service cups                 ..................[OK]
Disable the service firstboot            ..................[OK]
Disable the service readahead_early      ..................[OK]

printf "%40s ..................%s\n" "Disable the service $1" "[$2]"

                Disable the service apmd ..................[OK]
           Disable the service bluetooth ..................[OK]
                Disable the service hidd ..................[OK]
                Disable the service cups ..................[OK]
           Disable the service firstboot ..................[OK]
     Disable the service readahead_early ..................[OK] 
```

數字四捨五入

```bash
printf "%.0f" 9.46666667 # output: 10

printf "%.2f" 9.46666667 # output: 9.47
```

表格式輸出

```bash
line="--------------------------------------------------"
printf "+-%.10s-+-%.5s-+-%.6s-+\n" "$line" "$line" "$line"
printf "| %-10s | %-5s | %-6s |\n" "DATE" "CALLS" "ASR(%)"
printf "+-%.10s-+-%.5s-+-%.6s-+\n" "$line" "$line" "$line"

printf "| %.10s | %5d | %.2f  | %s\n" "$day" $calls $asr "$flag"

+------------+-------+--------+
| DATE       | CALLS | ASR(%) |
+------------+-------+--------+
| 2025-07-10 |  1442 | 97.57  |
| 2025-07-11 |  1266 | 96.99  |
| 2025-07-12 |  1162 | 97.24  |
| 2025-07-13 |   949 | 96.62  |
| 2025-07-14 |  1178 | 98.13  |
| 2025-07-15 |  1665 | 97.05  |
| 2025-07-16 |  1163 | 87.61  |
+------------+-------+--------+

```

```bash
#/bin/bash
seperator=--------------------
seperator=$seperator$seperator
rows="%-15s| %.7d| %3d| %c\n"
TableWidth=37

printf "%-15s| %-7s| %.3s| %s\n" Name ID Age Grades
printf "%.${TableWidth}s\n" "$seperator"
printf "$rows" "Sherlock Holmes" 122 23 A
printf "$rows" "James Bond" 7 27 F
printf "$rows" "Hercules Poirot" 6811 59 G
printf "$rows" "Jane Marple" 1234567 71 C


Name           | ID     | Age| Grades
-------------------------------------
Sherlock Holmes| 0000122|  23| A
James Bond     | 0000007|  27| F
Hercules Poirot| 0006811|  59| G
Jane Marple    | 1234567|  71| C
```

##### 印出一個長符號

```
printf -- '-%.0s' {1..80}
printf -- '=%.0s' {1..80}
```

##### ping: 掃描一個 IP 範圍

```shell
{ for p in {1..254}; do ping -c1 -w1 10.22.9.$p & done } | grep "64 bytes"
```

##### nice: Reduce CPU and Disk load of backup scripts

```shell
# Reduce the I/O priority of the /usr/local/bin/backup.sh script so that it does not interfere with other processes
# The -n parameter must be between 0 and 7, where lower numbers mean higher priority
/usr/bin/ionice -c2 -n7 /usr/local/bin/backup.sh

# To reduce the CPU priority, use the command nice
# The -n parameter can range from -20 to 19, where lower numbers mean higher priority
/usr/bin/nice -n 19 /usr/local/bin/backup.sh

# Nice and ionice can also be combined, to run a script at low I/O and CPU priority
/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /usr/local/bin/backup.sh
```

##### Hex to ASCII  


```shell
# hex = 54657374696e672031203220330
# ascii = Testing 1 2 3

# xxd
echo 54657374696e672031203220330 | xxd -r -p && echo ''

# printf
printf '\x54\x65\x73\x74\x69\x6e\x67\x20\x31\x20\x32\x20\x33\x0' && echo ''

# sed
echo -n 54657374696e67203120322033 | sed 's/\([0-9A-F]\{2\}\)/\\\\\\x\1/gI' | xargs printf && echo ''
```

##### 隨機密碼生成

```shell
genpasswd() { 
	local l=$1
       	[ "$l" == "" ] && l=16
      	tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs 
}
```

```shell
tr -dc A-Za-z0-9_ < /dev/urandom | head -c 16 | xargs

# Generate more than one
tr -dc A-Za-z0-9_ < /dev/urandom | fold -16 | head -5

#
echo FooBar$RANDOM | md5sum | base64 | cut -c 1-12
```

##### ls: 進階技巧  


```shell
# Find the biggest zip file
ls -lSrh ~/Downloads/*.zip
```

##### stat: 檔案屬性

```bash
❯ stat --printf='Name: %n\nPermissions: %a\n' my.log
Name: my.log
Permissions: 777

❯ stat --format="%F" my.log
regular file

# Symlink
❯ stat ~/bin/FoxitReader
  File: /home/alang/bin/FoxitReader -> /home/alang/opt/foxitsoftware/foxitreader/FoxitReader.sh
  Size: 56        	Blocks: 0          IO Block: 4096   symbolic link
Device: 10302h/66306d	Inode: 787474      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/   alang)   Gid: ( 1000/   alang)
Access: 2023-07-16 10:26:13.412193581 +0800
Modify: 2023-02-26 12:13:50.234374171 +0800
Change: 2023-02-26 12:13:50.234374171 +0800
 Birth: 2023-02-26 12:13:50.234374171 +0800

❯ stat -L ~/bin/FoxitReader
  File: /home/alang/bin/FoxitReader
  Size: 120       	Blocks: 8          IO Block: 4096   regular file
Device: 10302h/66306d	Inode: 1457222     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/   alang)   Gid: ( 1000/   alang)
Access: 2023-07-02 14:34:09.957428285 +0800
Modify: 2023-02-26 12:13:50.246374250 +0800
Change: 2023-02-26 12:13:50.246374250 +0800
 Birth: 2023-02-26 12:13:48.530362986 +0800
```

##### less: 進階技巧

顯示有包含色碼的內容

```bash
less -r yourfile
```