Linux Administration

校時與時區設定

Tutorials

Chrony

Built-in on CentOS/RedHat

Install

yum install chrony

Config: /etc/chrony.conf

# NTP Public Servers for Taiwan
server tw.pool.ntp.org
server time.stdtime.gov.tw
server clock.stdtime.gov.tw
server jp.pool.ntp.org

# Allow NTP client access from local network.
#allow 192.168.0.0/16

Starting

systemctl start chronyd

Verify

[root@localhost ~]# chronyc tracking
Reference ID    : 7A75FDF6 (static.home.twn.sciuridae.cloud)
Stratum         : 3
Ref time (UTC)  : Sun Aug 16 03:35:36 2020
System time     : 0.000073180 seconds slow of NTP time
Last offset     : -0.001259724 seconds
RMS offset      : 0.001259724 seconds
Frequency       : 21.057 ppm slow
Residual freq   : -0.000 ppm
Skew            : 28.166 ppm
Root delay      : 0.032856792 seconds
Root dispersion : 0.002069760 seconds
Update interval : 64.3 seconds
Leap status     : Normal    <===

[root@tpemimtst99 ~]# timedatectl
               Local time: Tue 2021-05-25 09:09:09 CST
           Universal time: Tue 2021-05-25 01:09:09 UTC
                 RTC time: Tue 2021-05-25 01:09:09
                Time zone: Asia/Taipei (CST, +0800)
System clock synchronized: yes    <===
              NTP service: active
          RTC in local TZ: no

[root@tpemimtst99 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 192.168.21.86                 3  10   377   754  +1543ns[+4123ns] +/-   63ms

systemd-timesyncd

Build-in on Ubuntu

The configuration file for systemd-timesyncd is /etc/systemd/timesyncd.conf.

[Time]
#NTP=
#FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048

Specify the time server as follows

NTP=<your-time-server-ip>

Start timesync

systemctl enable systemd-timesyncd.service
systemctl start systemd-timesyncd.service

timedatectl

# See the information of the datetime
sudo timedatectl
               Local time: Mon 2021-11-01 14:37:09 UTC
           Universal time: Mon 2021-11-01 14:37:09 UTC
                 RTC time: Mon 2021-11-01 14:37:10    
                Time zone: UTC (UTC, +0000)           
System clock synchronized: yes                        
              NTP service: active   <====                    
          RTC in local TZ: no 

# List timezones supported
sudo timedatectl list-timezones

# Change the system’s timezone
sudo timedatectl set-timezone Asia/Taipei

# Check the status
sudo timedatectl timesync-status
       Server: 91.189.91.157 (ntp.ubuntu.com)   
Poll interval: 17min 4s (min: 32s; max 34min 8s)
         Leap: normal                           
      Version: 4                                
      Stratum: 2                                
    Reference: 11FD227B                         
    Precision: 1us (-24)                        
Root distance: 72.295ms (max: 5s)               
       Offset: +5.311ms                         
        Delay: 67.290ms                         
       Jitter: 2.211ms                          
 Packet count: 6                                
    Frequency: +17.421ppm               

ntpd

Install

# Ubuntu
sudo apt install ntp

vi /etc/ntp.conf

#pool 0.ubuntu.pool.ntp.org iburst
#pool 1.ubuntu.pool.ntp.org iburst
#pool 2.ubuntu.pool.ntp.org iburst
#pool 3.ubuntu.pool.ntp.org iburst

# Use Ubuntu's ntp server as a fallback.
#pool ntp.ubuntu.com

# Added the local time server
server 192.168.21.86 prefer iburst

Restart the ntpd

# Ubuntu 16.04
systemctl stop ntp
systemctl start ntp

# Check the timeserver
ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
+eterna.binary.n 216.218.192.202  2 u   32   64    3   45.986   -0.081   0.021
+pacific.latt.ne 193.187.181.6    3 u   30   64    3    8.205    1.151   0.026
-mis.wci.com     216.218.192.202  2 u   32   64    3   37.928    4.692   0.176
-europa.ellipse. 209.180.247.49   2 u   31   64    3   51.792    4.175   0.027
#38.229.56.9     172.16.21.35     2 u   30   64    3   68.917   -4.451   0.190
#time.richiemcin 97.183.206.88    2 u   29   64    3   69.281   -3.073   0.248
*t1.time.gq1.yah 208.71.46.33     2 u   30   64    3   31.630   -0.081   0.011
-lofn.fancube.co 220.181.254.66   2 u   30   64    3   37.871   -1.088   0.009
#2606:6680:8:1:: 107.46.198.112   2 u   26   64    3  198.284   66.201   0.061
-titan.crash-ove 129.7.1.66       2 u   30   64    3   32.708   -1.474   0.200
-2620:1d5:100:43 47.187.174.51    2 u   29   64    3   70.418    0.592   0.197
 2001:67c:1560:8 17.253.34.123    2 u   32   64    3  133.490   -2.818   0.030
#38.229.57.9     172.16.21.35     2 u   30   64    3  177.896    4.081   6.291
#ntp2i.versadns. 217.180.209.214  2 u   28   64    3   62.795   -2.897   0.017
-104.194.8.227   192.12.19.20     2 u   25   64    3    0.979   -0.737   0.029
-pacific.latt.ne 193.187.181.6    3 u   26   64    3    8.770    1.212   0.050
 2001:67c:1560:8 17.253.34.123    2 u   33   64    3  133.602   -2.792   0.007



系統管理技巧

完整複製 Home 目錄

由於 User 的 Home 目錄內有許多隱藏檔,若要完整複製它們,有兩個方法:

方法一:可以複製成一個新目錄

cd /home
cp -a user1/ user1_new/

方法二:複製到一個現有目錄內

cd /home
cp -a user1/.[^.]* user1_new/ 

手動建立一個新的 Home 目錄

cp -r /etc/skel /home/user1
chown -R user1.group1 /home/user1
chmod 0700 /home/user1 
自訂 PATH

PATH 新增一個客制 bin 目錄

~/.bashrc :

case :$PATH: in
    *:/home/$USER/bin:*) ;;
    *) PATH=/home/$USER/bin:$PATH ;;
esac
清除 Zombie 程序(defunct)

One may deal with zombie processes in any one of the following ways:

列出 zombie processes

ps aux |grep "defunct"
ps aux |grep Z

# How many Zombie process running on your server
ps aux | awk {'print $8'}|grep -c Z

# List the PID of Zombie
ps aux | awk '{ print $8 " " $2 }' | grep -w Z

Kill zombie process

# find the parent process list
pstree -paul

kill -9 <PARENT-PID>

RHEL Documents:

資安 & Auditing 相關
# Parse /var/log/secure
grep "authentication failure" /var/log/secure | awk '{ print $13 }' | cut -b7-  | sort | uniq -c

# Login failed attempts
lastb -F
lastb -F <username>

Check Linux Login History

#!/bin/bash
#Filename: intruder_detect.sh
#Description: Check Linux Login History
AUTHLOG=/var/log/secure

if [[ -n $1 ]];
then
  AUTHLOG=$1
  echo Using Log file : $AUTHLOG
fi

# Collect the failed login attempts
FAILED_LOG=/tmp/failed.$$.log
egrep "Failed pass" $AUTHLOG > $FAILED_LOG 

# Collect the successful login attempts
SUCCESS_LOG=/tmp/success.$$.log
egrep "Accepted password|Accepted publickey|keyboard-interactive" $AUTHLOG > $SUCCESS_LOG

# extract the users who failed
failed_users=$(cat $FAILED_LOG | awk '{ print $(NF-5) }' | sort | uniq)

# extract the users who successfully logged in
success_users=$(cat $SUCCESS_LOG | awk '{ print $(NF-5) }' | sort | uniq)
# extract the IP Addresses of successful and failed login attempts
failed_ip_list="$(egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" $FAILED_LOG | sort | uniq)"
success_ip_list="$(egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" $SUCCESS_LOG | sort | uniq)"

# Print the heading
printf "%-10s|%-10s|%-10s|%-15s|%-15s|%s\n" "Status" "User" "Attempts" "IP address" "Host" "Time range"

# Loop through IPs and Users who failed.

for ip in $failed_ip_list;
do
  for user in $failed_users;
    do
    # Count failed login attempts by this user from this IP
    attempts=`grep $ip $FAILED_LOG | grep " $user " | wc -l`

    if [ $attempts -ne 0 ]
    then
      first_time=`grep $ip $FAILED_LOG | grep " $user " | head -1 | cut -c-16`
      time="$first_time"
      if [ $attempts -gt 1 ]
      then
        last_time=`grep $ip $FAILED_LOG | grep " $user " | tail -1 | cut -c-16`
        time="$first_time -> $last_time"
      fi
      HOST=$(host $ip 8.8.8.8 | tail -1 | awk '{ print $NF }' )
      printf "%-10s|%-10s|%-10s|%-15s|%-15s|%-s\n" "Failed" "$user" "$attempts" "$ip"  "$HOST" "$time";
    fi
  done
done

for ip in $success_ip_list;
do
  for user in $success_users;
    do
    # Count successful login attempts by this user from this IP
    attempts=`grep $ip $SUCCESS_LOG | grep " $user " | wc -l`

    if [ $attempts -ne 0 ]
    then
      first_time=`grep $ip $SUCCESS_LOG | grep " $user " | head -1 | cut -c-16`
      time="$first_time"
      if [ $attempts -gt 1 ]
      then
        last_time=`grep $ip $SUCCESS_LOG | grep " $user " | tail -1 | cut -c-16`
        time="$first_time -> $last_time"
      fi
      HOST=$(host $ip 8.8.8.8 | tail -1 | awk '{ print $NF }' )
      printf "%-10s|%-10s|%-10s|%-15s|%-15s|%-s\n" "Success" "$user" "$attempts" "$ip"  "$HOST" "$time";
    fi
  done
done

rm -f $FAILED_LOG
rm -f $SUCCESS_LOG

System Audit

# Install Audit
yum install audit
systemctl start auditd

# Authentication Report
# To get authentication report for all the attempts which was made
aureport -au -i | more
# To get authentication report for all the success attempts which was made
aureport -au -i --success | more
# To get authentication report for all the failed attempts which was made
aureport -au -i --failed | more
# To get success login information
aureport -l --success | more
# To get failed login information
aureport -l --failed | more
# To get success login summary report for all the success attempts which was made
aureport -l --success --summary -i | more

Check if a RHEL system is vulnerable to a specific CVE

# rpm -q --changelog [package-name] | grep [CVE-NUMBER]
rpm -q --changelog openssl | grep CVE-2021-3450
rpm -q --changelog openssl | grep CVE
rpm -q --changelog openssl | grep CVE-2021

# Using yum command
yum install yum-plugin-security
yum update yum
yum updateinfo info --cve CVE-2021-3445

Auditd

Auditing tool for UNIX/Linux like - Lynis

rsh

rsh server

# install on CentOS 6/7
yum install rsh-server

# Startup the service on CentOS 6
chkconfig rsh on
chkconfig rlogin on
service xinetd reload

# Startup the service on CentOS 7
systemctl start rsh.socket
systemctl start rlogin.socket
systemctl start rexec.socket
systemctl enable rsh.socket
systemctl enable rlogin.socket
systemctl enable rexec.socket
strace 程式除錯
# Trace the command
strace df -h

# Trace the process ID
strace -p 33259

# Get Summary of Linux Process
strace -c -p 3569

# Print Instruction Pointer During System Call
strace -i df -h

# Show Time of Day For Each Trace Output Line
strace -t df -h

# Print Command Time Spent in System Calls
strace -T df -h

# Trace Only Specific System Calls
strace -e trace=write df -h
strace -p 3569 -e poll
停用 suspend, hibernation
# disable the following systemd targets
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

# Then reboot the system and log in again
# Verify if the changes have been effected using the command
sudo systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target

# To re-enable the suspend and hibernation modes, run the command
sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target

To prevent the system from going into suspend state upon closing the lid, edit the /etc/systemd/logind.conf file.

[Login] 
HandleLidSwitch=ignore 
HandleLidSwitchDocked=ignore
磁碟資訊
# Approach #1
lsblk

nvme0n1     259:0    0 465.8G  0 disk 
├─nvme0n1p1 259:1    0   512M  0 part /boot/efi
└─nvme0n1p2 259:2    0 465.3G  0 part /
nvme1n1     259:3    0 953.9G  0 disk /media/alang/AlangsData

# Approach #2, requires to run as super-user.
sudo lshw -short -class disk,volume

H/W path                 Device           Class          Description
====================================================================
/0/100/14/0/3/4/0.0.0    /dev/sda         disk           Mass-Storage
/0/100/14/0/3/4/0.0.0/0  /dev/sda         disk       
last
# To check the last ten login attempts, you can pipe it with "head"
last | head -n 10

# using complete usernames and hostnames
last -w

# find the device used by the user
tty

# To find the last login by date,
last --since <date>
last --until <date>
last --since -2days

# find the last bad login attempts
sudo lastb
tail -f -n 100 /var/log/auth.log | grep -i failed

# find the last SSH logins
tail -f -n 100 /var/log/auth.log | grep -i sshd
sudo journalctl -r -u ssh | grep -i failed

# find last login times for all users
lastlog
lastlog -u <user>
檢測虛擬平台類型
dmidecode -s system-manufacturer
systemd-detec-virt
virt-what
硬體資訊
sudo lshw -short

H/W path                 Device           Class          Description
====================================================================
                                          system         NUC8i7HVK
/0                                        bus            NUC8i7HVB
/0/0                                      memory         64KiB BIOS
/0/2f                                     memory         16GiB System Memory
/0/2f/0                                   memory         8GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered)
/0/2f/1                                   memory         8GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered)
/0/34                                     memory         256KiB L1 cache
/0/35                                     memory         1MiB L2 cache
/0/36                                     memory         8MiB L3 cache
/0/37                                     processor      Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz
/0/100                                    bridge         Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRA
/0/100/1                                  bridge         Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe
/0/100/1/0               /dev/fb0         display        Polaris 22 [Radeon RX Vega M GH]
/0/100/1/0.1                              multimedia     Advanced Micro Devices, Inc. [AMD/ATI]
/0/100/1.1                                bridge         Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe
/0/100/1.1/0                              bus            ASMedia Technology Inc.
/0/100/1.1/0/0           usb3             bus            xHCI Host Controller
/0/100/1.1/0/1           usb4             bus            xHCI Host Controller
...
#
sudo lshw -html > HardwareSummary.html

Finding Number of Ram Slots

sudo dmidecode -t memory

sudo lshw -class memory
編譯與開發工具

Install

# RedHat/CentOS 6
yum install make libtool autoconf subversion git cvs wget libogg-devel gcc gcc-c++ pkgconfig

# RedHat/CentOS 7
yum group install "Development Tools"
dd
# 備份 MBR
dd if=/dev/hdx of=/path/to/image count=1 bs=512

# 光碟轉成 iso 檔
dd if=dev/cdrom of=/root/cd.iso

# 銷毀硬碟資料
dd if=/dev/urandom of=/dev/hda1

# 備份整個 USB-Flash
dd if=/dev/sdb | gzip > ./my-usb_flash.img.gz

# 還原 USB-Flash
gzip -dc ./my-usb_flash.img.gz | dd of=/dev/sdb

# 建立一個測試用的大檔案10GB
dd if=/dev/zero of=/path/to/image bs=1G count=10
# NOTE: 新版 Linux 可以改用指令
fallocate -l 1G test.img 

# Test network bandwidth between 2 Linux servers
dd if=/nas-mount-point/samplefile of=/dev/null bs=1M count=1024 iflag=direct
dd if=/dev/zero of=/nas-mount-point/samplefile bs=1M count=1024 oflag=direct 
# NOTE: the samplefile is greater than 1GB and the RAM is preferably more than 2GB.
history

See time stamp in bash history

echo 'export HISTTIMEFORMAT="%F %T "' >> ~/.bash_profile
系統效能管理

使用 Swap 的 processes

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
Swap 管理
# 檢查目前的 swap 配置
free
swapon -s

# 開啟/關閉 swap
swapon /dev/sda3
swapoff /dev/sda3

# 製作一個 swap 系統
mkswap /dev/sda3 

make-swapfile.sh:1GB

#!/bin/bash

dd if=/dev/zero of=/swapfile bs=1024 count=1024k
chown root:root /swapfile
chmod 0600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile          swap            swap    defaults        0 0" >> /etc/fstab
sysctl vm.swappiness=10
echo vm.swappiness=10 >> /etc/sysctl.conf
free -h
cat /proc/sys/vm/swappiness

Extend the existing SWAP partition:

[root@tycitpdb05-a ~]# grep -i swap /etc/fstab
/dev/mapper/rootvg-swap swap                    swap    defaults        0 0

[root@tycitpdb05-a ~]# ls -al /dev/mapper/rootvg-swap
lrwxrwxrwx 1 root root 7 Aug 15 11:49 /dev/mapper/rootvg-swap -> ../dm-1

[root@tycitpdb05-a ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4194300 0       -2

[root@tycitpdb05-a ~]# swapoff -v /dev/mapper/rootvg-swap
swapoff /dev/mapper/rootvg-swap

[root@tycitpdb05-a ~]# lvextend -L16G /dev/rootvg/swap
  Size of logical volume rootvg/swap changed from 4.00 GiB (1024 extents) to 16.00 GiB (4096 extents).
  Logical volume rootvg/swap successfully resized.
  
[root@tycitpdb05-a ~]# mkswap /dev/rootvg/swap
mkswap: /dev/rootvg/swap: warning: wiping old swap signature.
Setting up swapspace version 1, size = 16777212 KiB
no label, UUID=06aac9ae-9e8c-48bd-9f16-5f3d0c32b31f

[root@tycitpdb05-a ~]# swapon -v /dev/rootvg/swap
swapon /dev/rootvg/swap
swapon: /dev/mapper/rootvg-swap: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/mapper/rootvg-swap: pagesize=4096, swapsize=17179869184, devsize=17179869184
xfs 檔案系統

修復 xfs 檔案系統

sudo mount -a
mount: /data: mount(2) system call failed: Structure needs cleaning.

sudo umount /data

# with '-n' option to perform a dry run
sudo xfs_repair -n /dev/sdb1
# repair the filesystem
sudo xfs_repair /dev/sdb1

xfs 備份與回復

# Install the xfsdump
dnf install xfsdump

# Create a full-backup
# -L: for dump session
# -M: for media in drive
# -f: the backup destination
xfsdump -L session_0 -M datapart -f /data/boot.xfsdump /boot

# Create a incremental backup with the level 1
# -l: the backup level (0-9)
xfsdump -l 1 -L session_1 -M datapart -f /data/boot.xfsdump1 /boot

# Restore a full-backup
xfsrestore -f /data/boot.xfsdump /test

# Restore an incremental backup
xfsrestore -r -f /data/boot.xfsdump /test
xfsrestore -r -f /data/boot.xfsdump1 /test
Linux Module
監視檔案與目錄異動
比對兩個檔案

列出 A.txt 中没有包含 B.txt 内容的行

diff A.txt B.txt | grep "^<" | cut -c3-
比對兩個目錄
# tree
tree dir1
tree dir2

# diff
diff -q /path/to/dir1 /path/to/dir2
diff -q dir1 dir2
diff -qr dir1 dir2
diff -qrs dir1 dir2
複製整個目錄架構,但不要包含檔案
# tree
tree -dfi --noreport dir1
tree -dfi --noreport dir1 | xargs -I{} mkdir -p "$HOME/Downloads/{}"
tree -a $HOME/Downloads/dir1

# find + xargs
find dir1 -type d
find dir1 -type d | xargs -I{} mkdir -p "$HOME/Documents/{}" 
tree -a $HOME/Documents/dir1

# find + exec
find dir1 -type d -exec mkdir -p "$HOME/Desktop/{}" \;
記憶體管理
# Using meminfo
cat /proc/meminfo
grep -E --color 'Mem|Cache|Swap' /proc/meminfo
# Using free
free -h
# Repeat printing free command output every N seconds.
free -s 5 -c 10
# Using vmstat
vmstat -w
特殊字元的檔名

Dash

# The filename with -- or -
rm -i -v -- -foo
rm -i -v -- --foo
rm -i -v ./-foo

# The filename with -- and whitespaces
rm -i -v -- '-- My Resume . txt'
rm -i -v -- '/path/to/dir/-- My Resume . txt'
rm -i -v -- "/path/to/dir/-- My Resume . txt"

# Using find
find . -name '--my-FileNameGoes-Here' -delete
find /path/to/directory/ -name '---filename with a white spaces' --delete
cp: 強制覆蓋多檔案
# 因為內建 alias=cp -i,以致於 -f 參數無效
yes | cp -r /source /target


Q & A

ARP Cache 不會更新 (Send out Gratuitous ARP)

問題說明:兩部相同規格的 Linux 主機,平時互作備援,網路設定各有一個固定 IP 與共用一個 VIP,VIP 使用 Alias IP 方式。每次移動 VIP 至另一部主機時,都會遇到其他鄰近的不同 vLAN 的主機無法 ping 這 VIP,原因是它們的 Switch 設備與 Core Switch 不會立即更新 ARP Cache,直到 在那些 Switch 上手動清除舊的 ARP 紀錄。

解決方案:要讓 Core Switch 立即更新 ARP Cache,可以再切換 VIP 後,從目的端 Linux 主機上執行任一個指令

arping -U -c 10 -I eth0:1 your.vip.address
arping -A -c 10 -I eth0:1 your.vip.address

其他解決方案:

啟動 VIP 時,使用 ifup 指令可以使主機傳送 Gratuitous ARP Request 更新訊號給 Switch

ifconfig eth0:1 10.4.1.110/24
ifup eth0:1

或者以下動作也可能可以傳送 Gratuitous ARP Request。

NFS 遠端目錄無法卸載

當遭遇某些異常情況時,原先的 NFS 目錄可能無法用指令 umount 正常卸載,且主機因為還在營運生產中,不能執行重開機時,下述的指令可能會有幫助。

# 檢查那些程序及用戶使用該目錄
fuser -m -v <mount_point>
lsof | grep <mount_point>
lsof +f -- <path-to-dir>

# 刪除無效的程序
fuser -i -k <mount_point>

# 以上都無效,可重啟 NFS Client daemon,但其他 NFS 目錄也會被迫中斷。
service netfs stop
service netfs start

# 網友提供的另一個方法
umount -l <mount_point>
檢查線上 NFS 掛載參數, nfs 版本
cat /proc/mounts | grep nfs 
Can't mount remotely Linux host on AIX

Try to run the following commands on your AIX.

nfso -o nfs_use_reserved_ports=1
nfso -o portcheck=1
Remove Shared Memory
# see all shared memory segments
# status: dest -> destroyed
# nattach: how many application being attached to the shared memory. 
ipcs -m
ipcs -m -i <shmid>

# remove the shared memory segment
ipcrm shm <shmid>
ipcrm -m <shmid>
CentOS Upgrade Kernel: linux-firmware package files conflicts

說明:要升級 CentOS 7.6 Kernel to CentOS 7.7,遇到 linux-firmware 套件衝突問題。

rpm -ivh kernel-3.10.0-1062.el7.x86_64.rpm

warning: kernel-3.10.0-1062.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
error: Failed dependencies:
        linux-firmware >= 20190429-72 is needed by kernel-3.10.0-1062.el7.x86_64
        

rpm -ivh linux-firmware-20190429-72.gitddde598.el7.noarch.rpm kernel-3.10.0-1062.el7.x86_64.rpm

warning: linux-firmware-20190429-72.gitddde598.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
warning: kernel-3.10.0-1062.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                          ################################# [100%]
        file /usr/lib/firmware/amd-ucode/microcode_amd_fam17h.bin from install of linux-firmware-20190429-72.gitddde598.el7.noarch conflicts with file from package linux-firmware-20180911-69.git85c5d90.el7.noarch
        file /usr/lib/firmware/amd-ucode/microcode_amd_fam17h.bin.asc from install of linux-firmware-20190429-72.gitddde598.el7.noarch conflicts with file from package linux-firmware-20180911-69.git85c5d90.el7.noarch
        
# Solution:
rpm -Uvh linux-firmware-20190429-72.gitddde598.el7.noarch.rpm
rpm -ivh kernel-3.10.0-1062.el7.x86_64.rpm
Adjust reserved 5% of disk space by default

Reduce it to 2%

sudo tune2fs -m2 /dev/sda1
指令 du 與 df 顯示的使用空間有很大的落差

一般來說,問題通常是 df 的使用空間遠大於 du 的使用空間。

原因是 du 不會對開啟中的檔案進行計算,如果有開啟中的大檔案佔用目錄,du 是無法發現的。

解決方法:

  1. 使用指令 lsof /path/to/dir 檢查目錄裡有沒有程式正在執行中
  2. 確實關閉程式後再用 du 檢查一遍

SELinux

安全增強式 Security-Enhanced Linux(SELinux)是一個在內核中實踐的強制存取控制(MAC)安全性機制。SELinux 首先在 CentOS 4 出現,並在其後的 CentOS 發行版本獲得重大改善。這些改善代表用 SELinux 解決問題的方法亦隨著時間而改變。

基本指令

To check if SELinux is enabled

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

# getenforce
Permissive

To temporarily set SELinux to Enforcing/Permissive

# setenforce 1   // Enforcing

# setenforce 0   // Permissive

Permanently change SELinux
Edit the file /etc/selinux/config

## Change this line 
SELinux=disabled
延伸閱讀

 

 

 

patch & diff

單一檔案
# 建立更新檔
diff -uN old.code new.code > patch.file

# 套用更新檔
patch < patch.file

# 回復更新前
patch -RE < patch.file
連續多個檔案
# 建立更新檔
diff -rupN olddir newdir > patch.dir

# 套用更新檔
patch -p1 < patch.dir

# 回復更新前
patch -RE -p1 < patch.dir
找出內容不一致的檔案
# 目錄一: asterisk_orig/
# 目錄二: asterisk_patched/

cd asterisk_orig; find ./ -name "*.c" -exec md5sum -b {} \; > ../asterisk_orig.md5; cd ../
cd asterisk_patched; find ./ -name "*.c" -exec md5sum -b {} \; > ../asterisk_patched.md5; cd ../

diff asterisk_orig.md5 asterisk_patched.md5 

273c273
< 894a111d1efa5901471820e203503039 *./channels/chan_sip.c
---
> 00116baac23473049b5801c9287fb4be *./channels/chan_sip.c

注意:前提是 asterisk_orig 與 asterisk_patched 目錄架構及檔案名稱必須完全一樣,這通常用來找出 asterisk_patched 目錄內被更新過了哪些 .c 檔案。

diff 與 curl 的應用
diff <(curl http://somesite/file1) <(curl http://somesite/file2)
Tutorials

Subnet mask v.s. CIDR


Netmask                  Netmask (binary)             CIDR     Notes    
 _____________________________________________________________________________ 
255.255.255.255  11111111.11111111.11111111.11111111  /32  Host (single addr) 
255.255.255.254  11111111.11111111.11111111.11111110  /31  Unuseable 
255.255.255.252  11111111.11111111.11111111.11111100  /30    2  useable 
255.255.255.248  11111111.11111111.11111111.11111000  /29    6  useable 
255.255.255.240  11111111.11111111.11111111.11110000  /28   14  useable 
255.255.255.224  11111111.11111111.11111111.11100000  /27   30  useable 
255.255.255.192  11111111.11111111.11111111.11000000  /26   62  useable 
255.255.255.128  11111111.11111111.11111111.10000000  /25  126  useable 
255.255.255.0    11111111.11111111.11111111.00000000  /24 "Class C" 254 useable  
255.255.254.0    11111111.11111111.11111110.00000000  /23    2  Class C's 
255.255.252.0    11111111.11111111.11111100.00000000  /22    4  Class C's 
255.255.248.0    11111111.11111111.11111000.00000000  /21    8  Class C's 
255.255.240.0    11111111.11111111.11110000.00000000  /20   16  Class C's 
255.255.224.0    11111111.11111111.11100000.00000000  /19   32  Class C's 
255.255.192.0    11111111.11111111.11000000.00000000  /18   64  Class C's 
255.255.128.0    11111111.11111111.10000000.00000000  /17  128  Class C's 
255.255.0.0      11111111.11111111.00000000.00000000  /16  "Class B"       
255.254.0.0      11111111.11111110.00000000.00000000  /15    2  Class B's 
255.252.0.0      11111111.11111100.00000000.00000000  /14    4  Class B's 
255.248.0.0      11111111.11111000.00000000.00000000  /13    8  Class B's 
255.240.0.0      11111111.11110000.00000000.00000000  /12   16  Class B's 
255.224.0.0      11111111.11100000.00000000.00000000  /11   32  Class B's 
255.192.0.0      11111111.11000000.00000000.00000000  /10   64  Class B's 
255.128.0.0      11111111.10000000.00000000.00000000  /9   128  Class B's 
255.0.0.0        11111111.00000000.00000000.00000000  /8   "Class A"   
254.0.0.0        11111110.00000000.00000000.00000000  /7 
252.0.0.0        11111100.00000000.00000000.00000000  /6 
248.0.0.0        11111000.00000000.00000000.00000000  /5 
240.0.0.0        11110000.00000000.00000000.00000000  /4 
224.0.0.0        11100000.00000000.00000000.00000000  /3 
192.0.0.0        11000000.00000000.00000000.00000000  /2 
128.0.0.0        10000000.00000000.00000000.00000000  /1 
0.0.0.0          00000000.00000000.00000000.00000000  /0   IP space

subnet_table.jpg

IPv4_Subnet.jpg


ps

以 PID 查詢
ps -fp <PID>
ps -fp <PID#1>,<PID#2>,<PID#3> 
分析 CPU/Memory 使用

CPU 使用率最高前10排名

# for CentOS
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -11

ps aux | head -1; ps aux | sort -rn +2 | head -10
ps -auxf | sort -nr -k 3 | head -10

# for AIX 7.2
ps aux | head -1; ps aux | sort -rn -k 3 | head -10

Memory 使用率最高前10排名

# for CentOS
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head -11

ps aux | head -1; ps aux | sort -rn +3 | head
ps aux | head -1; ps aux | sort -rn -k 4 | head
ps -auxf | sort -nr -k 4 | head -10

ps -eo cmd,pid,ppid,%mem,%cpu --sort=-%mem | head -n 6

ps aux  | awk '{print $6/1024 " MB\t\t" $11}'  | sort -rn | head -25
優先權最高的程序
ps -eakl | sort -n +6 | head
# or
ps -eal | sort -n -k 7 | head
程序以 nice 值排序
ps -eakl | sort -n +7

ps -eal | sort -n -k 8
程序以執行時間排序
ps vx | head -1;ps vx | grep -v PID | sort -rn +3 | head -10

ps vx | head -1;ps vx | grep -v PID | sort -rn -k 4 | head -10
程序以 I/O 排序
ps vx | head -1; ps vx | grep -v PID | sort -rn +4 | head -10

ps vx | head -1; ps vx | grep -v PID | sort -rn -k 5 | head -10
等待中的程序
ps vg | head -1; ps vg | grep -w wait
指定 PID 的啟動時間與總執行時間
ps -p <PID> -o etime,start
ps -p <PID> -o etimes
每個用戶的程序數量
ps -ef | awk '{print$1}' | sort | uniq -c | sort -nr
顯示用戶 vivek 的所有程序
ps -U vivek -u vivek u
指定要顯示的欄位
ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm 
ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm 
ps -eopid,tt,user,fname,tmout,f,wchan
顯示程序 lighttpd 的 pid
ps -C lighttpd -o pid=
顯示 pid XXX 的程序名稱
ps -p 55977 -o comm=

 

Rsync

限制傳檔的網路頻寬
--bwlimit=30000

30000 = 30000 KB/ps = 30 MB/ps

本機內不同個資料夾做複製或同步
rsync -av --delete --exclude='path1/to/exclude' --exclude='path2/to/exclude' source destination
遠端複製大檔案需要續傳
rsync --partial --progress --rsh=ssh <local_file> user@host:<remote_file>
複製 Linux 整個系統
mount /dev/sdb1 /mnt
rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt
How does rsync work

https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/ 

Other similar solutions


screen

screen 常用指令
啟用 screen
#> screen
or
#> screen -S <session-name>

列出所有 session
#> screen -ls

取回 (resume) 某個 session
#> screen -r <session-id>

無法取回 session 時,先 deattach 在 resume
#> screen -D <session-id>
#> screen -r <session-id>

強制終止 session
#> screen -X -S <session-id> kill

結束 screen 的 session
#> exit

進入複製模式,可複製視窗上指定的文字,操作方式:
使用方向鍵任意移動游標,按一次 space 開始標記游標的文字,再按一次 space 結束標記文字並離開複製模式。
Ctrl-a + [
要貼上剛複製的內容
Ctrl-a + ]  
screen 用來連接 RS232
screen /dev/ttyACM0 115200

TIP: 如果出現 screen is terminating,須改用 sudo。

環境配置檔

~/.screenrc:

# Start message
startup_message off

# Disable vbell
vbell off

# Set hardstatus always on
hardstatus alwayslastline " %-Lw%{= Bw}%n%f %t%{-}%+Lw %=| %M %d %0c:%s "

# Set default encoding using utf8
defutf8 on

# 
term xterm

# Set the number of scrollback lines
#defscrollback 10000

# Fix termcapinfo for xterm to allow column resizing
# xterm emulation is used by PuTTY
# Uncomment the line below if running on CentOS 6.x/7.x 
#termcapinfo  xterm Z0=\E[?3h:Z1=\E[?3l:is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l

# Fix termcapinfo for cleaning the terminal after detach
# Uncomment the line below if running on AIX 7.x
#termcapinfo xterm*|rxvt*  te=\E[?1049l:ti=\E[?1049h:

TIPs:

start_message 關閉開啟後的提示訊息

vbell                      關閉可以避免按 tab 出現閃畫面

hardstatus        在畫面下方固定顯示視窗編號,非常實用的功能

defscrollback  回捲內容的行數

進入 screen 模式後指令

基本操作

離開 screen 但不中止 session (deattach)

Ctrl-a + d

離開 screen 並中止 session

Ctrl-a + k

複製模式

進入複製模式,可複製視窗上指定的文字,操作方式:
使用方向鍵任意移動游標,按一次 space 開始標記游標的文字,再按一次 space 結束標記文字並離開複製模式。

Ctrl-a + [

貼上剛複製的內容

Ctrl-a + ]  

多視窗操作

水平切割 <Ctrl-a> + |

垂直切割 <Ctrl-a> + S

切換視窗 <Ctrl-a> + tab

目前視窗最大化 <Ctrl-a> + Q

移除目前視窗 <Ctrl-a> + X     ;這不會終止 session

多個終端機操作

新增一個 screen 終端機

<Ctrl-a> + c

或者是在目前的 screen 終端機再執行一次 screen
NOTE:如果已經 deattach,再執行 screen 時,會開啟另一個不同的 screen 終端機

檢視目前已開啟的所有視窗

<Ctrl-a> + w

移動至前一個或後一個終端機

<Ctrl-a> + p 前一個 (視窗編號減一)

<Ctrl-a> + n 後一個 (視窗編號加一)

<Ctrl-a> + 視窗編號

Q & A

用 Putty 連線,每次 attach 時,視窗都會自動縮回預設大小

將設定檔以下這行移除註解

termcapinfo  xterm Z0=\E[?3h:Z1=\E[?3l:is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l 
用 tab 時,畫面都會閃一下,該如何解決

在 $HOME 目錄內新增 .screenrc

vbell off
如何很快的確認,目前是否在 screen 終端機內

Ctrl + a + a

Learning 拾人牙慧

系統管理 
檔案救援
FTP/SFTP
RedHat
檔案加密
Web Terminal
GlusterFS - 一個 Open Source 可以橫向擴充的檔案儲存解決方案
網路管理
Load Balancing
Hardening Security
Files and Directories Monitoring
GPU Monitoring
System Log

logwatch

Secure Boot
YubiKey with Linux




fsck

教學連結

Cheat Sheets

Bash
Curl
Git

Git_cheat_sheet.jpeg