SAN & iSCSI
SAN - Storage Area Network, iSCSI - Internet Small Computer System Interface
iSCSI and multipath on CentOS 7
系統環境
- iSCSI Storage: NetApp FAS8200
- Linux OS: CentOS 7
- Server H/W: HP DL380 Gen10
- iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2
設定網路卡
使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask
注意: 這兩張網卡不要作 Bonding。
設定 iSCSI
安裝 iscsi 套件
yum install iscsi-initiator-utils
設定 iscsid.conf: /etc/iscsi/iscsid.conf
# 調整 timeout
#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5
# 調整 node 自動 Login
node.startup = automatic
設定 initiator name: /etc/iscsi/initiatorname.iscsi
命名規則可以參考
- iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk - initiator name 必須與 Storage 端的設定相同
- 不同的 Host 之間的名稱不能相同
安裝 multipath 與 netapp_linux_unified_host_utilities
yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm
如果沒有安裝 libhbaapi,執行 sanlun 會出現錯誤。
連接 iSCSI storage
新增網卡裝置
iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=new
Discovery for the storage
iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260
如果 NetApp storage 有設置多個 iSCSI IPs,只需要對其中一個 IP 作偵測,輸出結果就會顯示所有 IP。
[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
Login nodes
iscsiadm -m node -l all
輸出結果會顯示多行如下資訊
[root@bdb7-b RPMs]# iscsiadm -m node -l all
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
使用 sanlun 確認 LUN Disks
[root@bdb7-b RPMs]# sanlun lun show
controller(7mode/E-Series)/ device host lun
vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdag host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdaf host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdae host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdad host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdac host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdab host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdz host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdaa host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdy host14 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdx host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdw host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdt host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdu host15 iSCSI 500.1g cDOT
...
設定 multipath
初始化後會自動產生設定檔 multipath.conf
mpathconf --enable --with_multipathd y
建立 device alias name 與 LUN WWID 的對應
注意:在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱,而只會顯示 WWID,詳細原因請見附檔。為了爾後方便管理,必須手動建立磁碟名稱與 WWID 對應。
編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行
...
multipaths {
multipath {
wwid 3600a0980383136312d3f4e387455564b
alias mpath1
}
multipath {
wwid 3600a0980383136312d3f4e387455564c
alias mpath2
}
multipath {
wwid 3600a0980383136312d3f4e387455564d
alias mpath3
}
multipath {
wwid 3600a0980383136312d3f4e387455564e
alias mpath4
}
}
wwid 可以從指令 multipath -l 取得
alias 可以設定任一名稱,LUN disks的順序可以從指令 sanlun lun show 取得
載入新設定
systemctl reload multipathd
執行 multipath -l,應該要顯示 WWID 與 Alias Name
mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 12:0:0:1 sdo 8:224 active undef running
| |- 13:0:0:1 sds 65:32 active undef running
| |- 15:0:0:1 sdaa 65:160 active undef running
| `- 16:0:0:1 sdae 65:224 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
|- 10:0:0:1 sdg 8:96 active undef running
|- 11:0:0:1 sdk 8:160 active undef running
|- 14:0:0:1 sdw 65:96 active undef running
`- 9:0:0:1 sdc 8:32 active undef running
開機掛載 iSCSI LUNs
注意:若沒有以下設定,將導致系統無法正常開機。
編輯 /etc/fstab , 在 iSCSI disk 每一行都加上 _netdev
#device mount point FS Options Backup fsck
/dev/myvg/mylv /mydisk xfs _netdev 0 0
設定 netfs 服務自動啟動
## CentOS 7
systemctl enable remote-fs.target
## CentOS 4/5/6
chkconfig netfs on
加大 MTU 設定
網路卡預設 MTU 是 1500,這不適合 iSCSI 的應用,建議加大至 9000。除了 Linux 設定以外,網路連接的 Switch 與 iSCSI Storage (iSCSI Target) 也需要一併做調整。
RedHat 7.x) 檢查目前網路設置是否可接受大 frame 的網路封包傳遞
UNSUCCESSFUL
# ping -M do -c 4 -s 8196 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 8185(8213) bytes of data.
PING 10.1.1.21(10.1.1.21) 8196(8224) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
SUCCESSFUL
# ping -M do -c 4 -s 1408 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 1408(1436) bytes of data.
1416 bytes from 10.1.1.21: icmp_seq=1 ttl=58 time=224 ms
立即調整 MTU (NOTE: 非永久性設置)
ip link set dev ens2f0 mtu 9000
永久設置方法,在 iSCSI 網路裝置的設定檔 ifcfg-ethXX,加上這行:
MTU="9000"
重啟系統後生效。
擴充新硬碟 LUN disk
1. iSCSI Storage 新增 LUN disk 並分派給要使用的 Linux host。
2. 以下步驟都是 Linux host 要完成的。
3. 掃描(discover)剛剛新增的 LUN disk
[root@bdb7-b ~]# /usr/bin/rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: HPE Model: DVDROM DUD0N Rev: UMD1
Type: CD-ROM ANSI SCSI revision: 05
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for all SCSI target IDs, all LUNs
Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 7 0 0 0 ...
4. 使用 sanlun 指令檢查新 LUN disk
[root@bdb7-b ~]# sanlun lun show all
controller(7mode/E-Series)/ device host lun
vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaw host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdav host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdau host14 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdat host13 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdas host12 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdar host11 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaq host10 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdap host9 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdao host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdan host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdam host16 iSCSI 500.1g cDOT
...
5. 使用 multipath 指令檢查新 LUN disk
[root@bdb7-b alang]# multipath -ll
mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:1 sdf 8:80 active ready running
| |- 12:0:0:1 sdt 65:48 active ready running
| |- 16:0:0:1 sdah 66:16 active ready running
| `- 15:0:0:1 sdai 66:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 11:0:0:1 sdg 8:96 active ready running
|- 10:0:0:1 sde 8:64 active ready running
|- 14:0:0:1 sdu 65:64 active ready running
`- 13:0:0:1 sdv 65:80 active ready running
mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:0 sdc 8:32 active ready running
| |- 12:0:0:0 sdr 65:16 active ready running
| |- 16:0:0:0 sdaf 65:240 active ready running
| `- 15:0:0:0 sdag 66:0 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 11:0:0:0 sdd 8:48 active ready running
|- 10:0:0:0 sdb 8:16 active ready running
|- 13:0:0:0 sdq 65:0 active ready running
`- 14:0:0:0 sds 65:32 active ready running
3600a098038313631305d4e2d6f77474a dm-26 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:5 sdap 66:144 active ready running
| |- 12:0:0:5 sdas 66:192 active ready running
| |- 15:0:0:5 sdav 66:240 active ready running
| `- 16:0:0:5 sdaw 67:0 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:5 sdaq 66:160 active ready running
|- 11:0:0:5 sdar 66:176 active ready running
|- 13:0:0:5 sdat 66:208 active ready running
`- 14:0:0:5 sdau 66:224 active ready running
...
6. 設定 LUN WWID 與 mpathX 的對應
編輯 /etc/multipath.conf
: 將新的 wwid 與 alias 加上。
# added by alang
multipaths {
multipath {
wwid 3600a098038313631305d4e2d6f774743
alias mpath1
}
multipath {
wwid 3600a098038313631305d4e2d6f774744
alias mpath2
}
multipath {
wwid 3600a098038313631305d4e2d6f774745
alias mpath3
}
multipath {
wwid 3600a098038313631305d4e2d6f774746
alias mpath4
}
multipath {
wwid 3600a098038313631305d4e2d6f774747
alias mpath5
}
multipath {
wwid 3600a098038313631305d4e2d6f77474a
alias mpath6
}
}
7. 載入 multipath 設定
[root@bdb7-b alang]# systemctl reload multipathd
[root@bdb7-b alang]# multipath -ll
mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:1 sdt 65:48 active ready running
| |- 15:0:0:1 sdai 66:32 active ready running
| |- 16:0:0:1 sdah 66:16 active ready running
| `- 9:0:0:1 sdf 8:80 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:1 sde 8:64 active ready running
|- 11:0:0:1 sdg 8:96 active ready running
|- 13:0:0:1 sdv 65:80 active ready running
`- 14:0:0:1 sdu 65:64 active ready running
mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:0 sdr 65:16 active ready running
| |- 15:0:0:0 sdag 66:0 active ready running
| |- 16:0:0:0 sdaf 65:240 active ready running
| `- 9:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:0 sdb 8:16 active ready running
|- 11:0:0:0 sdd 8:48 active ready running
|- 13:0:0:0 sdq 65:0 active ready running
`- 14:0:0:0 sds 65:32 active ready running
mpath6 (3600a098038313631305d4e2d6f77474a) dm-26 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:5 sdas 66:192 active ready running
| |- 15:0:0:5 sdav 66:240 active ready running
| |- 16:0:0:5 sdaw 67:0 active ready running
| `- 9:0:0:5 sdap 66:144 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:5 sdaq 66:160 active ready running
|- 11:0:0:5 sdar 66:176 active ready running
|- 13:0:0:5 sdat 66:208 active ready running
`- 14:0:0:5 sdau 66:224 active ready running
完成。新加的 disk mpath6 可以用 LVM 進行管理。
其他指令
顯示 disk path 更多資訊
- 主機磁碟名稱與 Storage LUN 名稱對應
- LUN 編號與 Size
- Storage 的 path 狀態資訊
[root@bdb7-a ~]# sanlun lun show -p
ONTAP Path: TPEDBISCSISVM01:/vol/BDB7_A_2_vol/BDB7_A_2
LUN: 1
LUN Size: 500.1g
Product: cDOT
Host Device: mpath2(3600a0980383136312d3f4e387455564c)
Multipath Policy: service-time 0
Multipath Provider: Native
--------- ---------- ------- ------------ ----------------------------------------------
host vserver
path path /dev/ host vserver
state type node adapter LIF
--------- ---------- ------- ------------ ----------------------------------------------
up primary sdo host12 TPEDBISCSISVM01_iSCSI2
up primary sds host13 TPEDBISCSISVM01_iSCSI2
up primary sdaa host15 TPEDBISCSISVM01_iSCSI1
up primary sdae host16 TPEDBISCSISVM01_iSCSI1
up secondary sdg host10 TPEDBISCSISVM01_iSCSI4
up secondary sdk host11 TPEDBISCSISVM01_iSCSI3
up secondary sdw host14 TPEDBISCSISVM01_iSCSI4
up secondary sdc host9 TPEDBISCSISVM01_iSCSI3
查詢 multipathd 的設定參數資訊
multipathd show config
技巧: 連接不同廠牌的 Storage,預設狀態會套用不同的參數組設定,而這些參數的資訊可以從這指令獲得。
查詢目前 iSCSI connections
# iscsiadm -m session --op show
tcp: [1] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [2] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [3] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [4] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [5] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [6] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [7] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [8] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
#iSCSI Logout All
iscsiadm --mode node --logoutall=all
#iSCSI Login All
iscsiadm -m node -l
Learning
iSCSI
Multipath on RedHat (NetApp FAS2240)
Multipath 是 CentOS/RedHat 內建套件,可用作兩 port 的 HBA 卡連接 Storage時,有雙路由的備援架構。
系統環境
- RedHat 4.7
- NetApp FAS2240
- SAN Switch x 2
- HBA Card: QLogic, port x 2
#>lspci
Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
RedHat 的 HBA Card 驅動
建議安裝方法為,在安裝 RedHat 前先安裝 HBA 卡,然後安裝 RedHat 時會自動驅動 HBA 卡。
如果無法重新安裝 RedHat 試試修改 /etc/modprobe.conf,加上這一行
alias scsi_hostadapter2 qla2500
scsi_hostadapterX 編號視實際狀況修改
查出 HBA 卡的 WWN
#> cat /proc/scsi/qla2xxx/2
SCSI Device Information:
scsi-qla0-adapter-node=5001438004253c81;
scsi-qla0-adapter-port=5001438004253c80; <---- WWN
scsi-qla0-target-0=50001fe1500e28fd;
scsi-qla0-target-1=50001fe1500e28f9;
scsi-qla0-target-2=500a09829140a66c;
另一個 HBA port 裝置檔為 /proc/scsi/qla2xxx/3,以此類推
每個 port 有獨立的 WWN
NetApp storage 分配 LUN 以此 WWN 作為 Host Mapping 的主要參數
驅動後的模組列表
# lsmod | grep qla
qla2500 209985 0
qla2xxx 203681 13 qla2500
scsi_transport_fc 19521 1 qla2xxx
scsi_mod 153489 7 ib_srp,qla2xxx,scsi_transport_fc,sr_mod,libata,cciss,sd_mo d
SAN Storge 設定與接線
- 將 NetApp storage、SAN switch、RedHat 接上各自的光纖線
- 找出 RedHat 的 HBA 卡 兩 port 各自的 WWN
- 設定 NetApp,將 LUNs 分配至兩個 WWN Hosts
設定 Multipath
更新:RedHat 客戶中心有提供設定 multipath.conf 線上協助導引,網頁必須先登入才能瀏覽
https://access.redhat.com/labs/multipathhelper/
必要的套件安裝
rpm -q device-mapper
rpm -q device-mapper-multipath
yum install device-mapper
yum install device-mapper-multipath
CentOS 5.x)
編輯 /etc/multipath.conf
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
devnode_blacklist {
# devnode "*" << 註解這一行
}
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes
}
注意:即使啟動了 multipathd 服務,預設是抓不到任何 LUNs 的,必須註解上述的那一行
啟動 multipathd 服務
#> service multipathd start
#> chkconfig multipathd on
CentOS 6/7)
預設沒有設定檔 /etc/multipath.conf,執行這個指令會自動產生,並且將服務 multipathd 設定成自動執行。
#> mpathconf --enable --with_multipathd y
不用修改 multipath.conf
檢視 multupah 的裝置
CentOS 5.x
[root@cdb3-b ~]# multipath -ll
mpath1 (360a98000383035537824474d3936556a) dm-4 NETAPP,LUN
[size=700G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=0][rw]
\_ round-robin 0 [prio=4][active]
\_ 1:0:1:0 sdb 8:16 [active][ready]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:0:0 sda 8:0 [active][ready]
\_ 2:0:0:0 sdc 8:32 [active][ready]
CentOS 6.x
[root@cdb3 ~]# multipath -ll
mpathb (360a98000383035537824474d39365568) dm-2 NETAPP,LUN
size=700G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=4 status=active
| |- 1:0:1:0 sdc 8:32 active ready running
| `- 2:0:1:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
|- 1:0:0:0 sdb 8:16 active ready running
`- 2:0:0:0 sdd 8:48 active ready running
安裝 NetApp 管理工具
*建議*安裝 LUN 管理工具 - NetApp Linux Host Utilities
yum install libnl
rpm -ivh QConvergeConsoleCLI-1.0.01-32.x86_64.rpm
rpm -ivh netapp_linux_host_utilities-6-2.x86_64.rpm
- 如果沒有安裝這套件,將無法使用 sanlun 指令
- 適用 CentOS 5.x/6.x
新增 LUN 需系統重新開機
一旦重新開機後,所有的 LUN disks (在 dual path SAN 架構下會有多個 LUN disks)應該都會被偵測到,並且會有一個 mpathXX 的裝置名稱。
#> multipath -l
mpath1 (360a98000424732615124444150534a35)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
\_ 2:0:2:0 sde 8:64 [active]
\_ 3:0:2:0 sdf 8:80 [active]
mpath0 (3600508b400068e5c00012000001a0000)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
\_ 2:0:0:1 sda 8:0 [active]
\_ 3:0:0:1 sdc 8:32 [active]
\_ round-robin 0 [enabled]
\_ 2:0:1:1 sdb 8:16 [active]
\_ 3:0:1:1 sdd 8:48 [active]
mpathX 這路徑可作為 LVM 的 PV 或直接格式化為 filesystem,完整路徑為 /dev/mapper/mpathX
2:0:0:1 這表示為 host : channel : ID : LUN
Host 2 與 Host 3 表示 HBA 卡有兩個 port 或者單 port 的卡兩張
新增/移除 LUN 系統不需重開機
參考教學:
- https://library.netapp.com/ecmdocs/ECMP1654943/html/GUID-F88A86D2-F3A9-4440-8C32-539BFEA6BED1.html
- https://access.redhat.com/solutions/3941
CentOS/RedHat 5.x 以後
從 CentOS/RedHat 5 以後或者 SLES 10 以後,可以安裝以下套件:
yum install sg3_utils
不停機下掃描新的 LUN Disk
檢視目前現有的 LUN Disks
#> sanlun lun show all
掃描所有 LUN
#> scsi-rescan
NOTE: 如果偵測不到新 LUN,試試加上 -a
#> scsi-rescan -a
再次檢視目前的 LUN Disks
#> sanlun lun show all
移除 LUN 的程序
要移除現有的 LUN disk 必須依照以下步驟完成:
- 執行
umount
卸載 - 執行
lvremove
移除 LV - 需要時,執行
vgremove
移除 VG - 執行
pvremove
移除 PV - 執行
multipath -f mpathXX
移除 multipathXX
NOTE: 如果沒有執行這步驟,一旦移除 LUN 後系統可能會 Hang。 - 從 Storage 管理介面移除這個 LUN disk
NOTE: 當 LUN 被移除後,執行sanlun lun show all
,那些移除的 LUN 會顯示 unknown。 - 執行
scsi-rescan -r
移除所有未使用的 LUN disk
NOTE: 執行後輸出訊息,應該要顯示類似
4 device(s) removed.
[5:0:1:0]
[5:0:3:0]
[6:0:1:0]
[6:0:3:0]
重新執行sanlun lun show all
,那些 unknown 就會消失。
CentOS/RedHat 4.x:
使用附檔區的兩個 script
- remove-LUNs.sh
- rescan-LUNs.sh
新增 LUN disk 步驟
- 在 NetApp storage 上完成 LUN Mapping 的設定
- 在 RedHat 上執行
rescan-LUNs.sh
- 在 RedHat 上執行
multipath -l
移除現有 LUN disk
- 在 RedHat 上卸載不用的 mpathX
- 在 RedHat 上執行
sanlun lun show all
找出 LUN 的名稱與 NetApp 上路徑
註:要執行 sanlun 指令必須先安裝 NetApp 的套件 netapp_linux_host_utilities-6-2.x86_64.rpm - 在 NetApp 上移除所對應的 LUN
- 在 RedHat 上執行
multipath -l
- 找出要移除 LUN disks 的 host:channel:id:LUN
- 執行
remove-LUNs.sh <host> <channel> <id> <LUN>
現有 LUN disk 擴充空間
1. 確認 LUN ID 與 sd[X] 的對應
smartctl -a /dev/sd[X]
2. 掃描 LUN disk,確認最新容量
echo 1 > /sys/block/sd[X]/device/rescan
lsblk | grep sd[X]
3. Resize PV
pvresize /dev/sd[X]
4. 完成。接著就按 SOP 進行 LV 與 Filesystem 的擴充。
LVM 管理
新增 LUN disk 成一個 PV
#> pvcreate /dev/mapper/mpath6
PV 新增後,執行 pvdisplay 時的裝置名稱會是 dm-X。
新增一個 VG (需指定 PV)
#> vgcreate dbVG /dev/mapper/mpath6
指令更多用法
multipath
/etc/init.d/multipathd start | Manually starts the multipathing service on the Linux host. |
multipath -ll | Displays the multipathing configuration. |
multipath -v2 | Creates the initial multipathing configuration. |
multipath -f device_map | Removes an existing LUN from the multipathing configuration. |
multipath -ll -v2 -d | Displays the multipathing configuration in verbose mode. |
multipath -t 或 multipathd show config | 顯示目前的設定參數 |
Accesing LUNs on host
sanlun fcp show adapter -v | Displays HBA information. |
sanlun fcp show adapter | Display WWPN of HBA cards |
sanlun lun show all | Lists available LUNs as seen by the sanlun script. |
sanlun lun show -p | Provides multipath information for both DM-Multipath and Veritas Storage Foundation TM dynamic multipathing solutions. |
ls -lh /dev/mapper/ | Lists DM-Multipath devices generated for LUNs. |
e2label device label | Creates a persistent label for a file system. device is the partition; label is a unique string up to 16 characters. Not needed with DM-Multipath. |
Q & A
Q: RHEL 7 沒有使用 alias path
使用 NetApp Storage,執行 multipath -ll
360a98000443437576f2444726e556d76 dm-2 NETAPP ,LUN
size=800G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 6:0:3:0 sdc 8:32 active ready running不會顯示 alias name: mpathXX
參閱官方手冊:https://access.redhat.com/solutions/2061463
檢視 multipath 可支援的裝置清單
multipath -t
沒有看到 NetApp 的裝置。
解決方法:編輯 multipath.conf,加入 NetApp 的參數設定
NOTE: 僅適合 NetApp Storage。
defaults {
user_friendly_names yes
find_multipaths yes
}
devices {
device {
vendor "NETAPP"
product "LUN.*"
path_grouping_policy "group_by_prio"
path_checker "tur"
features "3 queue_if_no_path pg_init_retries 50"
hardware_handler "0"
prio "ontap"
failback immediate
rr_weight "uniform"
rr_min_io 128
flush_on_last_del "yes"
dev_loss_tmo "infinity"
user_friendly_names yes
retain_attached_hw_handler yes
detect_prio yes
}
載入新設定
#> systemctl reload multipathd
#> multipath -ll
mpatha (360a98000443437576f2444726e556d76) dm-2 NETAPP ,LUN
size=800G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 4:0:0:0 sdb 8:16 active ready running
| `- 6:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 4:0:3:0 sdc 8:32 active ready running
`- 6:0:3:0 sde 8:64 active ready running
Q: 安裝 netapp_linux_host_utilities 時,出現錯誤訊息
Warning: libnl.so library not found, some sanlun commands may not work. Refer Li nux Host Utilities Installation and Set up Guide for more details
Warning: libHBAAPI.so library not found, some sanlun commands may not work. Refe r Linux Host Utilities Installation and Setup Guide for more details
Solution: 先安裝 libnl
yum install libnl
再安裝以下套件任一個
- http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/ResourceByOS.aspx?productid=1043&oemid=372&oemcatid=111969
- http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByOs.aspx?ProductCategory=39&OsCategory=2&Os=65&OsCategoryName=Linux&ProductCategoryName=Fibre+Channel+HBAs&OSName=Linux+Red+Hat+%2864-bit%29
- QConvergeConsole CLI (x64)
- SANsurfer CLI (x86/x64)
Q: 執行 pvscan、vgscan 或開機時出現以下訊息
Found duplicate PV XXXXXXXXXXXXXXXX: using /dev/sdb not /dev/sda
Solution:
這通常會發生在系統有作 multipath 與 LVM 設定後,原因是 CentOS 4/5 預設 LVM 的設定會掃描所有的磁碟裝置,然而,當系統設定有 multipath 時,LVM 應該管理 mpathX 或 dm-X 的裝置,而不再是 sdX 裝置。
要使 LVM 不再對 sdX 裝置作管理,可以修改 /etc/lvm/lvm.conf,此外,如果要使每次系統開機時,LVM 不要對 sdX 裝置作掃描,還必須要重建 initrd 開機映像檔,並且修改 grub.conf 相關的設定。
操作步驟如下:
1. 編輯 /etc/lvm/lvm.conf
...
filter = [ "r|/dev/sd.*|", "a|/.*|" ]
...
r|/dev/sd.*| 這設定會排除所有 sda, sdb, ...等所有磁碟裝置掃描,如果系統有使用其中某個裝置,必須自行修改這規則。
存檔離開
執行 pvscan 或 vgscan,如果不再出現 Warning,繼續步驟 2。
2. 重建 initrd 開機映像檔
#> cd /boot
#> mkinitrd -f -v /boot/initrd-$(uname -r).multipath.img $(uname -r)
編輯 /boot/grub/grub.conf
複製 title 段落的所有行,並依照實際狀況修改
title CentOS (2.6.18-398.el5-Multipath) <== 加上 Multipath
root (hd0,0)
kernel /vmlinuz-2.6.18-398..........
initrd /initrd-2.6.18-398.el5.multipath.img <== 重建後的initrd 開機映像檔名
存檔離開
3. 重新開機後,出現開機選單後,選擇第二個開機項目。
如果開機訊息不再出現警告文字,而且系統一切正常,就可以再修改 grub.conf 的 default=1。
FAQ
如何確認 /dev/sda 是 Local Disk 還是 SAN DIsk
Solution:
#> lsscsi
[2:0:0:0] cd/dvd hp DVD-ROM DTA0N WBD0 /dev/sr0
state=running queue_depth=1 scsi_level=6 type=5 device_blocked=0 timeout=0
[3:0:0:0] tape IBM ULT3580-HH5 D2AD /dev/st0
state=running queue_depth=32 scsi_level=7 type=1 device_blocked=0 timeout=900
[3:0:1:0] tape IBM ULT3580-HH5 D2AD /dev/st1
state=running queue_depth=32 scsi_level=7 type=1 device_blocked=0 timeout=900
[3:0:1:1] mediumx IBM 3573-TL C.20 -
state=running queue_depth=32 scsi_level=6 type=8 device_blocked=0 timeout=0
[3:0:2:0] disk IBM 1726-4xx FAStT 0617 /dev/sda <======
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
[3:0:2:31] disk IBM Universal Xport 0617 -
state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60
IBM FAStT 是 IBM Storage 產品識別
如果沒有 lsscsi 指令,執行 yum install lsscsi
如何找出 HBA Card 的 port WWN
方法一: 用 systool
# Install
yum install sysfsutils
#
systool -c fc_host -v
systool -c fc_transport -v
方法二:/sys/class/fc_host/{host#}/port_name
#> cat /sys/class/fc_host/host5/port_name
0x2100001b32837bcd
方法三:/proc/scsi/{adapter-type}/{host#}
#> cat /proc/scsi/qla2xxx/2
SCSI Device Information:
scsi-qla0-adapter-node=5001438004253c81;
scsi-qla0-adapter-port=5001438004253c80; <---- WWN
scsi-qla0-target-0=50001fe1500e28fd;
scsi-qla0-target-1=50001fe1500e28f9;
scsi-qla0-target-2=500a09829140a66c;
mpathX v.s. dm-X
如果有啟用 multipath,所有 LUN 的裝置路徑為 /dev/mapper/mpathX,不過當使用 pvcreate 將 LUN 新增為 PV 後,執行 pvdisplay -C ,此時的裝置名稱會改為 /dev/dm-X,那爾後作系統維護時,要如何查詢 mpathX 對應哪個 dm-X?
方法一:dmsetup
[root@bdb3 ~]# dmsetup ls
mpath1 (253, 4)
rootVG-varLV (253, 1)
rootVG-swapLV (253, 3)
rootVG-rootLV (253, 0)
rootVG-worktmpLV (253, 2)
mpath1 (253,4),4 就是指裝置 /dev/dm-4
方法二:multipath
[root@cdb3-b ~]# multipath -ll
mpath1 (360a98000383035537824474d3936556a) dm-4 NETAPP,LUN <====== 這裡
[size=700G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=0][rw]
\_ round-robin 0 [prio=4][active]
\_ 1:0:1:0 sdb 8:16 [active][ready]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:0:0 sda 8:0 [active][ready]
\_ 2:0:0:0 sdc 8:32 [active][ready]
新 partition 無法使用
對 mpathb 新建 partition 後,無法立即被使用,必須重開機後才能生效
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
不重開機要能立即使用,可以再切完 partition 後,執行
partprobe /dev/mapper/mpathb