iSCSI and multipath on CentOS 7
系統環境
- iSCSI Storage: NetApp FAS8200
- Linux OS: CentOS 7
- Server H/W: HP DL380 Gen10
- iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2
設定網路卡
使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask
注意: 這兩張網卡不要作 Bonding。
設定 iSCSI
安裝 iscsi 套件
yum install iscsi-initiator-utils設定 iscsid.conf: /etc/iscsi/iscsid.conf
# 調整 timeout
#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5
# 調整 node 自動 Login
node.startup = automatic設定 initiator name: /etc/iscsi/initiatorname.iscsi
命名規則可以參考
- iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk - initiator name 必須與 Storage 端的設定相同
- 不同的 Host 之間的名稱不能相同
安裝 multipath 與 netapp_linux_unified_host_utilities
yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm如果沒有安裝 libhbaapi，執行 sanlun 會出現錯誤。
連接 iSCSI storage
新增網卡裝置
iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=newDiscovery for the storage
iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260如果 NetApp storage 有設置多個 iSCSI IPs，只需要對其中一個 IP 作偵測，輸出結果就會顯示所有 IP。
[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16Login nodes
iscsiadm -m node -l all輸出結果會顯示多行如下資訊
[root@bdb7-b RPMs]# iscsiadm -m node -l all
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.使用 sanlun 確認 LUN Disks
[root@bdb7-b RPMs]# sanlun lun show
controller(7mode/E-Series)/ device host lun
vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdag host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdaf host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdae host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdad host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdac host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdab host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdz host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdaa host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdy host14 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdx host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdw host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdt host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdu host15 iSCSI 500.1g cDOT
...設定 multipath
初始化後會自動產生設定檔 multipath.conf
mpathconf --enable --with_multipathd y
建立 device alias name 與 LUN WWID 的對應
注意：在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱，而只會顯示 WWID，詳細原因請見附檔。為了爾後方便管理，必須手動建立磁碟名稱與 WWID 對應。
編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行
...
multipaths {
multipath {
wwid 3600a0980383136312d3f4e387455564b
alias mpath1
}
multipath {
wwid 3600a0980383136312d3f4e387455564c
alias mpath2
}
multipath {
wwid 3600a0980383136312d3f4e387455564d
alias mpath3
}
multipath {
wwid 3600a0980383136312d3f4e387455564e
alias mpath4
}
}wwid 可以從指令 multipath -l 取得
alias 可以設定任一名稱，LUN disks的順序可以從指令 sanlun lun show 取得
載入新設定
systemctl reload multipathd執行 multipath -l，應該要顯示 WWID 與 Alias Name
mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 12:0:0:1 sdo 8:224 active undef running
| |- 13:0:0:1 sds 65:32 active undef running
| |- 15:0:0:1 sdaa 65:160 active undef running
| `- 16:0:0:1 sdae 65:224 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
|- 10:0:0:1 sdg 8:96 active undef running
|- 11:0:0:1 sdk 8:160 active undef running
|- 14:0:0:1 sdw 65:96 active undef running
`- 9:0:0:1 sdc 8:32 active undef running開機掛載 iSCSI LUNs