iSCSI and multipath on CentOS 7
系統環境
- iSCSI Storage: NetApp FAS8200
- Linux OS: CentOS 7
- Server H/W: HP DL380 Gen10
- iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2
設定網路卡
使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask
注意: 這兩張網卡不要作 Bonding。
設定 iSCSI
安裝 iscsi 套件
yum install iscsi-initiator-utils設定 iscsid.conf: /etc/iscsi/iscsid.conf
# 調整 timeout
#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5
# 調整 node 自動 Login
node.startup = automatic設定 initiator name: /etc/iscsi/initiatorname.iscsi
命名規則可以參考
- iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk - initiator name 必須與 Storage 端的設定相同
- 不同的 Host 之間的名稱不能相同
安裝 multipath 與 netapp_linux_unified_host_utilities
yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm如果沒有安裝 libhbaapi，執行 sanlun 會出現錯誤。
連接 iSCSI storage
新增網卡裝置
iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=newDiscovery for the storage
iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260如果 NetApp storage 有設置多個 iSCSI IPs，只需要對其中一個 IP 作偵測，輸出結果就會顯示所有 IP。
[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16Login nodes
iscsiadm -m node -l all輸出結果會顯示多行如下資訊
[root@bdb7-b RPMs]# iscsiadm -m node -l all
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.使用 sanlun 確認 LUN Disks
[root@bdb7-b RPMs]# sanlun lun show
controller(7mode/E-Series)/ device host lun
vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdag host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdaf host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdae host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdad host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdac host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdab host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdz host20 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdaa host19 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdy host14 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdx host18 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdw host17 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdt host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdu host15 iSCSI 500.1g cDOT
...設定 multipath
初始化後會自動產生設定檔 multipath.conf
mpathconf --enable --with_multipathd y
建立 device alias name 與 LUN WWID 的對應
注意：在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱，而只會顯示 WWID，詳細原因請見附檔。為了爾後方便管理，必須手動建立磁碟名稱與 WWID 對應。
編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行
...
multipaths {
multipath {
wwid 3600a0980383136312d3f4e387455564b
alias mpath1
}
multipath {
wwid 3600a0980383136312d3f4e387455564c
alias mpath2
}
multipath {
wwid 3600a0980383136312d3f4e387455564d
alias mpath3
}
multipath {
wwid 3600a0980383136312d3f4e387455564e
alias mpath4
}
}wwid 可以從指令 multipath -l 取得
alias 可以設定任一名稱，LUN disks的順序可以從指令 sanlun lun show 取得
載入新設定
systemctl reload multipathd執行 multipath -l，應該要顯示 WWID 與 Alias Name
mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 12:0:0:1 sdo 8:224 active undef running
| |- 13:0:0:1 sds 65:32 active undef running
| |- 15:0:0:1 sdaa 65:160 active undef running
| `- 16:0:0:1 sdae 65:224 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
|- 10:0:0:1 sdg 8:96 active undef running
|- 11:0:0:1 sdk 8:160 active undef running
|- 14:0:0:1 sdw 65:96 active undef running
`- 9:0:0:1 sdc 8:32 active undef running開機掛載 iSCSI LUNs
注意：若沒有以下設定，將導致系統無法正常開機。
編輯 /etc/fstab , 在 iSCSI disk 每一行都加上 _netdev
#device mount point FS Options Backup fsck
/dev/myvg/mylv /mydisk xfs _netdev 0 0設定 netfs 服務自動啟動
## CentOS 7
systemctl enable remote-fs.target
## CentOS 4/5/6
chkconfig netfs on加大 MTU 設定
網路卡預設 MTU 是 1500，這不適合 iSCSI 的應用，建議加大至 9000。除了 Linux 設定以外，網路連接的 Switch 與 iSCSI Storage (iSCSI Target) 也需要一併做調整。
RedHat 7.x) 檢查目前網路設置是否可接受大 frame 的網路封包傳遞
UNSUCCESSFUL
# ping -M do -c 4 -s 8196 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 8185(8213) bytes of data.
PING 10.1.1.21(10.1.1.21) 8196(8224) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
SUCCESSFUL
# ping -M do -c 4 -s 1408 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 1408(1436) bytes of data.
1416 bytes from 10.1.1.21: icmp_seq=1 ttl=58 time=224 ms立即調整 MTU (NOTE: 非永久性設置)
ip link set dev ens2f0 mtu 9000永久設置方法，在 iSCSI 網路裝置的設定檔 ifcfg-ethXX，加上這行：
MTU="9000"重啟系統後生效。
擴充新硬碟 LUN disk
1. iSCSI Storage 新增 LUN disk 並分派給要使用的 Linux host。
2. 以下步驟都是 Linux host 要完成的。
3. 掃描(discover)剛剛新增的 LUN disk
[root@bdb7-b ~]# /usr/bin/rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: HPE Model: DVDROM DUD0N Rev: UMD1
Type: CD-ROM ANSI SCSI revision: 05
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for all SCSI target IDs, all LUNs
Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 7 0 0 0 ...
4. 使用 sanlun 指令檢查新 LUN disk
[root@bdb7-b ~]# sanlun lun show all
controller(7mode/E-Series)/ device host lun
vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaw host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdav host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdau host14 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdat host13 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdas host12 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdar host11 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaq host10 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdap host9 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdao host15 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdan host16 iSCSI 500.1g cDOT
TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdam host16 iSCSI 500.1g cDOT
...5. 使用 multipath 指令檢查新 LUN disk
[root@bdb7-b alang]# multipath -ll
mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:1 sdf 8:80 active ready running
| |- 12:0:0:1 sdt 65:48 active ready running
| |- 16:0:0:1 sdah 66:16 active ready running
| `- 15:0:0:1 sdai 66:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 11:0:0:1 sdg 8:96 active ready running
|- 10:0:0:1 sde 8:64 active ready running
|- 14:0:0:1 sdu 65:64 active ready running
`- 13:0:0:1 sdv 65:80 active ready running
mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:0 sdc 8:32 active ready running
| |- 12:0:0:0 sdr 65:16 active ready running
| |- 16:0:0:0 sdaf 65:240 active ready running
| `- 15:0:0:0 sdag 66:0 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 11:0:0:0 sdd 8:48 active ready running
|- 10:0:0:0 sdb 8:16 active ready running
|- 13:0:0:0 sdq 65:0 active ready running
`- 14:0:0:0 sds 65:32 active ready running
3600a098038313631305d4e2d6f77474a dm-26 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:5 sdap 66:144 active ready running
| |- 12:0:0:5 sdas 66:192 active ready running
| |- 15:0:0:5 sdav 66:240 active ready running
| `- 16:0:0:5 sdaw 67:0 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:5 sdaq 66:160 active ready running
|- 11:0:0:5 sdar 66:176 active ready running
|- 13:0:0:5 sdat 66:208 active ready running
`- 14:0:0:5 sdau 66:224 active ready running
...6. 設定 LUN WWID 與 mpathX 的對應
編輯 /etc/multipath.conf: 將新的 wwid 與 alias 加上。
# added by alang
multipaths {
multipath {
wwid 3600a098038313631305d4e2d6f774743
alias mpath1
}
multipath {
wwid 3600a098038313631305d4e2d6f774744
alias mpath2
}
multipath {
wwid 3600a098038313631305d4e2d6f774745
alias mpath3
}
multipath {
wwid 3600a098038313631305d4e2d6f774746
alias mpath4
}
multipath {
wwid 3600a098038313631305d4e2d6f774747
alias mpath5
}
multipath {
wwid 3600a098038313631305d4e2d6f77474a
alias mpath6
}
}7. 載入 multipath 設定
[root@bdb7-b alang]# systemctl reload multipathd
[root@bdb7-b alang]# multipath -ll
mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:1 sdt 65:48 active ready running
| |- 15:0:0:1 sdai 66:32 active ready running
| |- 16:0:0:1 sdah 66:16 active ready running
| `- 9:0:0:1 sdf 8:80 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:1 sde 8:64 active ready running
|- 11:0:0:1 sdg 8:96 active ready running
|- 13:0:0:1 sdv 65:80 active ready running
`- 14:0:0:1 sdu 65:64 active ready running
mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:0 sdr 65:16 active ready running
| |- 15:0:0:0 sdag 66:0 active ready running
| |- 16:0:0:0 sdaf 65:240 active ready running
| `- 9:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:0 sdb 8:16 active ready running
|- 11:0:0:0 sdd 8:48 active ready running
|- 13:0:0:0 sdq 65:0 active ready running
`- 14:0:0:0 sds 65:32 active ready running
mpath6 (3600a098038313631305d4e2d6f77474a) dm-26 NETAPP ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 12:0:0:5 sdas 66:192 active ready running
| |- 15:0:0:5 sdav 66:240 active ready running
| |- 16:0:0:5 sdaw 67:0 active ready running
| `- 9:0:0:5 sdap 66:144 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 10:0:0:5 sdaq 66:160 active ready running
|- 11:0:0:5 sdar 66:176 active ready running
|- 13:0:0:5 sdat 66:208 active ready running
`- 14:0:0:5 sdau 66:224 active ready running
完成。新加的 disk mpath6 可以用 LVM 進行管理。
其他指令
顯示 disk path 更多資訊
- 主機磁碟名稱與 Storage LUN 名稱對應
- LUN 編號與 Size
- Storage 的 path 狀態資訊
[root@bdb7-a ~]# sanlun lun show -p
ONTAP Path: TPEDBISCSISVM01:/vol/BDB7_A_2_vol/BDB7_A_2
LUN: 1
LUN Size: 500.1g
Product: cDOT
Host Device: mpath2(3600a0980383136312d3f4e387455564c)
Multipath Policy: service-time 0
Multipath Provider: Native
--------- ---------- ------- ------------ ----------------------------------------------
host vserver
path path /dev/ host vserver
state type node adapter LIF
--------- ---------- ------- ------------ ----------------------------------------------
up primary sdo host12 TPEDBISCSISVM01_iSCSI2
up primary sds host13 TPEDBISCSISVM01_iSCSI2
up primary sdaa host15 TPEDBISCSISVM01_iSCSI1
up primary sdae host16 TPEDBISCSISVM01_iSCSI1
up secondary sdg host10 TPEDBISCSISVM01_iSCSI4
up secondary sdk host11 TPEDBISCSISVM01_iSCSI3
up secondary sdw host14 TPEDBISCSISVM01_iSCSI4
up secondary sdc host9 TPEDBISCSISVM01_iSCSI3查詢 multipathd 的設定參數資訊
multipathd show config技巧: 連接不同廠牌的 Storage，預設狀態會套用不同的參數組設定，而這些參數的資訊可以從這指令獲得。
查詢目前 iSCSI connections
# iscsiadm -m session --op show
tcp: [1] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [2] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [3] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [4] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [5] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [6] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [7] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [8] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
#iSCSI Logout All
iscsiadm --mode node --logoutall=all
#iSCSI Login All
iscsiadm -m node -l