# SAN & iSCSI SAN - Storage Area Network, iSCSI - Internet Small Computer System Interface # iSCSI and multipath on CentOS 7 ##### 系統環境 - iSCSI Storage: NetApp FAS8200 - Linux OS: CentOS 7 - Server H/W: HP DL380 Gen10 - iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2 ##### 設定網路卡 使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask > 注意: 這兩張網卡不要作 Bonding。 ##### 設定 iSCSI 安裝 iscsi 套件 ```shell yum install iscsi-initiator-utils ``` 設定 iscsid.conf: `/etc/iscsi/iscsid.conf` ``` # 調整 timeout #node.session.timeo.replacement_timeout = 120 node.session.timeo.replacement_timeout = 5 # 調整 node 自動 Login node.startup = automatic ``` 設定 initiator name: `/etc/iscsi/initiatorname.iscsi` 命名規則可以參考 - iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk 例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk - initiator name 必須與 Storage 端的設定相同 - 不同的 Host 之間的名稱不能相同 ##### 安裝 multipath 與 netapp\_linux\_unified\_host\_utilities ``` yum install device-mapper-multipath yum install libhbaapi rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm ``` > 如果沒有安裝 libhbaapi，執行 sanlun 會出現錯誤。 ##### 連接 iSCSI storage 新增網卡裝置 ``` iscsiadm -m iface -I iscsi_ens2f0 --op=new iscsiadm -m iface -I iscsi_ens3f0 --op=new ``` Discovery for the storage ``` iscsiadm -m discovery -t sendtargets -p :3260 ``` > 如果 NetApp storage 有設置多個 iSCSI IPs，只需要對其中一個 IP 作偵測，輸出結果就會顯示所有 IP。 ``` [root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 ``` Login nodes ``` iscsiadm -m node -l all ``` > 輸出結果會顯示多行如下資訊 ``` [root@bdb7-b RPMs]# iscsiadm -m node -l all Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple) Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple) Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple) Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple) Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple) Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple) Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple) Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple) Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful. Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful. Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful. Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful. Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful. Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful. Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful. Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful. ``` 使用 sanlun 確認 LUN Disks ``` [root@bdb7-b RPMs]# sanlun lun show controller(7mode/E-Series)/ device host lun vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product --------------------------------------------------------------------------------------------------------------- TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdag host19 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdaf host20 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdae host17 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdad host18 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdac host16 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdab host15 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdz host20 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdaa host19 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdy host14 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdx host18 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdw host17 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdt host16 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_3_vol/BDB7_B_3 /dev/sdu host15 iSCSI 500.1g cDOT ... ``` ##### 設定 multipath 初始化後會自動產生設定檔 multipath.conf ``` mpathconf --enable --with_multipathd y ``` 建立 device alias name 與 LUN WWID 的對應 >
注意：在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes
但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱，而只會顯示 WWID，詳細原因請見附檔。
為了爾後方便管理，必須手動建立磁碟名稱與 WWID 對應。
編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行 ``` ... multipaths { multipath { wwid 3600a0980383136312d3f4e387455564b alias mpath1 } multipath { wwid 3600a0980383136312d3f4e387455564c alias mpath2 } multipath { wwid 3600a0980383136312d3f4e387455564d alias mpath3 } multipath { wwid 3600a0980383136312d3f4e387455564e alias mpath4 } } ``` > wwid 可以從指令 multipath -l 取得 > > alias 可以設定任一名稱，LUN disks的順序可以從指令 sanlun lun show 取得 載入新設定 ``` systemctl reload multipathd ``` 執行 multipath -l，應該要顯示 WWID 與 Alias Name ``` mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=0 status=active | |- 12:0:0:1 sdo 8:224 active undef running | |- 13:0:0:1 sds 65:32 active undef running | |- 15:0:0:1 sdaa 65:160 active undef running | `- 16:0:0:1 sdae 65:224 active undef running `-+- policy='service-time 0' prio=0 status=enabled |- 10:0:0:1 sdg 8:96 active undef running |- 11:0:0:1 sdk 8:160 active undef running |- 14:0:0:1 sdw 65:96 active undef running `- 9:0:0:1 sdc 8:32 active undef running ``` ##### 開機掛載 iSCSI LUNs > 注意：若沒有以下設定，將導致系統無法正常開機。 編輯 /etc/fstab , 在 iSCSI disk 每一行都加上 \_netdev ``` #device mount point FS Options Backup fsck /dev/myvg/mylv /mydisk xfs _netdev 0 0 ``` 設定 netfs 服務自動啟動 ``` ## CentOS 7 systemctl enable remote-fs.target ## CentOS 4/5/6 chkconfig netfs on ``` ##### 加大 MTU 設定 網路卡預設 MTU 是 1500，這不適合 iSCSI 的應用，建議加大至 9000。除了 Linux 設定以外，網路連接的 Switch 與 iSCSI Storage (iSCSI Target) 也需要一併做調整。 RedHat 7.x) 檢查目前網路設置是否可接受大 frame 的網路封包傳遞 ``` UNSUCCESSFUL # ping -M do -c 4 -s 8196 10.1.1.21 PING 10.1.1.21 (10.1.1.21) 8185(8213) bytes of data. PING 10.1.1.21(10.1.1.21) 8196(8224) bytes of data. ping: local error: Message too long, mtu=1500 ping: local error: Message too long, mtu=1500 SUCCESSFUL # ping -M do -c 4 -s 1408 10.1.1.21 PING 10.1.1.21 (10.1.1.21) 1408(1436) bytes of data. 1416 bytes from 10.1.1.21: icmp_seq=1 ttl=58 time=224 ms ``` 立即調整 MTU (NOTE: 非永久性設置) ``` ip link set dev ens2f0 mtu 9000 ``` 永久設置方法，在 iSCSI 網路裝置的設定檔 ifcfg-ethXX，加上這行： ``` MTU="9000" ``` 重啟系統後生效。 ##### 擴充新硬碟 LUN disk 1\. iSCSI Storage 新增 LUN disk 並分派給要使用的 Linux host。 2\. 以下步驟都是 Linux host 要完成的。 3\. 掃描(discover)剛剛新增的 LUN disk ``` [root@bdb7-b ~]# /usr/bin/rescan-scsi-bus.sh Scanning SCSI subsystem for new devices Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning for device 3 0 0 0 ... OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00 Vendor: HPE Model: DVDROM DUD0N Rev: UMD1 Type: CD-ROM ANSI SCSI revision: 05 Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning host 6 for all SCSI target IDs, all LUNs Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning for device 7 0 0 0 ... ``` 4\. 使用 sanlun 指令檢查新 LUN disk ``` [root@bdb7-b ~]# sanlun lun show all controller(7mode/E-Series)/ device host lun vserver(cDOT/FlashRay) lun-pathname filename adapter protocol size product --------------------------------------------------------------------------------------------------------------- TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaw host16 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdav host15 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdau host14 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdat host13 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdas host12 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdar host11 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdaq host10 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_6_vol/BDB7_B_6 /dev/sdap host9 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdao host15 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_5_vol/BDB7_B_5 /dev/sdan host16 iSCSI 500.1g cDOT TPEDBISCSISVM01 /vol/BDB7_B_4_vol/BDB7_B_4 /dev/sdam host16 iSCSI 500.1g cDOT ... ``` 5\. 使用 multipath 指令檢查新 LUN disk ``` [root@bdb7-b alang]# multipath -ll mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 9:0:0:1 sdf 8:80 active ready running | |- 12:0:0:1 sdt 65:48 active ready running | |- 16:0:0:1 sdah 66:16 active ready running | `- 15:0:0:1 sdai 66:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 11:0:0:1 sdg 8:96 active ready running |- 10:0:0:1 sde 8:64 active ready running |- 14:0:0:1 sdu 65:64 active ready running `- 13:0:0:1 sdv 65:80 active ready running mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 9:0:0:0 sdc 8:32 active ready running | |- 12:0:0:0 sdr 65:16 active ready running | |- 16:0:0:0 sdaf 65:240 active ready running | `- 15:0:0:0 sdag 66:0 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 11:0:0:0 sdd 8:48 active ready running |- 10:0:0:0 sdb 8:16 active ready running |- 13:0:0:0 sdq 65:0 active ready running `- 14:0:0:0 sds 65:32 active ready running 3600a098038313631305d4e2d6f77474a dm-26 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 9:0:0:5 sdap 66:144 active ready running | |- 12:0:0:5 sdas 66:192 active ready running | |- 15:0:0:5 sdav 66:240 active ready running | `- 16:0:0:5 sdaw 67:0 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 10:0:0:5 sdaq 66:160 active ready running |- 11:0:0:5 sdar 66:176 active ready running |- 13:0:0:5 sdat 66:208 active ready running `- 14:0:0:5 sdau 66:224 active ready running ... ``` 6\. 設定 LUN WWID 與 mpathX 的對應 編輯 `/etc/multipath.conf`: 將新的 wwid 與 alias 加上。 ``` # added by alang multipaths { multipath { wwid 3600a098038313631305d4e2d6f774743 alias mpath1 } multipath { wwid 3600a098038313631305d4e2d6f774744 alias mpath2 } multipath { wwid 3600a098038313631305d4e2d6f774745 alias mpath3 } multipath { wwid 3600a098038313631305d4e2d6f774746 alias mpath4 } multipath { wwid 3600a098038313631305d4e2d6f774747 alias mpath5 } multipath { wwid 3600a098038313631305d4e2d6f77474a alias mpath6 } } ``` 7\. 載入 multipath 設定 ```shell [root@bdb7-b alang]# systemctl reload multipathd [root@bdb7-b alang]# multipath -ll mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 12:0:0:1 sdt 65:48 active ready running | |- 15:0:0:1 sdai 66:32 active ready running | |- 16:0:0:1 sdah 66:16 active ready running | `- 9:0:0:1 sdf 8:80 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 10:0:0:1 sde 8:64 active ready running |- 11:0:0:1 sdg 8:96 active ready running |- 13:0:0:1 sdv 65:80 active ready running `- 14:0:0:1 sdu 65:64 active ready running mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 12:0:0:0 sdr 65:16 active ready running | |- 15:0:0:0 sdag 66:0 active ready running | |- 16:0:0:0 sdaf 65:240 active ready running | `- 9:0:0:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 10:0:0:0 sdb 8:16 active ready running |- 11:0:0:0 sdd 8:48 active ready running |- 13:0:0:0 sdq 65:0 active ready running `- 14:0:0:0 sds 65:32 active ready running mpath6 (3600a098038313631305d4e2d6f77474a) dm-26 NETAPP ,LUN C-Mode size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | |- 12:0:0:5 sdas 66:192 active ready running | |- 15:0:0:5 sdav 66:240 active ready running | |- 16:0:0:5 sdaw 67:0 active ready running | `- 9:0:0:5 sdap 66:144 active ready running `-+- policy='service-time 0' prio=10 status=enabled |- 10:0:0:5 sdaq 66:160 active ready running |- 11:0:0:5 sdar 66:176 active ready running |- 13:0:0:5 sdat 66:208 active ready running `- 14:0:0:5 sdau 66:224 active ready running ``` 完成。新加的 disk mpath6 可以用 LVM 進行管理。 ##### 其他指令 顯示 disk path 更多資訊 - 主機磁碟名稱與 Storage LUN 名稱對應 - LUN 編號與 Size - Storage 的 path 狀態資訊 ``` [root@bdb7-a ~]# sanlun lun show -p ONTAP Path: TPEDBISCSISVM01:/vol/BDB7_A_2_vol/BDB7_A_2 LUN: 1 LUN Size: 500.1g Product: cDOT Host Device: mpath2(3600a0980383136312d3f4e387455564c) Multipath Policy: service-time 0 Multipath Provider: Native --------- ---------- ------- ------------ ---------------------------------------------- host vserver path path /dev/ host vserver state type node adapter LIF --------- ---------- ------- ------------ ---------------------------------------------- up primary sdo host12 TPEDBISCSISVM01_iSCSI2 up primary sds host13 TPEDBISCSISVM01_iSCSI2 up primary sdaa host15 TPEDBISCSISVM01_iSCSI1 up primary sdae host16 TPEDBISCSISVM01_iSCSI1 up secondary sdg host10 TPEDBISCSISVM01_iSCSI4 up secondary sdk host11 TPEDBISCSISVM01_iSCSI3 up secondary sdw host14 TPEDBISCSISVM01_iSCSI4 up secondary sdc host9 TPEDBISCSISVM01_iSCSI3 ``` 查詢 multipathd 的設定參數資訊 ``` multipathd show config ``` > 技巧: 連接不同廠牌的 Storage，預設狀態會套用不同的參數組設定，而這些參數的資訊可以從這指令獲得。 查詢目前 iSCSI connections ``` # iscsiadm -m session --op show tcp: [1] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [2] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [3] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [4] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [5] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [6] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [7] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) tcp: [8] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash) #iSCSI Logout All iscsiadm --mode node --logoutall=all #iSCSI Login All iscsiadm -m node -l ``` # Learning ##### iSCSI - [How to configure iSCSI target & initiator on RHEL/CentOS 7.6](https://www.linuxteck.com/how-to-configure-iscsi-target-initiator-on-rhel-centos-7-6/) # Multipath on RedHat (NetApp FAS2240) Multipath 是 CentOS/RedHat 內建套件，可用作兩 port 的 HBA 卡連接 Storage時，有雙路由的備援架構。 系統環境 - RedHat 4.7 - NetApp FAS2240 - SAN Switch x 2 - HBA Card: QLogic, port x 2 ``` #>lspci Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02) ``` #### RedHat 的 HBA Card 驅動 建議安裝方法為，在安裝 RedHat 前先安裝 HBA 卡，然後安裝 RedHat 時會自動驅動 HBA 卡。 如果無法重新安裝 RedHat 試試修改 /etc/modprobe.conf，加上這一行 ``` alias scsi_hostadapter2 qla2500 ``` > scsi\_hostadapterX 編號視實際狀況修改 查出 HBA 卡的 WWN ``` #> cat /proc/scsi/qla2xxx/2 SCSI Device Information: scsi-qla0-adapter-node=5001438004253c81; scsi-qla0-adapter-port=5001438004253c80; <---- WWN scsi-qla0-target-0=50001fe1500e28fd; scsi-qla0-target-1=50001fe1500e28f9; scsi-qla0-target-2=500a09829140a66c; ``` > 另一個 HBA port 裝置檔為 /proc/scsi/qla2xxx/3，以此類推 > > 每個 port 有獨立的 WWN > > NetApp storage 分配 LUN 以此 WWN 作為 Host Mapping 的主要參數 驅動後的模組列表 ``` # lsmod | grep qla qla2500 209985 0 qla2xxx 203681 13 qla2500 scsi_transport_fc 19521 1 qla2xxx scsi_mod 153489 7 ib_srp,qla2xxx,scsi_transport_fc,sr_mod,libata,cciss,sd_mo d ``` #### SAN Storge 設定與接線 1. 將 NetApp storage、SAN switch、RedHat 接上各自的光纖線 2. 找出 RedHat 的 HBA 卡 兩 port 各自的 WWN 3. 設定 NetApp，將 LUNs 分配至兩個 WWN Hosts #### 設定 Multipath > 更新：RedHat 客戶中心有提供設定 multipath.conf 線上協助導引，網頁必須先登入才能瀏覽 > https://access.redhat.com/labs/multipathhelper/ 必要的套件安裝 ``` rpm -q device-mapper rpm -q device-mapper-multipath yum install device-mapper yum install device-mapper-multipath ``` ##### CentOS 5.x) 編輯 /etc/multipath.conf ``` # Blacklist all devices by default. Remove this to enable multipathing # on the default devices. devnode_blacklist { # devnode "*" << 註解這一行 } ## Use user friendly names, instead of using WWIDs as names. defaults { user_friendly_names yes } ``` > 注意：即使啟動了 multipathd 服務，預設是抓不到任何 LUNs 的，必須註解上述的那一行 啟動 multipathd 服務 ``` #> service multipathd start #> chkconfig multipathd on ``` ##### CentOS 6/7) 預設沒有設定檔 /etc/multipath.conf，執行這個指令會自動產生，並且將服務 multipathd 設定成自動執行。 ``` #> mpathconf --enable --with_multipathd y ``` > 不用修改 multipath.conf #### 檢視 multupah 的裝置 CentOS 5.x ``` [root@cdb3-b ~]# multipath -ll mpath1 (360a98000383035537824474d3936556a) dm-4 NETAPP,LUN [size=700G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=0][rw] \_ round-robin 0 [prio=4][active] \_ 1:0:1:0 sdb 8:16 [active][ready] \_ 2:0:1:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sda 8:0 [active][ready] \_ 2:0:0:0 sdc 8:32 [active][ready] ``` CentOS 6.x ``` [root@cdb3 ~]# multipath -ll mpathb (360a98000383035537824474d39365568) dm-2 NETAPP,LUN size=700G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=4 status=active | |- 1:0:1:0 sdc 8:32 active ready running | `- 2:0:1:0 sde 8:64 active ready running `-+- policy='round-robin 0' prio=1 status=enabled |- 1:0:0:0 sdb 8:16 active ready running `- 2:0:0:0 sdd 8:48 active ready running ``` #### 安裝 NetApp 管理工具 ＊建議＊安裝 LUN 管理工具 - NetApp Linux Host Utilities ``` yum install libnl rpm -ivh QConvergeConsoleCLI-1.0.01-32.x86_64.rpm rpm -ivh netapp_linux_host_utilities-6-2.x86_64.rpm ``` > \- 如果沒有安裝這套件，將無法使用 sanlun 指令 > > \- 適用 CentOS 5.x/6.x #### 新增 LUN 需系統重新開機 一旦重新開機後，所有的 LUN disks (在 dual path SAN 架構下會有多個 LUN disks)應該都會被偵測到，並且會有一個 mpathXX 的裝置名稱。 ``` #> multipath -l mpath1 (360a98000424732615124444150534a35) [size=100 GB][features="1 queue_if_no_path"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:2:0 sde 8:64 [active] \_ 3:0:2:0 sdf 8:80 [active] mpath0 (3600508b400068e5c00012000001a0000) [size=100 GB][features="1 queue_if_no_path"][hwhandler="0"] \_ round-robin 0 [active] \_ 2:0:0:1 sda 8:0 [active] \_ 3:0:0:1 sdc 8:32 [active] \_ round-robin 0 [enabled] \_ 2:0:1:1 sdb 8:16 [active] \_ 3:0:1:1 sdd 8:48 [active] ``` > mpathX 這路徑可作為 LVM 的 PV 或直接格式化為 filesystem，完整路徑為 /dev/mapper/mpathX > > 2:0:0:1 這表示為 host : channel : ID : LUN > > Host 2 與 Host 3 表示 HBA 卡有兩個 port 或者單 port 的卡兩張 #### 新增/移除 LUN 系統不需重開機 參考教學： - [https://library.netapp.com/ecmdocs/ECMP1654943/html/GUID-F88A86D2-F3A9-4440-8C32-539BFEA6BED1.html](https://library.netapp.com/ecmdocs/ECMP1654943/html/GUID-F88A86D2-F3A9-4440-8C32-539BFEA6BED1.html) - [https://access.redhat.com/solutions/3941](https://access.redhat.com/solutions/3941) ##### CentOS/RedHat 5.x 以後 從 CentOS/RedHat 5 以後或者 SLES 10 以後，可以安裝以下套件： ``` yum install sg3_utils ``` 不停機下掃描新的 LUN Disk ``` 檢視目前現有的 LUN Disks #> sanlun lun show all 掃描所有 LUN #> scsi-rescan NOTE: 如果偵測不到新 LUN，試試加上 -a #> scsi-rescan -a 再次檢視目前的 LUN Disks #> sanlun lun show all ``` 移除 LUN 的程序 要移除現有的 LUN disk 必須依照以下步驟完成： 1. 執行 `umount` 卸載 2. 執行 `lvremove` 移除 LV 3. 需要時，執行 `vgremove` 移除 VG 4. 執行 `pvremove` 移除 PV 5. 執行 `multipath -f mpathXX` 移除 multipathXX NOTE: 如果沒有執行這步驟，一旦移除 LUN 後系統可能會 Hang。 6. 從 Storage 管理介面移除這個 LUN disk NOTE: 當 LUN 被移除後，執行 `sanlun lun show all`，那些移除的 LUN 會顯示 unknown。 7. 執行 `scsi-rescan -r` 移除所有未使用的 LUN disk NOTE: 執行後輸出訊息，應該要顯示類似 4 device(s) removed. \[5:0:1:0\] \[5:0:3:0\] \[6:0:1:0\] \[6:0:3:0\] 重新執行 `sanlun lun show all`，那些 unknown 就會消失。 ##### CentOS/RedHat 4.x： - [Vmware Linux Guest Add a New Hard Disk Without Rebooting Guest](https://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html) 使用附檔區的兩個 script 1. remove-LUNs.sh 2. rescan-LUNs.sh 新增 LUN disk 步驟 1. 在 NetApp storage 上完成 LUN Mapping 的設定 2. 在 RedHat 上執行 `rescan-LUNs.sh` 3. 在 RedHat 上執行 `multipath -l` 移除現有 LUN disk 1. 在 RedHat 上卸載不用的 mpathX 2. 在 RedHat 上執行 `sanlun lun show all` 找出 LUN 的名稱與 NetApp 上路徑 註：要執行 sanlun 指令必須先安裝 NetApp 的套件 netapp\_linux\_host\_utilities-6-2.x86\_64.rpm 3. 在 NetApp 上移除所對應的 LUN 4. 在 RedHat 上執行 `multipath -l` 5. 找出要移除 LUN disks 的 host:channel:id:LUN 6. 執行 `remove-LUNs.sh ` #### 現有 LUN disk 擴充空間 1\. 確認 LUN ID 與 sd\[X\] 的對應 ```shell smartctl -a /dev/sd[X] ``` 2\. 掃描 LUN disk，確認最新容量 ```shell echo 1 > /sys/block/sd[X]/device/rescan



lsblk | grep sd[X] ``` 3\. Resize PV ```shell pvresize /dev/sd[X] ``` 4\. 完成。接著就按 SOP 進行 LV 與 Filesystem 的擴充。 #### LVM 管理 新增 LUN disk 成一個 PV ``` #> pvcreate /dev/mapper/mpath6 ``` > PV 新增後，執行 pvdisplay 時的裝置名稱會是 dm-X。 新增一個 VG (需指定 PV) ``` #> vgcreate dbVG /dev/mapper/mpath6 ``` #### 指令更多用法 multipath
/etc/init.d/multipathd startManually starts the multipathing service on the Linux host.
multipath -llDisplays the multipathing configuration.
multipath -v2Creates the initial multipathing configuration.
multipath -f *device\_map*Removes an existing LUN from the multipathing configuration.
multipath -ll -v2 -dDisplays the multipathing configuration in verbose mode.
multipath -t 或 multipathd show config顯示目前的設定參數
Accesing LUNs on host
sanlun fcp show adapter -vDisplays HBA information.
sanlun fcp show adapterDisplay WWPN of HBA cards
sanlun lun show allLists available LUNs as seen by the sanlun script.
sanlun lun show -pProvides multipath information for both DM-Multipath and Veritas Storage Foundation TM dynamic multipathing solutions.
ls -lh /dev/mapper/Lists DM-Multipath devices generated for LUNs.
e2label *device label*Creates a persistent label for a file system. device is the partition; label is a unique string up to 16 characters. Not needed with DM-Multipath.
#### Q & A ##### Q: RHEL 7 沒有使用 alias path > 使用 NetApp Storage，執行 multipath -ll > > 360a98000443437576f2444726e556d76 dm-2 NETAPP ,LUN > size=800G features='4 queue\_if\_no\_path pg\_init\_retries 50 retain\_attached\_hw\_handle' hwhandler='1 alua' wp=rw > |-+- policy='service-time 0' prio=50 status=active > | `- 6:0:0:0 sdb 8:16 active ready running > `-+- policy='service-time 0' prio=10 status=enabled > `- 6:0:3:0 sdc 8:32 active ready running > > 不會顯示 alias name: mpathXX > > 參閱官方手冊：https://access.redhat.com/solutions/2061463 檢視 multipath 可支援的裝置清單 ``` multipath -t ``` > 沒有看到 NetApp 的裝置。 解決方法：編輯 multipath.conf，加入 NetApp 的參數設定 > NOTE: 僅適合 NetApp Storage。 ``` defaults {

user_friendly_names yes

find_multipaths yes

}



devices {

device {

vendor "NETAPP"

product "LUN.*"

path_grouping_policy "group_by_prio"

path_checker "tur"

features "3 queue_if_no_path pg_init_retries 50"

hardware_handler "0"

prio "ontap"

failback immediate

rr_weight "uniform"

rr_min_io 128

flush_on_last_del "yes"

dev_loss_tmo "infinity"

user_friendly_names yes

retain_attached_hw_handler yes

detect_prio yes

} ``` 載入新設定 ``` #> systemctl reload multipathd



#> multipath -ll

mpatha (360a98000443437576f2444726e556d76) dm-2 NETAPP ,LUN

size=800G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active

| |- 4:0:0:0 sdb 8:16 active ready running

| `- 6:0:0:0 sdd 8:48 active ready running

`-+- policy='service-time 0' prio=10 status=enabled

|- 4:0:3:0 sdc 8:32 active ready running

`- 6:0:3:0 sde 8:64 active ready running ``` ##### Q: 安裝 netapp\_linux\_host\_utilities 時，出現錯誤訊息 > Warning: libnl.so library not found, some sanlun commands may not work. Refer Li nux Host Utilities Installation and Set up Guide for more details > Warning: libHBAAPI.so library not found, some sanlun commands may not work. Refe r Linux Host Utilities Installation and Setup Guide for more details Solution: 先安裝 libnl ``` yum install libnl ``` 再安裝以下套件任一個 - [http://driverdownloads.qlogic.com/QLogicDriverDownloads\_UI/ResourceByOS.aspx?productid=1043&oemid=372&oemcatid=111969](http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/ResourceByOS.aspx?productid=1043&oemid=372&oemcatid=111969) - [http://driverdownloads.qlogic.com/QLogicDriverDownloads\_UI/SearchByOs.aspx?ProductCategory=39&OsCategory=2&Os=65&OsCategoryName=Linux&ProductCategoryName=Fibre+Channel+HBAs&OSName=Linux+Red+Hat+%2864-bit%29](http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByOs.aspx?ProductCategory=39&OsCategory=2&Os=65&OsCategoryName=Linux&ProductCategoryName=Fibre+Channel+HBAs&OSName=Linux+Red+Hat+%2864-bit%29) - QConvergeConsole CLI (x64) - SANsurfer CLI (x86/x64) ##### Q: 執行 pvscan、vgscan 或開機時出現以下訊息 > Found duplicate PV XXXXXXXXXXXXXXXX: using /dev/sdb not /dev/sda Solution: 這通常會發生在系統有作 multipath 與 LVM 設定後，原因是 CentOS 4/5 預設 LVM 的設定會掃描所有的磁碟裝置，然而，當系統設定有 multipath 時，LVM 應該管理 mpathX 或 dm-X 的裝置，而不再是 sdX 裝置。 要使 LVM 不再對 sdX 裝置作管理，可以修改 /etc/lvm/lvm.conf，此外，如果要使每次系統開機時，LVM 不要對 sdX 裝置作掃描，還必須要重建 initrd 開機映像檔，並且修改 grub.conf 相關的設定。 操作步驟如下: 1\. 編輯 /etc/lvm/lvm.conf ``` ...

filter = [ "r|/dev/sd.*|", "a|/.*|" ]

... ``` > r|/dev/sd.\*| 這設定會排除所有 sda, sdb, ...等所有磁碟裝置掃描，如果系統有使用其中某個裝置，必須自行修改這規則。 存檔離開 執行 pvscan 或 vgscan，如果不再出現 Warning，繼續步驟 2。 2\. 重建 initrd 開機映像檔 ``` #> cd /boot

#> mkinitrd -f -v /boot/initrd-$(uname -r).multipath.img $(uname -r) ``` 編輯 /boot/grub/grub.conf 複製 title 段落的所有行，並依照實際狀況修改 ``` title CentOS (2.6.18-398.el5-Multipath) <== 加上 Multipath

root (hd0,0)

kernel /vmlinuz-2.6.18-398..........

initrd /initrd-2.6.18-398.el5.multipath.img <== 重建後的initrd 開機映像檔名 ``` 存檔離開 3\. 重新開機後，出現開機選單後，選擇第二個開機項目。 如果開機訊息不再出現警告文字，而且系統一切正常，就可以再修改 grub.conf 的 default=1。 # FAQ ##### 如何確認 /dev/sda 是 Local Disk 還是 SAN DIsk Solution: ``` #> lsscsi [2:0:0:0] cd/dvd hp DVD-ROM DTA0N WBD0 /dev/sr0 state=running queue_depth=1 scsi_level=6 type=5 device_blocked=0 timeout=0 [3:0:0:0] tape IBM ULT3580-HH5 D2AD /dev/st0 state=running queue_depth=32 scsi_level=7 type=1 device_blocked=0 timeout=900 [3:0:1:0] tape IBM ULT3580-HH5 D2AD /dev/st1 state=running queue_depth=32 scsi_level=7 type=1 device_blocked=0 timeout=900 [3:0:1:1] mediumx IBM 3573-TL C.20 - state=running queue_depth=32 scsi_level=6 type=8 device_blocked=0 timeout=0 [3:0:2:0] disk IBM 1726-4xx FAStT 0617 /dev/sda <====== state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60 [3:0:2:31] disk IBM Universal Xport 0617 - state=running queue_depth=32 scsi_level=6 type=0 device_blocked=0 timeout=60 ``` > IBM FAStT 是 IBM Storage 產品識別 > > 如果沒有 lsscsi 指令，執行 yum install lsscsi ##### 如何找出 HBA Card 的 port WWN 方法一: 用 systool ```bash # Install yum install sysfsutils # systool -c fc_host -v systool -c fc_transport -v ``` 方法二：`/sys/class/fc_host/{host#}/port_name` ``` #> cat /sys/class/fc_host/host5/port_name 0x2100001b32837bcd ``` 方法三：`/proc/scsi/{adapter-type}/{host#}` ``` #> cat /proc/scsi/qla2xxx/2 SCSI Device Information: scsi-qla0-adapter-node=5001438004253c81; scsi-qla0-adapter-port=5001438004253c80; <---- WWN scsi-qla0-target-0=50001fe1500e28fd; scsi-qla0-target-1=50001fe1500e28f9; scsi-qla0-target-2=500a09829140a66c; ``` ##### mpathX v.s. dm-X 如果有啟用 multipath，所有 LUN 的裝置路徑為 /dev/mapper/mpathX，不過當使用 pvcreate 將 LUN 新增為 PV 後，執行 pvdisplay -C ，此時的裝置名稱會改為 /dev/dm-X，那爾後作系統維護時，要如何查詢 mpathX 對應哪個 dm-X? 方法一：`dmsetup` ``` [root@bdb3 ~]# dmsetup ls mpath1 (253, 4) rootVG-varLV (253, 1) rootVG-swapLV (253, 3) rootVG-rootLV (253, 0) rootVG-worktmpLV (253, 2) ``` > mpath1 (253,4)，4 就是指裝置 /dev/dm-4 方法二：`multipath` ``` [root@cdb3-b ~]# multipath -ll mpath1 (360a98000383035537824474d3936556a) dm-4 NETAPP,LUN <====== 這裡 [size=700G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=0][rw] \_ round-robin 0 [prio=4][active] \_ 1:0:1:0 sdb 8:16 [active][ready] \_ 2:0:1:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sda 8:0 [active][ready] \_ 2:0:0:0 sdc 8:32 [active][ready] ``` ##### 新 partition 無法使用 對 mpathb 新建 partition 後，無法立即被使用，必須重開機後才能生效 > WARNING: Re-reading the partition table failed with error 22: Invalid argument. > The kernel still uses the old table. The new table will be used at > the next reboot or after you run partprobe(8) or kpartx(8) 不重開機要能立即使用，可以再切完 partition 後，執行 ```bash partprobe /dev/mapper/mpathb ```