iSCSI and multipath on CentOS 7

系統環境

• iSCSI Storage: NetApp FAS8200
• Linux OS: CentOS 7
• Server H/W: HP DL380 Gen10
• iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2

設定網路卡

使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask

注意: 這兩張網卡不要作 Bonding。

設定 iSCSI

安裝 iscsi 套件

yum install iscsi-initiator-utils

設定 iscsid.conf

vi /etc/iscsi/iscsid.conf

調整 timeout

#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5

調整 node 自動 Login

node.startup = automatic

設定 initiator name

vi /etc/iscsi/initiatorname.iscsi

命名規則可以參考

iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
例如
InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk

initiator name 必須與 Storage 端的設定相同

不同的 Host 之間的名稱不能相同

安裝 multipath 與 netapp_linux_unified_host_utilities

yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm

如果沒有安裝 libhbaapi，執行 sanlun 會出現錯誤。

連接 iSCSI storage

新增網卡裝置

iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=new

Discovery for the storage

iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260

如果 NetApp storage 有設置多個 iSCSI IPs，只需要對其中一個 IP 作偵測，輸出結果就會顯示所有 IP。

[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16

Login nodes

iscsiadm -m node -l all

輸出結果會顯示多行如下資訊

[root@bdb7-b RPMs]# iscsiadm -m node -l all
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.

使用 sanlun 確認 LUN Disks

sanlun lun show

[root@bdb7-b RPMs]# sanlun lun show
controller(7mode/E-Series)/                                  device          host                  lun
vserver(cDOT/FlashRay)        lun-pathname                   filename        adapter    protocol   size    product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdag       host19     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdaf       host20     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdae       host17     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdad       host18     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdac       host16     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdab       host15     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdz        host20     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdaa       host19     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdy        host14     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdx        host18     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdw        host17     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdt        host16     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdu        host15     iSCSI      500.1g  cDOT
...

設定 multipath

初始化後會自動產生設定檔 multipath.conf

mpathconf --enable --with_multipathd y

建立 device alias name 與 LUN WWID 的對應

注意：在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes
但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱，而只會顯示 WWID，詳細原因請見附檔。

為了爾後方便管理，必須手動建立磁碟名稱與 WWID 對應。

編輯 /etc/multipath.conf
在檔案最下方新增以下幾行

...
multipaths {
        multipath {
                wwid    3600a0980383136312d3f4e387455564b
                alias   mpath1
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564c
                alias   mpath2
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564d
                alias   mpath3
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564e
                alias   mpath4
        }
}

wwid 可以從指令 multipath -l 取得

alias 可以設定任一名稱，LUN disks的順序可以從指令 sanlun lun show 取得

載入新設定

systemctl reload multipathd

執行 multipath -l，應該要顯示 WWID 與 Alias Name

mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP  ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| |- 12:0:0:1 sdo  8:224  active undef running
| |- 13:0:0:1 sds  65:32  active undef running
| |- 15:0:0:1 sdaa 65:160 active undef running
| `- 16:0:0:1 sdae 65:224 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
  |- 10:0:0:1 sdg  8:96   active undef running
  |- 11:0:0:1 sdk  8:160  active undef running
  |- 14:0:0:1 sdw  65:96  active undef running
  `- 9:0:0:1  sdc  8:32   active undef running

開機掛載 iSCSI LUNs

注意：若沒有以下設定，將導致系統無法正常開機。

編輯 /etc/fstab
在 iSCSI disk 每一行都加上 _netdev

#device         mount point     FS      Options Backup  fsck
/dev/myvg/mylv  /mydisk         xfs     _netdev   0      0

設定 netfs 服務自動啟動

## CentOS 7
systemctl enable remote-fs.target

## CentOS 4/5/6
chkconfig netfs on

加大 MTU 設定

網路卡預設 MTU 是 1500，這不適合 iSCSI 的應用，建議加大至 9000。除了 Linux 設定以外，網路連接的 Switch 與 iSCSI Storage (iSCSI Target) 也需要一併做調整。

RedHat 7.x)

檢查目前網路設置是否可接受大 frame 的網路封包傳遞

UNSUCCESSFUL

# ping -M do -c 4 -s  8196 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 8185(8213) bytes of data.
PING 10.1.1.21(10.1.1.21) 8196(8224) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500

SUCCESSFUL

# ping -M do -c 4 -s 1408 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 1408(1436) bytes of data.
1416 bytes from 10.1.1.21: icmp_seq=1 ttl=58 time=224 ms

立即調整 MTU (NOTE: 非永久性設置)

ip link set dev ens2f0 mtu 9000

永久設置方法，在 iSCSI 網路裝置的設定檔 ifcfg-ethXX，加上這行：

MTU="9000"

重啟系統後生效。

其他指令

顯示 disk path 更多資訊

• 主機磁碟名稱與 Storage LUN 名稱對應
• LUN 編號與 Size
• Storage 的 path 狀態資訊

[root@bdb7-a ~]# sanlun lun show -p

                    ONTAP Path: TPEDBISCSISVM01:/vol/BDB7_A_2_vol/BDB7_A_2
                           LUN: 1
                      LUN Size: 500.1g
                       Product: cDOT
                   Host Device: mpath2(3600a0980383136312d3f4e387455564c)
              Multipath Policy: service-time 0
            Multipath Provider: Native
--------- ---------- ------- ------------ ----------------------------------------------
host      vserver
path      path       /dev/   host         vserver
state     type       node    adapter      LIF
--------- ---------- ------- ------------ ----------------------------------------------
up        primary    sdo     host12       TPEDBISCSISVM01_iSCSI2
up        primary    sds     host13       TPEDBISCSISVM01_iSCSI2
up        primary    sdaa    host15       TPEDBISCSISVM01_iSCSI1
up        primary    sdae    host16       TPEDBISCSISVM01_iSCSI1
up        secondary  sdg     host10       TPEDBISCSISVM01_iSCSI4
up        secondary  sdk     host11       TPEDBISCSISVM01_iSCSI3
up        secondary  sdw     host14       TPEDBISCSISVM01_iSCSI4
up        secondary  sdc     host9        TPEDBISCSISVM01_iSCSI3

查詢 multipathd 的設定參數資訊

multipathd show config

技巧: 連接不同廠牌的 Storage，預設狀態會套用不同的參數組設定，而這些參數的資訊可以從這指令獲得。

查詢目前 iSCSI connections

# iscsiadm -m session --op show
tcp: [1] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [2] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [3] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [4] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [5] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [6] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [7] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [8] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)

#iSCSI Logout All
iscsiadm --mode node --logoutall=all

#iSCSI Login All
iscsiadm -m node -l