Skip to main content

iSCSI and multipath on CentOS 7

系統環境
  • iSCSI Storage: NetApp FAS8200
  • Linux OS: CentOS 7
  • Server H/W: HP DL380 Gen10
  • iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2
設定網路卡

使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask

注意: 這兩張網卡不要作 Bonding。

設定 iSCSI

安裝 iscsi 套件

yum install iscsi-initiator-utils

設定 iscsid.conf: /etc/iscsi/iscsid.conf

# 調整 timeout
#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5

# 調整 node 自動 Login
node.startup = automatic

設定 initiator name: /etc/iscsi/initiatorname.iscsi

命名規則可以參考

  • iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
    例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk
  • initiator name 必須與 Storage 端的設定相同
  • 不同的 Host 之間的名稱不能相同
安裝 multipath 與 netapp_linux_unified_host_utilities
yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm

如果沒有安裝 libhbaapi,執行 sanlun 會出現錯誤。

連接 iSCSI storage

新增網卡裝置

iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=new

Discovery for the storage

iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260

如果 NetApp storage 有設置多個 iSCSI IPs,只需要對其中一個 IP 作偵測,輸出結果就會顯示所有 IP。

[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
    10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16

Login nodes

iscsiadm -m node -l all

輸出結果會顯示多行如下資訊

[root@bdb7-b RPMs]# iscsiadm -m node -l all
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.

使用 sanlun 確認 LUN Disks

[root@bdb7-b RPMs]# sanlun lun show
    controller(7mode/E-Series)/                                  device          host                  lun
    vserver(cDOT/FlashRay)        lun-pathname                   filename        adapter    protocol   size    product
    ---------------------------------------------------------------------------------------------------------------
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdag       host19     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdaf       host20     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdae       host17     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdad       host18     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdac       host16     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdab       host15     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdz        host20     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdaa       host19     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdy        host14     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdx        host18     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdw        host17     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdt        host16     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdu        host15     iSCSI      500.1g  cDOT
    ...
設定 multipath

初始化後會自動產生設定檔 multipath.conf

mpathconf --enable --with_multipathd y

建立 device alias name 與 LUN WWID 的對應

注意:在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes
    但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱,而只會顯示 WWID,詳細原因請見附檔。
為了爾後方便管理,必須手動建立磁碟名稱與 WWID 對應。

編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行

...
multipaths {
        multipath {
                wwid    3600a0980383136312d3f4e387455564b
                alias   mpath1
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564c
                alias   mpath2
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564d
                alias   mpath3
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564e
                alias   mpath4
        }
}

wwid 可以從指令 multipath -l 取得

alias 可以設定任一名稱,LUN disks的順序可以從指令 sanlun lun show 取得

載入新設定

systemctl reload multipathd

執行 multipath -l,應該要顯示 WWID 與 Alias Name

    mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP  ,LUN C-Mode
    size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
    |-+- policy='service-time 0' prio=0 status=active
    | |- 12:0:0:1 sdo  8:224  active undef running
    | |- 13:0:0:1 sds  65:32  active undef running
    | |- 15:0:0:1 sdaa 65:160 active undef running
    | `- 16:0:0:1 sdae 65:224 active undef running
    `-+- policy='service-time 0' prio=0 status=enabled
      |- 10:0:0:1 sdg  8:96   active undef running
      |- 11:0:0:1 sdk  8:160  active undef running
      |- 14:0:0:1 sdw  65:96  active undef running
      `- 9:0:0:1  sdc  8:32   active undef running
開機掛載 iSCSI LUNs