Skip to main content

iSCSI and multipath on CentOS 7

系統環境
  • iSCSI Storage: NetApp FAS8200
  • Linux OS: CentOS 7
  • Server H/W: HP DL380 Gen10
  • iSCSI 用網路卡: 10G SFP 光纖網路卡 x 2
設定網路卡

使用指令 nmtui 分別設定兩張網卡的 IP 與 Mask

注意: 這兩張網卡不要作 Bonding。

設定 iSCSI

安裝 iscsi 套件

yum install iscsi-initiator-utils

設定 iscsid.conf: /etc/iscsi/iscsid.conf

# 調整 timeout
#node.session.timeo.replacement_timeout = 120
node.session.timeo.replacement_timeout = 5

# 調整 node 自動 Login
node.startup = automatic

設定 initiator name: /etc/iscsi/initiatorname.iscsi

命名規則可以參考

  • iqn.<year-to-date>.<your-domain>.<your-hostname>:redhat-disk
    例如 InitiatorName=iqn.2019-02.mycompany.bdb7-a:redhat-disk
  • initiator name 必須與 Storage 端的設定相同
  • 不同的 Host 之間的名稱不能相同
安裝 multipath 與 netapp_linux_unified_host_utilities
yum install device-mapper-multipath
yum install libhbaapi
rpm -ivh netapp_linux_unified_host_utilities-7-1.x86_64.rpm

如果沒有安裝 libhbaapi,執行 sanlun 會出現錯誤。

連接 iSCSI storage

新增網卡裝置

iscsiadm -m iface -I iscsi_ens2f0 --op=new
iscsiadm -m iface -I iscsi_ens3f0 --op=new

Discovery for the storage

iscsiadm -m discovery -t sendtargets -p <ip-to-iscsi-storage>:3260

如果 NetApp storage 有設置多個 iSCSI IPs,只需要對其中一個 IP 作偵測,輸出結果就會顯示所有 IP。

[root@bdb7-b RPMs]# iscsiadm -m discovery -t sendtargets -p 10.18.104.32:3260
    10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16
    10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16

Login nodes

iscsiadm -m node -l all

輸出結果會顯示多行如下資訊

[root@bdb7-b RPMs]# iscsiadm -m node -l all
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] (multiple)
    Logging in to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
    Logging in to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] (multiple)
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.32,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.34,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.33,3260] successful.
    Login to [iface: iscsi_ens2f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.
    Login to [iface: iscsi_ens3f0, target: iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16, portal: 10.18.104.31,3260] successful.

使用 sanlun 確認 LUN Disks

[root@bdb7-b RPMs]# sanlun lun show
    controller(7mode/E-Series)/                                  device          host                  lun
    vserver(cDOT/FlashRay)        lun-pathname                   filename        adapter    protocol   size    product
    ---------------------------------------------------------------------------------------------------------------
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdag       host19     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdaf       host20     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdae       host17     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdad       host18     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdac       host16     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdab       host15     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdz        host20     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdaa       host19     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdy        host14     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdx        host18     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdw        host17     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdt        host16     iSCSI      500.1g  cDOT
    TPEDBISCSISVM01               /vol/BDB7_B_3_vol/BDB7_B_3     /dev/sdu        host15     iSCSI      500.1g  cDOT
    ...
設定 multipath

初始化後會自動產生設定檔 multipath.conf

mpathconf --enable --with_multipathd y

建立 device alias name 與 LUN WWID 的對應

注意:在 RedHat 7.6 的 multipathd 預設雖然已設定 user_friendly_names yes
    但對於 NetApp 的 LUN disks 仍不會自動建立 mpath X 的磁碟名稱,而只會顯示 WWID,詳細原因請見附檔。
為了爾後方便管理,必須手動建立磁碟名稱與 WWID 對應。

編輯 /etc/multipath.conf , 在檔案最下方新增以下幾行

...
multipaths {
        multipath {
                wwid    3600a0980383136312d3f4e387455564b
                alias   mpath1
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564c
                alias   mpath2
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564d
                alias   mpath3
        }
        multipath {
                wwid    3600a0980383136312d3f4e387455564e
                alias   mpath4
        }
}

wwid 可以從指令 multipath -l 取得

alias 可以設定任一名稱,LUN disks的順序可以從指令 sanlun lun show 取得

載入新設定

systemctl reload multipathd

執行 multipath -l,應該要顯示 WWID 與 Alias Name

    mpath2 (3600a0980383136312d3f4e387455564c) dm-5 NETAPP  ,LUN C-Mode
    size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
    |-+- policy='service-time 0' prio=0 status=active
    | |- 12:0:0:1 sdo  8:224  active undef running
    | |- 13:0:0:1 sds  65:32  active undef running
    | |- 15:0:0:1 sdaa 65:160 active undef running
    | `- 16:0:0:1 sdae 65:224 active undef running
    `-+- policy='service-time 0' prio=0 status=enabled
      |- 10:0:0:1 sdg  8:96   active undef running
      |- 11:0:0:1 sdk  8:160  active undef running
      |- 14:0:0:1 sdw  65:96  active undef running
      `- 9:0:0:1  sdc  8:32   active undef running
開機掛載 iSCSI LUNs

注意:若沒有以下設定,將導致系統無法正常開機。

編輯 /etc/fstab , 在 iSCSI disk 每一行都加上 _netdev

#device         mount point     FS      Options Backup  fsck
/dev/myvg/mylv  /mydisk         xfs     _netdev   0      0

設定 netfs 服務自動啟動

## CentOS 7
systemctl enable remote-fs.target

## CentOS 4/5/6
chkconfig netfs on
加大 MTU 設定

網路卡預設 MTU 是 1500,這不適合 iSCSI 的應用,建議加大至 9000。除了 Linux 設定以外,網路連接的 Switch 與 iSCSI Storage (iSCSI Target) 也需要一併做調整。

RedHat 7.x) 檢查目前網路設置是否可接受大 frame 的網路封包傳遞

UNSUCCESSFUL

# ping -M do -c 4 -s  8196 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 8185(8213) bytes of data.
PING 10.1.1.21(10.1.1.21) 8196(8224) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500

SUCCESSFUL

# ping -M do -c 4 -s 1408 10.1.1.21
PING 10.1.1.21 (10.1.1.21) 1408(1436) bytes of data.
1416 bytes from 10.1.1.21: icmp_seq=1 ttl=58 time=224 ms

立即調整 MTU (NOTE: 非永久性設置)

永久設置方法,在 iSCSI 網路裝置的設定檔 ifcfg-ethXX,加上這行:

MTU="9000"

重啟系統後生效。

擴充新硬碟 LUN disk

1. iSCSI Storage 新增 LUN disk 並分派給要使用的 Linux host。

2. 以下步驟都是 Linux host 要完成的。

3. 掃描(discover)剛剛新增的 LUN disk

[root@bdb7-b ~]# /usr/bin/rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: HPE      Model: DVDROM DUD0N     Rev: UMD1
      Type:   CD-ROM                           ANSI SCSI revision: 05
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  all SCSI target IDs, all LUNs
Scanning host 7 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 7 0 0 0 ...

4. 使用 sanlun 指令檢查新 LUN disk

[root@bdb7-b ~]# sanlun lun show all
controller(7mode/E-Series)/                                  device          host                  lun
vserver(cDOT/FlashRay)        lun-pathname                   filename        adapter    protocol   size    product
---------------------------------------------------------------------------------------------------------------
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdaw       host16     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdav       host15     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdau       host14     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdat       host13     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdas       host12     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdar       host11     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdaq       host10     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_6_vol/BDB7_B_6     /dev/sdap       host9      iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_5_vol/BDB7_B_5     /dev/sdao       host15     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_5_vol/BDB7_B_5     /dev/sdan       host16     iSCSI      500.1g  cDOT
TPEDBISCSISVM01               /vol/BDB7_B_4_vol/BDB7_B_4     /dev/sdam       host16     iSCSI      500.1g  cDOT
...

5. 使用 multipath 指令檢查新 LUN disk

[root@bdb7-b alang]# multipath -ll
mpath2 (3600a098038313631305d4e2d6f774744) dm-6 NETAPP  ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:1  sdf  8:80   active ready running
| |- 12:0:0:1 sdt  65:48  active ready running
| |- 16:0:0:1 sdah 66:16  active ready running
| `- 15:0:0:1 sdai 66:32  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 11:0:0:1 sdg  8:96   active ready running
  |- 10:0:0:1 sde  8:64   active ready running
  |- 14:0:0:1 sdu  65:64  active ready running
  `- 13:0:0:1 sdv  65:80  active ready running
mpath1 (3600a098038313631305d4e2d6f774743) dm-5 NETAPP  ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:0  sdc  8:32   active ready running
| |- 12:0:0:0 sdr  65:16  active ready running
| |- 16:0:0:0 sdaf 65:240 active ready running
| `- 15:0:0:0 sdag 66:0   active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 11:0:0:0 sdd  8:48   active ready running
  |- 10:0:0:0 sdb  8:16   active ready running
  |- 13:0:0:0 sdq  65:0   active ready running
  `- 14:0:0:0 sds  65:32  active ready running
3600a098038313631305d4e2d6f77474a dm-26 NETAPP  ,LUN C-Mode
size=500G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 9:0:0:5  sdap 66:144 active ready running
| |- 12:0:0:5 sdas 66:192 active ready running
| |- 15:0:0:5 sdav 66:240 active ready running
| `- 16:0:0:5 sdaw 67:0   active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 10:0:0:5 sdaq 66:160 active ready running
  |- 11:0:0:5 sdar 66:176 active ready running
  |- 13:0:0:5 sdat 66:208 active ready running
  `- 14:0:0:5 sdau 66:224 active ready running
...

6. 設定 LUN WWID 與 mpathX 的對應

編輯 /etc/multipath.conf: 將新的 wwid 與 alias 加上。

# added by alang
multipaths {
        multipath {
                wwid    3600a098038313631305d4e2d6f774743
                alias   mpath1
        }
        multipath {
                wwid    3600a098038313631305d4e2d6f774744
                alias   mpath2
        }
        multipath {
                wwid    3600a098038313631305d4e2d6f774745
                alias   mpath3
        }
        multipath {
                wwid    3600a098038313631305d4e2d6f774746
                alias   mpath4
        }
        multipath {
                wwid    3600a098038313631305d4e2d6f774747
                alias   mpath5
        }
        multipath {
                wwid    3600a098038313631305d4e2d6f77474a
                alias   mpath6
        }
}

7. 載入 multipath 設定

systemctl reload multipathd
其他指令

顯示 disk path 更多資訊

  • 主機磁碟名稱與 Storage LUN 名稱對應
  • LUN 編號與 Size
  • Storage 的 path 狀態資訊
[root@bdb7-a ~]# sanlun lun show -p

                    ONTAP Path: TPEDBISCSISVM01:/vol/BDB7_A_2_vol/BDB7_A_2
                           LUN: 1
                      LUN Size: 500.1g
                       Product: cDOT
                   Host Device: mpath2(3600a0980383136312d3f4e387455564c)
              Multipath Policy: service-time 0
            Multipath Provider: Native
--------- ---------- ------- ------------ ----------------------------------------------
host      vserver
path      path       /dev/   host         vserver
state     type       node    adapter      LIF
--------- ---------- ------- ------------ ----------------------------------------------
up        primary    sdo     host12       TPEDBISCSISVM01_iSCSI2
up        primary    sds     host13       TPEDBISCSISVM01_iSCSI2
up        primary    sdaa    host15       TPEDBISCSISVM01_iSCSI1
up        primary    sdae    host16       TPEDBISCSISVM01_iSCSI1
up        secondary  sdg     host10       TPEDBISCSISVM01_iSCSI4
up        secondary  sdk     host11       TPEDBISCSISVM01_iSCSI3
up        secondary  sdw     host14       TPEDBISCSISVM01_iSCSI4
up        secondary  sdc     host9        TPEDBISCSISVM01_iSCSI3

查詢 multipathd 的設定參數資訊

multipathd show config

技巧: 連接不同廠牌的 Storage,預設狀態會套用不同的參數組設定,而這些參數的資訊可以從這指令獲得。

查詢目前 iSCSI connections

# iscsiadm -m session --op show
tcp: [1] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [2] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [3] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [4] 10.18.104.34:3260,1040 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [5] 10.18.104.33:3260,1039 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [6] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [7] 10.18.104.31:3260,1037 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)
tcp: [8] 10.18.104.32:3260,1038 iqn.1992-08.com.netapp:sn.58042162340f11e9892700a098e3ab45:vs.16 (non-flash)

#iSCSI Logout All
iscsiadm --mode node --logoutall=all

#iSCSI Login All
iscsiadm -m node -l