Linux iSCSI multipath on NetApp
Install the iSCSI and multipath rpms.
yum install iscsi-initiator-utils yum install device-mapper-multipath
A little note about initiator naming. I tried to change mine to be similar to how we set up targets on Linux and fumbled around for hours. When I finally moved the original initiatorname.iscsi file back, everything started working. Just leave it alone.
cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:e178b44934eb
This NetApp link has settings for the iSCSI setup on Debian. I used most of their settings, but once I got it working I did not make further changes.
cat /etc/iscsi/iscsid.conf | grep -v \# | sed /^$/d iscsid.startup = /etc/rc.d/init.d/iscsid force-start node.startup = automatic node.leading_login = No node.session.timeo.replacement_timeout = 15 node.conn[0].timeo.login_timeout = 15 node.conn[0].timeo.logout_timeout = 15 node.conn[0].timeo.noop_out_interval = 5 node.conn[0].timeo.noop_out_timeout = 5 node.session.err_timeo.abort_timeout = 15 node.session.err_timeo.lu_reset_timeout = 30 node.session.err_timeo.tgt_reset_timeout = 30 node.session.initial_login_retry_max = 8 node.session.cmds_max = 128 node.session.queue_depth = 32 node.session.xmit_thread_priority = -20 node.session.iscsi.InitialR2T = No node.session.iscsi.ImmediateData = Yes node.session.iscsi.FirstBurstLength = 262144 node.session.iscsi.MaxBurstLength = 16776192 node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768 node.conn[0].iscsi.HeaderDigest = None node.session.nr_sessions = 1 node.session.iscsi.FastAbort = Yes
Turn on iSCSI daemon.
service iscsid start chkconfig iscsid on
Check to make sure iSCSI is set to come on.
chkconfig iscsid --list iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
What iSCSI targets are advertised?
iscsiadm -m discovery -t st -p 172.22.251.11 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996 172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996 172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996
Here is my network setup. On 172.22.251.0/24 I have two interfaces that are not routed for the iSCSI connections.
ifconfig
eth0 Link encap:Ethernet HWaddr 00:10:18:78:66:4C
inet addr:172.22.251.10 Bcast:172.22.251.255 Mask:255.255.255.0
inet6 addr: fe80::210:18ff:fe78:664c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:56716 errors:0 dropped:0 overruns:0 frame:0
TX packets:50155 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:30259444 (28.8 MiB) TX bytes:4528192 (4.3 MiB)
eth2 Link encap:Ethernet HWaddr 00:1E:C9:EA:C0:55
inet addr:172.22.100.53 Bcast:172.22.100.255 Mask:255.255.255.0
inet6 addr: fe80::21e:c9ff:feea:c055/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1338563 errors:0 dropped:0 overruns:0 frame:0
TX packets:3185 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:199985983 (190.7 MiB) TX bytes:637730 (622.7 KiB)
Interrupt:16
eth3 Link encap:Ethernet HWaddr 00:1E:C9:EA:C0:56
inet addr:172.22.251.76 Bcast:172.22.251.255 Mask:255.255.255.0
inet6 addr: fe80::21e:c9ff:feea:c056/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:18673 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2084406 (1.9 MiB) TX bytes:532 (532.0 b)
Interrupt:17
I have three choices, and because I want to use the 172.22.251.0/24 subnet. So I log in and make it persistent.
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 -l iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 --op update -n node.startup -v automatic
See that I actually have three sessions.
iscsiadm -m session tcp: [1] 172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996 tcp: [2] 172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996 tcp: [3] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996
Now I have three sessions attached, because of the different IP addresses. I want to log out of two of the sessions.
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.200.96 -u Logging out of session [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.200.96,3260] Logout of [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.200.96,3260] successful.
And log out of the other session.
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 -u Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] successful.
Ensure that these two sessions do not come back.
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.200.96 --op update -n node.startup -v manual iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 --op update -n node.startup -v manual
Test to make sure everything is correct.
shutdown -r now
This is for debug output.
iscsiadm -m session -P[0-3]
What drives do we see now?
fdisk -l Disk /dev/sda: 79.5 GB, 79456894976 bytes 255 heads, 63 sectors/track, 9660 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000141ea Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 9661 77081600 8e Linux LVM Disk /dev/mapper/vg_sannfsr11-lv_root: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_sannfsr11-lv_swap: 4160 MB, 4160749568 bytes 255 heads, 63 sectors/track, 505 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_sannfsr11-lv_home: 21.1 GB, 21080571904 bytes 255 heads, 63 sectors/track, 2562 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdc: 104 MB, 104857600 bytes 4 heads, 50 sectors/track, 1024 cylinders Units = cylinders of 200 * 512 = 102400 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 65536 bytes Disk identifier: 0x00000000 Disk /dev/sdd: 104 MB, 104857600 bytes 4 heads, 50 sectors/track, 1024 cylinders Units = cylinders of 200 * 512 = 102400 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 65536 bytes Disk identifier: 0x00000000
Troubleshoot iSCSI
I found this site that explains the iscsiadm commands nicely. I do not want to take credit for this, I just want to make sure that is easily accessible to me.
Discover available targets from a discovery portal
iscsiadm -m discovery -t sendtargets -p ipaddress
iscsiadm -m discovery -t st -p 172.22.251.11 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996 172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996 172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996
Log into a specific target.
iscsiadm -m node -T targetname -p ipaddress -l
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 -l
Log out of a specific target.
iscsiadm -m node -T targetname -p ipaddress -u
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 -u Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] successful.
Display information about a target.
iscsiadm -m node -T targetname -p ipaddress
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 # BEGIN RECORD 6.2.0-873.2.el6 node.name = iqn.1992-08.com.netapp:sn.1575046996 node.tpgt = 2000 node.startup = automatic node.leading_login = No iface.hwaddress = <empty> ... output removed for brevity ...
Display statistics about a target.
iscsiadm -m node -s -T targetname -p ipaddress
iscsiadm -m node -s -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 Stats for session [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.251.11,3260] iSCSI SNMP: txdata_octets: 1348576 rxdata_octets: 35674144 noptx_pdus: 0 scsicmd_pdus: 15527 tmfcmd_pdus: 0 login_pdus: 0 text_pdus: 0 dataout_pdus: 0 logout_pdus: 0 snack_pdus: 0 noprx_pdus: 0 scsirsp_pdus: 15525 tmfrsp_pdus: 0 textrsp_pdus: 0 datain_pdus: 15455 logoutrsp_pdus: 0 r2t_pdus: 0 async_pdus: 0 rjt_pdus: 0 digest_err: 0 timeout_err: 0 iSCSI Extended: tx_sendpage_failures: 0 rx_discontiguous_hdr: 0 eh_abort_cnt: 0
Display list of all current sessions logged in.
iscsiadm -m session
iscsiadm -m session tcp: [1] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996
View iSCSI database regarding discovery
iscsiadm -m discovery -o show
iscsiadm -m discovery -o show 172.22.251.11:3260 via sendtargets
View iSCSI database regarding targets to log into
iscsiadm -m node -o show
iscsiadm -m node -o show # BEGIN RECORD 6.2.0-873.2.el6 node.name = iqn.1992-08.com.netapp:sn.1575046996 node.tpgt = 2000 node.startup = automatic node.leading_login = No iface.hwaddress = <empty> iface.ipaddress = <empty> .. output removed for brevity ...
View iSCSI database regarding sessions logged into
iscsiadm -m session -o show
iscsiadm -m session -o show tcp: [1] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996
Multipath Configuration
From /etc/multipath.conf
blacklist {
devnode "^hd[a-z]"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^cciss.*"
}
What are my WWIDs for each LUN? I know from above that sda is my internal drive while sdc and sdd are my iSCSI targets.
scsi_id -g -u /dev/sda 3600508e0000000006f508d7a67af8d03 scsi_id -g -u /dev/sdc 360a98000375435454b24426264574552 scsi_id -g -u /dev/sdd 360a98000375435454b24426264574554
So I want to exclude /dev/sda[*]. I did it using both the WWID and a regular expression. Please note that the getuid_callout and the prio_callout/prio are different that what came in the file from Red Hat.
Because user_friendly_names is set to yes I also added aliases in the multipaths section below.
cat /etc/multipath.conf
defaults {
max_fds 4096
user_friendly_names yes
}
#}
# All data under blacklist must be specific to your system.
blacklist {
wwid 3600508e0000000006f508d7a67af8d03
devnode "^hd[a-z]"
devnode "^sd[a]$"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^cciss.*"
}
#
devices {
device {
vendor "NETAPP"
product "LUN"
#getuid_callout "/sbin/scsi_id -g -u /block/%n"
#prio_callout "/sbin/mpath_prio_alua /dev/%n"
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
prio ontap
features "1 queue_if_no_path"
hardware_handler "0"
path_selector "round-robin 0"
path_grouping_policy multibus
failback immediate
rr_weight uniform
rr_min_io 128
path_checker directio
flush_on_last_del yes
}
}
multipaths {
multipath {
wwid 360a98000375435454b24426264574552
alias test1
}
multipath {
wwid 360a98000375435454b24426264574554
alias test2
}
}
Fire up multipath.
service multipathd restart chkconfig multipathd on chkconfig --list multipathd multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Now to check for multipathing.
multipath -ll test2 (360a98000375435454b24426264574554) dm-4 NETAPP,LUN size=100M features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=2 status=active `- 8:0:0:1 sdd 8:48 active ready running test1 (360a98000375435454b24426264574552) dm-3 NETAPP,LUN size=100M features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=2 status=active `- 8:0:0:0 sdc 8:32 active ready running
Troubleshooting Multipath
You can run multipathd from the command line to get an interactive shell. Use ? or help to get a listing of available commands.
multipathd -k multipathd> show config ...output removed for brevity... multipathd> show paths hcil dev dev_t pri dm_st chk_st dev_st next_check 8:0:0:0 sdc 8:32 2 active ready running XXXXX..... 10/20 8:0:0:1 sdd 8:48 2 active ready running XXX....... 7/20 multipathd> show status path checker states: down 1 up 2 multipathd> paths count Paths: 2 Busy: False multipathd> show maps name sysfs uuid test1 dm-3 360a98000375435454b24426264574552 test2 dm-4 360a98000375435454b2442626457455
Here is a list of very helpful article that covers similar commands, etc.
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/DM_Multipath/#mpio_configfile
http://kbase.redhat.com/faq/docs/DOC-6388
https://library.netapp.com/ecmdocs/ECMP1217221/html/GUID-195E70DE-4E57-411D-BDE2-36AE635AFDBC.html
http://linux.netapp.com/docs/debian/iscsi-multipath-configuration-guide
https://library.netapp.com/ecm/ecm_get_file/ECMP1217221
ChainRingCircus 