Linux iSCSI Multipath on NetApp

Linux iSCSI multipath on NetApp

Install the iSCSI and multipath rpms.

yum install iscsi-initiator-utils
yum install device-mapper-multipath

A little note about initiator naming. I tried to change mine to be similar to how we set up targets on Linux and fumbled around for hours. When I finally moved the original initiatorname.iscsi file back, everything started working. Just leave it alone.

cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:e178b44934eb

This NetApp link has settings for the iSCSI setup on Debian. I used most of their settings, but once I got it working I did not make further changes.

cat /etc/iscsi/iscsid.conf  | grep -v \# | sed /^$/d
iscsid.startup = /etc/rc.d/init.d/iscsid force-start
node.startup = automatic
node.leading_login = No
node.session.timeo.replacement_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.initial_login_retry_max = 8
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.xmit_thread_priority = -20
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.conn[0].iscsi.HeaderDigest = None
node.session.nr_sessions = 1
node.session.iscsi.FastAbort = Yes

Turn on iSCSI daemon.

service iscsid start
chkconfig iscsid on

Check to make sure iSCSI is set to come on.

chkconfig iscsid --list
iscsid         	0:off	1:off	2:off	3:on	4:on	5:on	6:off

What iSCSI targets are advertised?

iscsiadm -m discovery -t st -p 172.22.251.11
172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996
172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996
172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996

Here is my network setup. On 172.22.251.0/24 I have two interfaces that are not routed for the iSCSI connections.

ifconfig
eth0      Link encap:Ethernet  HWaddr 00:10:18:78:66:4C
          inet addr:172.22.251.10  Bcast:172.22.251.255  Mask:255.255.255.0
          inet6 addr: fe80::210:18ff:fe78:664c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:56716 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50155 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:30259444 (28.8 MiB)  TX bytes:4528192 (4.3 MiB)

eth2      Link encap:Ethernet  HWaddr 00:1E:C9:EA:C0:55
          inet addr:172.22.100.53  Bcast:172.22.100.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:c9ff:feea:c055/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1338563 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3185 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:199985983 (190.7 MiB)  TX bytes:637730 (622.7 KiB)
          Interrupt:16

eth3      Link encap:Ethernet  HWaddr 00:1E:C9:EA:C0:56
          inet addr:172.22.251.76  Bcast:172.22.251.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:c9ff:feea:c056/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18673 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2084406 (1.9 MiB)  TX bytes:532 (532.0 b)
          Interrupt:17

I have three choices, and because I want to use the 172.22.251.0/24 subnet. So I log in and make it persistent.

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 -l
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 --op update -n node.startup -v automatic

See that I actually have three sessions.

iscsiadm -m session
tcp: [1] 172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996
tcp: [2] 172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996
tcp: [3] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996

Now I have three sessions attached, because of the different IP addresses. I want to log out of two of the sessions.

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.200.96 -u
Logging out of session [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.200.96,3260]
Logout of [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.200.96,3260] successful.

And log out of the other session.

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 -u
Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260]
Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] successful.

Ensure that these two sessions do not come back.

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.200.96 --op update -n node.startup -v manual
iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 --op update -n node.startup -v manual

Test to make sure everything is correct.

shutdown -r now

This is for debug output.

 
iscsiadm -m session -P[0-3]

What drives do we see now?

fdisk -l

Disk /dev/sda: 79.5 GB, 79456894976 bytes
255 heads, 63 sectors/track, 9660 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000141ea

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        9661    77081600   8e  Linux LVM

Disk /dev/mapper/vg_sannfsr11-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_sannfsr11-lv_swap: 4160 MB, 4160749568 bytes
255 heads, 63 sectors/track, 505 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_sannfsr11-lv_home: 21.1 GB, 21080571904 bytes
255 heads, 63 sectors/track, 2562 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdc: 104 MB, 104857600 bytes
4 heads, 50 sectors/track, 1024 cylinders
Units = cylinders of 200 * 512 = 102400 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0x00000000


Disk /dev/sdd: 104 MB, 104857600 bytes
4 heads, 50 sectors/track, 1024 cylinders
Units = cylinders of 200 * 512 = 102400 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0x00000000

Troubleshoot iSCSI
I found this site that explains the iscsiadm commands nicely. I do not want to take credit for this, I just want to make sure that is easily accessible to me.

Discover available targets from a discovery portal
iscsiadm -m discovery -t sendtargets -p ipaddress

iscsiadm -m discovery -t st -p 172.22.251.11
172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996
172.22.200.96:3260,2001 iqn.1992-08.com.netapp:sn.1575046996
172.22.225.32:3260,2002 iqn.1992-08.com.netapp:sn.1575046996

Log into a specific target.
iscsiadm -m node -T targetname -p ipaddress -l

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11 -l

Log out of a specific target.
iscsiadm -m node -T targetname -p ipaddress -u

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.225.32 -u
Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260]
Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.225.32,3260] successful.

Display information about a target.
iscsiadm -m node -T targetname -p ipaddress

iscsiadm -m node -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11
# BEGIN RECORD 6.2.0-873.2.el6
node.name = iqn.1992-08.com.netapp:sn.1575046996
node.tpgt = 2000
node.startup = automatic
node.leading_login = No
iface.hwaddress = <empty>
... output removed for brevity ...

Display statistics about a target.
iscsiadm -m node -s -T targetname -p ipaddress

iscsiadm -m node -s -T iqn.1992-08.com.netapp:sn.1575046996 -p 172.22.251.11
Stats for session [sid: 1, target: iqn.1992-08.com.netapp:sn.1575046996, portal: 172.22.251.11,3260]
iSCSI SNMP:
	txdata_octets: 1348576
	rxdata_octets: 35674144
	noptx_pdus: 0
	scsicmd_pdus: 15527
	tmfcmd_pdus: 0
	login_pdus: 0
	text_pdus: 0
	dataout_pdus: 0
	logout_pdus: 0
	snack_pdus: 0
	noprx_pdus: 0
	scsirsp_pdus: 15525
	tmfrsp_pdus: 0
	textrsp_pdus: 0
	datain_pdus: 15455
	logoutrsp_pdus: 0
	r2t_pdus: 0
	async_pdus: 0
	rjt_pdus: 0
	digest_err: 0
	timeout_err: 0
iSCSI Extended:
	tx_sendpage_failures: 0
	rx_discontiguous_hdr: 0
	eh_abort_cnt: 0

Display list of all current sessions logged in.
iscsiadm -m session

iscsiadm -m session
tcp: [1] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996

View iSCSI database regarding discovery
iscsiadm -m discovery -o show

iscsiadm -m discovery -o show
172.22.251.11:3260 via sendtargets

View iSCSI database regarding targets to log into
iscsiadm -m node -o show

iscsiadm -m node -o show
# BEGIN RECORD 6.2.0-873.2.el6
node.name = iqn.1992-08.com.netapp:sn.1575046996
node.tpgt = 2000
node.startup = automatic
node.leading_login = No
iface.hwaddress = <empty>
iface.ipaddress = <empty>
.. output removed for brevity ...

View iSCSI database regarding sessions logged into
iscsiadm -m session -o show

iscsiadm -m session -o show
tcp: [1] 172.22.251.11:3260,2000 iqn.1992-08.com.netapp:sn.1575046996

Multipath Configuration
From /etc/multipath.conf

blacklist {
devnode "^hd[a-z]"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^cciss.*"
}

What are my WWIDs for each LUN? I know from above that sda is my internal drive while sdc and sdd are my iSCSI targets.

scsi_id -g -u /dev/sda
3600508e0000000006f508d7a67af8d03

scsi_id -g -u /dev/sdc
360a98000375435454b24426264574552

scsi_id -g -u /dev/sdd
360a98000375435454b24426264574554

So I want to exclude /dev/sda[*]. I did it using both the WWID and a regular expression. Please note that the getuid_callout and the prio_callout/prio are different that what came in the file from Red Hat.

Because user_friendly_names is set to yes I also added aliases in the multipaths section below.

cat /etc/multipath.conf
defaults {
        max_fds                 4096
        user_friendly_names     yes
}
#}
# All data under blacklist must be specific to your system.
blacklist {
	wwid 3600508e0000000006f508d7a67af8d03
	devnode "^hd[a-z]"
	devnode "^sd[a]$"
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode "^cciss.*"
}
#
devices {
        device {
                vendor                  "NETAPP"
                product                 "LUN"
                #getuid_callout          "/sbin/scsi_id -g -u /block/%n"
                #prio_callout            "/sbin/mpath_prio_alua /dev/%n"
                getuid_callout		"/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                prio			ontap
                features                "1 queue_if_no_path"
                hardware_handler        "0"
                path_selector           "round-robin 0"
                path_grouping_policy    multibus
                failback                immediate
                rr_weight               uniform
                rr_min_io               128
                path_checker            directio
		flush_on_last_del       yes
        }
}

multipaths {
	multipath {
		wwid 360a98000375435454b24426264574552
		alias test1
	}
	multipath {
		wwid 360a98000375435454b24426264574554
		alias test2
	}
}

Fire up multipath.

service multipathd restart
chkconfig multipathd on
chkconfig --list multipathd
multipathd     	0:off	1:off	2:on	3:on	4:on	5:on	6:off

Now to check for multipathing.

multipath -ll
test2 (360a98000375435454b24426264574554) dm-4 NETAPP,LUN
size=100M features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  `- 8:0:0:1 sdd 8:48 active ready  running
test1 (360a98000375435454b24426264574552) dm-3 NETAPP,LUN
size=100M features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  `- 8:0:0:0 sdc 8:32 active ready  running

Troubleshooting Multipath
You can run multipathd from the command line to get an interactive shell. Use ? or help to get a listing of available commands.

multipathd -k
multipathd> show config
...output removed for brevity...

multipathd> show paths
hcil    dev dev_t pri dm_st  chk_st dev_st  next_check
8:0:0:0 sdc 8:32  2   active ready  running XXXXX..... 10/20
8:0:0:1 sdd 8:48  2   active ready  running XXX....... 7/20

multipathd> show status
path checker states:
down                1
up                  2

multipathd> paths count
Paths: 2
Busy: False

multipathd> show maps
name  sysfs uuid
test1 dm-3  360a98000375435454b24426264574552
test2 dm-4  360a98000375435454b2442626457455

Here is a list of very helpful article that covers similar commands, etc.
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/DM_Multipath/#mpio_configfile
http://kbase.redhat.com/faq/docs/DOC-6388
https://library.netapp.com/ecmdocs/ECMP1217221/html/GUID-195E70DE-4E57-411D-BDE2-36AE635AFDBC.html
http://linux.netapp.com/docs/debian/iscsi-multipath-configuration-guide
https://library.netapp.com/ecm/ecm_get_file/ECMP1217221
http://opensource.marshall.edu/papers/rhel5-iscsi-HOWTO.pdf

This entry was posted in Linux, Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s