iSCSI targets/extents 1:1 recommendation

[Daniele B.]

Hi everyone.
I'm trying to figure out a sane configuration for a FreeNAS box, acting as iSCSI server for VMs running on multiple hosts.

I read very carefully the FreeNAS documentation, many posts on this forum and other places over the net.

One thing I cannot understand is this recommendation on the FreeNAS docs:

Chapter 10.5.7, Sharing > Target/Extents

"It is recommended to always associate extents to targets in a 1:1 manner, even though the GUI will allow multiple extents to be associated with the same target."​

The documentation doesn't provide a clear explanation why targets should be associated to extents 1:1.

For me it would make sense to aggregate several extents on a few iSCSI targets, because it is simpler to see new LUNs when I add them as extents on FreeNAS, without having to rediscover iSCSI targets from every host running VMs (Linux kvm based hosts, not ESXi).

I found evidence of many virtualization environments running with multiple LUNs on a single iSCSI target, so my question is: why FreeNAS recommends to avoid multiple LUNs (iSCSI extents) exposed on the same iSCSI target? What kind of trouble could arise in case of doing so?

 

[mav@]

It is not about troubles, only some minor performance optimization. Usually initiators establish one iSCSI (TCP) connection per connected target. With separate target for every extent there will be separate connection for every extent, that may slightly improve performance on some loads. Though while it may have some little sense for 3-5 connections, obviously it won't have sense if you reach hundreds of them.

 

[Daniele B.]

It is not about troubles, only some minor performance optimization. Usually initiators establish one iSCSI (TCP) connection per connected target. With separate target for every extent there will be separate connection for every extent, that may slightly improve performance on some loads. Though while it may have some little sense for 3-5 connections, obviously it won't have sense if you reach hundreds of them.

Ok, it makes sense!
iSCSI is a network protocol, which transports SCSI commands, so the network stack needs to be considered too when thinking about performance (and, off course, we want to think about optimization most of the time).
I was more focused on the storage side of things and didn't pay enough attention to the network side of it. Obviously (now that you kindly pointed it out for me!) TCP has it's own characteristics, which can influence the performance of the I/O activity over the iSCSI protocol, depending on the load type.

I'm guessing TCP, because of its nature, can affect the available bandwidth and, in case of a high number of concurrent connections, can even affect the response time.
Now, if the network connection gets congested and packets start to get dropped, TCP connections will need to throttle and data will have to be retransmitted, delaying the response to SCSI commands issued by client machines.

So: the FreeNAS documentation is basically recommending to be careful with the amount of extents you associate to every iSCSI target, because it could affect performance. Maybe 1:1 is a little extreme as recommendation, but generally speaking should be ok. Off course, when implementing a storage system I must consider my own specific use case and make a decision.

 

[TheRealGreg]

Maybe it's a FreeNAS recommendation because there's an actual issue with having more then one extent associated with one target. Case in point... my ESXi cluster. Read on...

Cliff notes version: Having multiple Extents associated with a Single target caused major problems for me with Windows and ESXi. So now I do 1:1. It's slightly more work upfront, but doesn't result in storage outages for me.

More details:
For years now I've had multiple Extents associated with one Target, and all these extents presented as Datastores to my ESXi hosts. Everything ran great until just a few days ago when I started making changes. I was migrating to a new datastore. After cleanup, I gracefully unmounted the old datastore and detached the device. But as soon as I deleted the Associated Target in freenas, all my other datastores went offline - essentially yanking the disk from all VMs on them. I was like WTF!? I spent hours trying to get them back online and nothing worked. I ended up rebooting both ESXi hosts and all was good again (I chalked it up to a glitch and moved on).... until I made another similar change and same damn thing happened.

Since I'm not a legit storage guy, I didn't know what was wrong and just assumed I did something wrong with FreeNAS. A day or so later, I started iSCSI migrations on my Windows servers. I presented the new extent using the same target and everything showed up as expected. After successful migration, I deleted the Associated Target in FreeNAS and wouldn't you know... the new iSCSI disk in Windows went offline. Luckily all I had to do was bring it back online and all was good. But now I noticed a pattern.

This is when I created a target-extent 1:1 mapping for one of my Windows servers, presented both iSCSI disks, and then deleted the Associated Target for one of them in FreeNAS. But this time, the remaining one remained online and fully functional. Light bulb moment. I rinsed and repeated with all Windows servers and ESXi hosts, tested, and now I can safely delete Associated Targets in FreeNAS all day long without affecting any other Datastores or Windows disks.

Not sure if this 1:1 recommendation is just a FreeNAS thing or an industry standard. I guess I'll be asking some of my storage buddies in the coming weeks.