iSCSI "80% capacity" rule....?

[statikregimen]

Good day,
From 11.3 documentation (https://www.ixsystems.com/documentation/freenas/11.3-U1/sharing.html#iscsi-wizard), it states "For performance reasons and to avoid excessive fragmentation, keep the used space of the pool below 80% when using iSCSI.".

I haven't been able to find any more information past this. Before realizing this potential limitation, I bought 12x4TB HDDs, thinking I'd get a pool around 30TB, that I blindly assumed I could share via iSCSI (after RAID-z2 overhead)...The array ended up being 32TB, but if it is truly a big deal to use more than 80% of the array, that drops me to 25TB which is a no-go for my application.

My questions are:

1- Does this still apply (I've installed 12.2...again, docs above are from 11.3)?
2- If so, can I split the pool into 2 iSCSI shares as a workaround or does iSCSI just really need that extra space? That also leads me to wonder, if using iSCSI, do I always have to keep 20% of the pool free, with no other usage of that space possible(safe/ideal)? OR, if I want to just violate this rule, what would be the potential short/long-term consequences in more detail?
3- Any extra explanation to the documentation referenced above is bonus points

Thank you for reading, and very much for any help!

 

[no_connection]

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

It does depend on what you use it for as well. 20% is safe enough to not run into unexpected trouble, going to 90% may or may not lead to unusable performance depending on what you use it for.

 

[statikregimen]

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

It does depend on what you use it for as well. 20% is safe enough to not run into unexpected trouble, going to 90% may or may not lead to unusable performance depending on what you use it for.

That's the other trouble, is that it's not really clear on how the software will be using the space. It's a highly proprietary, advanced application for the mapping industry (UltraMap), used for processing large datasets of images, GPS locations, LiDAR, and other data as a critcal part of our company's workflow. So it really could go either way - it could be an intensive, very random read/write workload or it could be a fairly heavy random read, but very tame sequential write workload (if I were to hazard a guess, I'd say the later is most likely). I'm trying to get more details on that, but we need this storage online last week, so I doubt I'm going to have my answer before I have no choice but to make a command decision on the iSCSI target size.

By using the drives we bought as spares, I've got the final iSCSI target size up to 32TB, which is at the threshold of usable for a short time while we order more drives.

Anyway, thank you very much...going to read that post you linked now!

 

[jgreco]

Once you pass the 50% mark, you are already in territory where performance of the pool will suffer in the long term as fragmentation increases. The 80% mentioned in the manual is silly for most practical uses. Fragmentation will increase more slowly if you are doing large writes and then large free's. Stuff like metadata updates, either in ZFS or within the iSCSI volume, are the worst offenders because they typically target a 4K sector, and fragmentation results because each overwrite of a 4K sector results in 4K being freed, the new 4K being written elsewhere, and then there being a 4K "hole" in your previously-sequential data. This results in read speed degradation if you try to read the entire file, because it has to seek to get that overwritten block.

You can reduce the read degradation through use of ARC and L2ARC (make sure to have at least 64GB of RAM, should be able to add at least 256GB L2ARC) but nothing can make writes faster if ZFS is having to scan the free lists for small blocks of space to store your writes.

RAIDZ is also very suboptimal for iSCSI, and will perform poorly and may consume space in unexpected ways.

The ideal thing for iSCSI are mirrors, for a variety of reasons, including that it is much easier to "fix" performance issues by adding new vdevs or resizing existing vdevs. If you want 32TB of high performance usable space, you want a bare minimum of 64TB of pool space, probably 96TB, so for mirrors that 96TB of pool space is 192TB of raw space, so that works out to 12 x 16TB drives. This is much less likely to show significant performance degradation over time, but is of course more expensive too.

 

[no_connection]

Do you really need iSCSI? NSF or SMB could probably be used even if you need to symlink it. But I guess if they ask for iSCSI then they get iSCSI type of thing.

I have used the argument before to use stripes for read intensive data that you have somewhere else (ie it's fine if you loose it and can just transfer it back again) and mirrors for sensitive processed data.

But then again that's the "need something for the cheapest possible" way to go and only works for an extremely small set of requirements.

If it is as important as it sounds and was really needed last week then you probably have to spend the $ and cover all bases. Especially if you don't know what they are really asking. It's kinda there fault you need to spend more money by not being accurate.

 

[statikregimen]

Do you really need iSCSI? NSF or SMB could probably be used even if you need to symlink it. But I guess if they ask for iSCSI then they get iSCSI type of thing.

I have used the argument before to use stripes for read intensive data that you have somewhere else (ie it's fine if you loose it and can just transfer it back again) and mirrors for sensitive processed data.

But then again that's the "need something for the cheapest possible" way to go and only works for an extremely small set of requirements.

If it is as important as it sounds and was really needed last week then you probably have to spend the $ and cover all bases. Especially if you don't know what they are really asking. It's kinda there fault you need to spend more money by not being accurate.

I've considered NFS or Samba shares (well probably samba since the client machine runs windows), but before realizing the extent of ZFS's iSCSI requirements, I figured they'd be too slow (and again, I'm not even sure the application would like that). If we didn't already have most of the hardware to build a server (namely had the server+JBOD enclosures) already on hand from the previous IT manager who never set it up, I'd have just bought a pre-cooked solution like the EMC we currently use.

I'm still at a stage where we're not committed to anything, so I'm considering just installing a traditional Linux distro, setting up the hardware RAID controller, and setting up iSCSI manually. This seems like it would be the most appropriate for our use case, where ZFS is just kinda probably not...Especially considering that we may need to use the space for different applications in the future (namely for VMWare, where the issue seems like it'd be 10x worse).

Thanks for the reply!

Also think I forgot to thank jgreco!

 

[jgreco]

I'm still at a stage where we're not committed to anything, so I'm considering just installing a traditional Linux distro, setting up the hardware RAID controller, and setting up iSCSI manually. This seems like it would be the most appropriate for our use case, where ZFS is just kinda probably not...Especially considering that we may need to use the space for different applications in the future (namely for VMWare, where the issue seems like it'd be 10x worse).

It's only 10x worse if you insist on playing by your own rules. If you choose to play by ZFS's rules, it is capable of blowing by most other available "free" solutions, certainly easily blowing by a hardware RAID controller (which is just a microcontroller-on-a-board), and also significantly faster than many low-tier commercial offerings. But you "pay" for that by giving it lots of mirrored disks, lots of ARC, and lots of L2ARC. It's "free" in the sense that the software is free. ZFS is basically trading resources for performance, and when you give it those resources, it can do things most other solutions can't.

 

[no_connection]

I don't see why NFS or SMB would be slower. Could use multipath as well.

I don't really know what other hardware you have ether so guessing performance would be kinda hard.

I think that what you trade away with ZFS is that block IO to the system don't translate into block IO at the drive so you have to work around that.

I played around with ESOS for iSCSI so maybe give that a look unless you want something more mainstream.

ESOS - Enterprise Storage OS

ESOS - Enterprise Storage OS : An open source, high performance, block-level storage platform.

www.esos-project.com

 

[statikregimen]

Thanks for both replies...

ESOS is already on my radar. I actually tried it before we got the disks, but thought TrueNAS would be more user friendly before becomming aware of the iSCSI/block-level considerations of ZFS. My concern with NFS/SMB is that we don't know for sure how the current application(s) use the storage (or if they'd even let us use such shares), but we are certain in the future we'll be throwing VMs at it, and my research suggests that for such workloads, SMB/NFS is not ideal. Maybe this is less of an issue with a ZFS under the hood? Either way, we are certain iSCSI works well with our current applications, since we're already using it w/ our EMC SAN.

Ultimately, usable capacity & reliability at a reasonable performance/cost ratio is more important to our needs than raw performance.

The solution I've settled on, due to the time crunch we're under, is to stick to what we know works: hardware array + iSCSI, probably via ESOS (or failing that, Debian).

Thank you both very much for your help!! I still use TrueNAS at home - very fantastic product...Just wish I had more time to figure out a way to make it work for this application. Having a good web UI is so nice...but alas....