Looking for clarity on using iSCSI

[kitt001]

I'm aware, that for performance reasons, its only recommended to utilize only 50% capacity with regards to an ISCSI extent ... but I can't find a clear answer about where that 50% should be actually be held in reserve. I've searched, but this could be a matter of not searching with the correct terminology to yield useful results.

With an example system consisting of:
(6) 2TB spindle disks, configured as mirrors for a 6TB pool (using round numbers for simplicity).

Do you a) provision the largest possible zvol, and then limit filling it to 50% of its capacity?
Or b) ... in this example, do you simply create a 3TB zvol and let it fill up (leaving the un-provisioned space as the reserve)?

Apologies if this is an asked and answered question.

Thanks

 

[Ericloewe]

Do you a) provision the largest possible zvol, and then limit filling it to 50% of its capacity?
Or b) ... in this example, do you simply create a 3TB zvol and let it fill up (leaving the un-provisioned space as the reserve)?

I don't think it will make much of a difference to ZFS (assuming that UNMAPs are being issued properly), but the initiator side might not like having the disk look full.

 

[kitt001]

OK .. so if there isn't a hard and fast reason for it to be in one spot or another, that explains why I couldn't find it. Thanks!

 

[bestboy]

The 50% iSCSI rule is an extension of the 80% rule. Both rules are referring to the entire pool and try to deal with the fragmentation in a copy on write filesystem.
FreeNAS is going to display a warning in the GUI when the pool is filled more than 80%. However, the 50% rule is something you need to check as an admin.

Example: 10x 2TB drives as 2-way mirrors

• raw size: 20 TB
• pool size: 10 TB (100%)
  
• recommended maximum pool size: 8 TB (80%)
• recommended maximum pool size when using iSCSI: 5 TB (50%)

Given that pool you can create as many and as big zvols for your extents as you like. But the sum of their sizes should be less than 5 TB.
If you plan to create one huge zvol (which is weird), then I'd suggest to just make it 5 TB in size (i.e. answer b). Then you'll know from looking at it how big it can be at most.

 

[kitt001]

I didn't have a 'plan' to create things one way or the other, I just have a scenario that has evolved against my will, where I need to mix storage options using both NFS mounts (for Xen, and Oracle) and ISCSI extents (For ESXi and HyperV) on a single, well outfitted, FreeNAS box.
Since the Datasets infringe on the space not used by the ZVOLs, I was trying to determine the best way to ensure that all the required reserve space remained reserved. That's what started me down the path wondering where exactly the reserved space should actually be; to which there didn't seem to be an obvious answer.

 

[bestboy]

Ok I see. Well, you could add quotas to the datasets shared via NFS in order to make sure they do not grow into the 50% head room. Then you should be kinda safe, if the dataset quotas and the zvols add up to be less than 50% of the pool.

 

[kitt001]

That was the approach I was initially leaning towards before I got sucked down the rabbit hole wondering if I was going about it the right way :). Thanks for confirming my thought process though ... I second guess myself every time I commit to a plan with one of these things.

 

[bestboy]

Maybe there is even another, less cumbersome approach:
Create a dataset, reserve 50% of the pool space for it and then never-ever use it for anything :)

 

[Stux]

The 50% rule is about avoiding excessive free space fragmentation.

ZFS is faster when it can easily find new space to write new data in one continuous chunk.

50% helps ensure that.

It's not the end of the world if you use more than 50% of the pool.

At 90% full, ZFS switches allocator algorithm to a very slow algorithm (to minimize further fragmentation). Thus there is a performance cliff at 90%.

 

[DaveY]

Just want to get clarification (and confirmation) on OP since I use iSCSI heavily. So if I have a 6TB pool and I create a single 1TB zvol. Even if that 1TB zvol is filled up past 90%, it WILL NOT suffer performance hit due to the fact that there's still 5TB (83%) of free space left? Or are we saying that the 1TB zvol needs to maintain 50% free space to not suffer performance penalties? @Ericloewe may have answered it, but I want to make sure I'm understanding how ZFS treats zvols and free space.

 

[Ericloewe]

On spinning rust, ZFS write performance degrades massively as free space becomes scarce (and reads become harder due to increased fragmentation).

50% is the general recommendation for block storage workloads and it's something like 15% of top random write performance. Here's some reading: https://forums.freenas.org/index.ph...ce-from-freenas-and-vmware.42761/#post-283168

 

[DaveY]

Thanks Eric, but it still doesn't answer my question. Is the 50% recommendation on TOTAL zfs space? Or does it also apply to a zvol that may only be a fractional size of the entire ZFS pool?

 

[Ericloewe]

That's pool size. The zvol may need more overhead due to whatever filesystem the initiator is using, but it's not an issue on the ZFS side of things.

 

[DaveY]

ok great thanks for confirming!