zvols and free space required for the backing filesystem

[0n0w1c]

How much free space is needed for zvols?

Say I have 100TB of space. If I create zvols to export via iscsi, how much free space is required?
Can I make (10) 10TB zvols? Or do I need to reserve 15-20% and only create 8 zvols? This amount of free space seems excessive for what would 8-10 files.

I ask because we would likely mount them via iscsi and use them for zfs filesystems themselves, which of course would require an additional 15-20% free space.

 

[HoneyBadger]

I ask because we would likely mount them via iscsi and use them for zfs filesystems themselves, which of course would require an additional 15-20% free space.

Can I ask what the "bigger picture" solution you are hoping to build is? You would likely benefit more from setting up separate ZFS datasets on FreeNAS itself rather than doing ZFS on zvol on ZFS. This would also avoid excess fragmentation.

In regards to the free space question you could create sparse zvols and leverage compression to still "have 15-20% free space" in the main pool while having 10x10TB logical space available downstream, but I return to my question above regarding the bigger picture.

Welcome.

 

[0n0w1c]

We have acquired three identical freenas systems, two for our main site and the third at a remote site.
Ideally we would mirror between the two local freenas units and zfssend snapshots to the remote site.
We have approximately 80TB of binary data, compression does not help much.

In the past, we have used 10TB volumes served up via iscsi from our current backend to an NFS head, which then performs the mirroring and serving of the data.

So we desire an NFS head maintaining zpools of mirrored datasets. How best to do this with freenas appliances?

 

[HoneyBadger]

So you want the NFS head to have its own ZFS pool, using mirror vdevs made from the 10TB ZVOLs cut from the two FreeNAS appliances. I assume this is for high-availability reasons so that you can update/patch/reboot a FreeNAS appliance without causing downtime to the NFS head and filesystem?

I would have suggested asking iXsystems for a quote on TrueNAS which has integrated HA support in an active/passive controller setup, then you can serve the NFS exports directly from the TrueNAS box and replicate to a second one offsite. Have you already purchased the 3x FreeNAS appliances?

You can certainly do this by exporting ZVOLs to an NFS head and mirroring there as well, but it's a lot of overhead. You're effectively using 4 drives to get the usable space of 1 (assuming 2-way mirrors in each of the two FreeNAS appliances, and then mirroring again at the NFS head)

Regarding the free space, it has to do with free space fragmentation and sustained write performance. ZFS tries its best to write into sequential chunks of free space, but over time the space becomes more fragmented because of the copy-on-write behavior, so your incoming writes will slow down. "The more full a pool gets, the slower it tends to perform."

 

[0n0w1c]

We looked at the TrueNAS option and seriously considered it but in the end decided to go with the FreeNAS units.

So then the backing store, even for zvols requires the 10-15% of free space? I was hoping a block device would not fragment the backing filesystem and the free space would not be required.

 

[HoneyBadger]

Yes, the pool free space is what fragments.

You will also have to deal with that on the filesystem at the NFS head, and layering two copy-on-write filesystems exacerbates this. The top level will write to "free space" wherever possible, but that will then be spread across the underlying ZVOLs which will have no way to identify sequential files or regions.

Is it possible to use a single FreeNAS appliance as the NFS exporter and do more frequent snapshots to the second onsite unit, or would this setup be considered an unacceptable single point of failure?

 

[0n0w1c]

We will likely go with the NFS export and then snapshot to the other two units.

 

[HoneyBadger]

We will likely go with the NFS export and then snapshot to the other two units.

This will definitely take out a lot of layers of complexity, and likely make the system more efficient in general (no layered CoW, ZFS will be aware of the filesystem for prefetch and caching)