iSCSI, Snapshots and available space

[NugentS]

Hi,
I am fairly certain I know what is wrong here / what I am doing wrong, but thought I would post, almost as a sanity check to myself.

Storage Pool: SSDPool
3 Mirrored VDEVs of Crucial 1TB MX100 SSD's
Log on Optane
Cache on Optane

There is a single ZVOL for iSCSI (VMWare) that uses approximately 80% of the available space on the Pool (with no snapshots)
VMWare shows that I am using 346GB of 2.09TB Capacity which seems about right
I snapshot the Pool (in TrueNAS) on an hourly basis and keep the snapshots for three days which in a steady state works just fine.

I have just migrated to TrueNAS-12.0-U5 which involves migrating all active VM's to what I call "swing storage", aka another iSCSI target on a different NAS, I can then upgrade and then reboot the TrueNAS without the VM's having a hissy fit. I then migrate the VM's back. All good except that shortly afterwards I get an alert saying the snapshot task fails due to lack of disk space.

My understanding:
The process of removing the VM's and then replacing the VM's creates a snapshot scenario of equal size to the VM Size, this coupled with the original snaphot means that my snapshots are taking up a minimum space of 2*Total VM Space (2*346GB=700GB) along with any normal hourly churn. This space MUST be coming from the 501GB space left in the pool which is clearly impossible. The numbers get even worse if I do this more often inside the snapshot lifetime

Solution:
Either add another VDEV to the pool but don't change the size of the ZVOL to give more space (expensive, and given the usage, unnecessary)
Or reduce the size of the ZVOL down to 1TB giving lots of space for the snapshots to live in (cheap)

Am I thinking this through correctly or have I lost the plot and got this wrong?

 

[NugentS]

I have switched to using NFS rather than iSCSI. This released a load of space and snapshot are no longer an issue.
Also
Testing says that NFS is much faster than iSCSI - according to tests run from inside a VM

 

[Heracles]

There is a single ZVOL for iSCSI (VMWare) that uses approximately 80% of the available space

This is overloaded. For block storage like this, you should not go over 50% if you are looking for performance.

For your use case, you may be better to snapshot the VMs instead of the ZVol. What is your need exactly ?

Usually, the need is more to backup data first, config as a second. To backup OS and Apps (or free space) is less and less interesting. As such, you may be better to put your data in TrueNAS and share them (SMB, NFS, Nextcloud, ...) and then backup the data from there. As for backing up configs and settings, there are tools that are meant for that.

By taking snapshots of the zvol, you take snapshot of your VM's free space as well. Surely not optimal...

 

[NugentS]

My need - its to play (and learn). This is not a commercial system - more of a toy to keep me frustrated
I do not store data (SMB files) in the VM's - that's what other datasets are for (I never store SMB data on Virtual Server's - always use a NAS directly). VM's provide services (AD, Firewalls, PRTG etc)
The dataset was at 80% - but I am only using a free hundred MB of actual space in the datastore
VM's are all thin provisioned - so no major issues with snapshots
I snapshot the datastore as an emergency "oh my god - go back to where I was", and so I can replicate to my secondary TrueNAS. I also backup the VMs to a Synology NAS using the built in Synology Active Backup - which just works (so far)

I have 6 * 1TB SSD's in 3 mirrored vdevs. So (in round numbers) 6TB RAW, 3TB Useable and at 50% 1.5TB.
NFS seems a better choice all round here!!