Size exceeds zvol allocation

[chuck32]

Hello again,

sorry if this a total beginner question but I probably didn't find the correct passage in the documentation.

I have two storage pools mars (two mirrored SSDs) and neptune (4 HHDs in raidz2).

I created a 75 Gb zvol for Ubuntu (or rather it got created during creation of the VM). I replicate that to my HHD pool.

Now two things I observed:
1) The total used size exceeds the allocated zvol size. I assume this is because the snapshots consume space and that used space needs to be accounted for somewhere. However I can't derive the additional space (21 GB) from the snapshots. The used space is much smaller but the referenced space (for all 3) is much larger.

2) On the HDD pool the sizes are different (although LZ4 is enabled on both). I read somewhere that this may be because raidz2 needs a different overhead in space than a mirrored vdev.

Can someone explain or point me in the right direction to understand the fundamentals of that?

I'm not really space constrained so practially I don't care too much about that, hoewever I'm curious.

Best regards!

 

[sretalla]

1) The total used size exceeds the allocated zvol size.

It's a thick provisioned volume, so was written out at creation time as zeroes.

You installed an OS on it, so that whole used size was then written out in addition (and presumably snapshotted more than once as you continued to change things).

It's a CoW system, so no "overwritten" files (blocks) are actually overwritten, rather newly laid down on the parent pool.

Deletions will only take effect if you clear snapshots referring to them, so using the current "used" space from the OS guest isn't a good measure of the required space to manage the zvol.

2) On the HDD pool the sizes are different (although LZ4 is enabled on both)

I don't see anything unusual in what you show there... looking at the referenced column, that's the data present in the zvol at those snapshot points (more-or-less), growing over time. Remembering that the original can get changes in-between snapshots which impact the size differently than the replica which gets only the snapshot changes directly.

 

[chuck32]

Thank you!

So for 1) I was correct it's the 75Gb allocated initially, the 15Gb written are used within the 75Gb. But the snapshots add to the 75Gb? Because the changed files did not in fact get overwritten. And I can't really reconstruct the additional 21 Gb (75+21 = 96) because I don't know to what extent the snapshots stored "overwritten" data and to what extent they stored new data which doesn't add but stays within the 75Gb limit?

One could discuss the sensibility of this anyway. I can (and do) prepare backups of anything important inside the VM and do not backup the VM at all. From what I see using snapshots on an OS does lead to a lot of modified files which need to be included in the snapshots.
I set a 2 week retention now with snapshots every 2 weeks.
Basically I don't need that many snapshots, I just want to have at least one snapshot if I decide it's easier to recover the VM completely vs. redeploying and then importing all backups.

If I didn't overlook it, there's no way to set the zvol to sparse if created when creating the VM. You could create it beforehand and then use the zvol during creation. In the end the space demand doesn't change so I don't see any advantage of sparse in this case.
What, on the other hand would be helpful is, if you could use a sparse volume for the backup location.

2) I would have still thought since the snapshots are the delta between two points in time, that the referenced sizes should match. Or is it because the replication has a different base set of information and the snapshots, albeit being copies and not individual new snapshots, are referenced with respect to the original image. With image I mean the original file set that got replicated.

 

[sretalla]

I would have still thought since the snapshots are the delta between two points in time, that the referenced sizes should match.

The live copy can live through many creations, deletions and modifications to files that aren't there in the next snapshot. That tends to leave the replica a little tidier than the original, which I think is the case for you.

 

[chuck32]

Thanks @sretalla but upon further investigation I'm still at loss.

I created a new snapshot of my ubuntu VM (and according to my settings that one got immediately replicated to my HHD pool.)

The snapshot used 5.11 Mib and referenced 15.26 Gib on the SSD mirror where the VM runs. On the replication pool it used N/A Mib and referenced 18.62 Gib.

So far so good, however both (the SSD pool (mars) and the HDD pool (neptune)) datasets which consist the VM and its replica increased 3 Gib in size.

So to some extent the secret has been somewhat lifted. If a 5 Mib snapshot may increase the used size by 3 Gib, snapshots that in total use 5 Gib may as well increase the size by over 20 Gib.

Probably I've got the wrong impression here, but I thought all this would need to follow some logic: Take a snapshot that used X amount of space, X amount of space is used additionally. I can't see the pattern here.

Again, currently it's not that much of a problem since I still have enough space on either pool. However from a general perspective this makes managing space for VMs kinda hard. I allocated 75 Gb, used 16 Gb of that and I am already out 98 Gb of space. Relatively speaking that's almost 25 % more space used than originally allocated. If I'd allocated let's say 180 Gib of my 228 Gib for different VMs I'd have a problem.

Again, sorry if this is a total newbie question but I am genuinly confused (as you probably can tell).

Have a nice weekend!