Snapshot questions

[pricemc1]

Hi all,
I'm new to TrueNAS and I'm trying to understand somethings about how it handles snapshots. I have read a few things on here and in the OpenZFS docs but it still leaves me a bit confused.

My first question is: At what level is snapshot data stored? I assume the answer would be either at the pool or dataset level but its not been clear to me.

Question 2: How do I reserve space for storage of snapshots for a specific zvol/dataset? I I want to have space allocated so that snapshots dont overflow into the usable data space of the zvol/dataset. In NetApp terminology they would call this Snapshot Reserve, but I'm not sure what its called in TrueNAS and how its configured. In my case the goal would be to reserve space equal to the size of the actual zvol/dataset file system max capacity. The reason for doing this would be so that no matter how much change happens there would always be enough room to keep at least one snapshot. I recognize this is a very unlikely scenario in most cases but in my case I absolutely positively don't want to lose the ability to have at least one recoverable snapshot at all times.

Question 3: How does TrueNas handle things when space available for Snapshots is full? Typically most vendors seem to have an option to delete older snaps to make space for new ones or just stop taking snaps once it runs out of space to preserve access to data. If TrueNas has both of these options then how do I configure them?

Appreciate any replies received...

Thanks,
pricemc1

 

[Ericloewe]

My first question is: At what level is snapshot data stored? I assume the answer would be either at the pool or dataset level but its not been clear to me.

You've got the layering mixed up. In broad terms, you have:

1. OS block devices
2. The pool layer, which is a fancy key-value store
3. The Dataset and Snpashot Layer, which implements POSIX-compliant filesystems, zvols and snapshots

Snapshots are little more than the root pointer of a dataset that was preserved instead of being deleted as the dataset's state evolved.

Question 2: How do I reserve space for storage of snapshots for a specific zvol/dataset? I I want to have space allocated so that snapshots dont overflow into the usable data space of the zvol/dataset. In NetApp terminology they would call this Snapshot Reserve, but I'm not sure what its called in TrueNAS and how its configured. In my case the goal would be to reserve space equal to the size of the actual zvol/dataset file system max capacity. The reason for doing this would be so that no matter how much change happens there would always be enough room to keep at least one snapshot. I recognize this is a very unlikely scenario in most cases but in my case I absolutely positively don't want to lose the ability to have at least one recoverable snapshot at all times.

None of that applies to ZFS. Snapshots don't take up space (besides a few kB for the relevant metadata). The old data that did not get deleted is what takes up space. As such, you don't have "space for data" and "space for snapshots", nor can you restrict the size of snapshots, because that would be the same as restricting how much you can delete/overwrite.

Question 3: How does TrueNas handle things when space available for Snapshots is full? Typically most vendors seem to have an option to delete older snaps to make space for new ones or just stop taking snaps once it runs out of space to preserve access to data. If TrueNas has both of these options then how do I configure them?

Again, not a thing. Snapshots can be automatically pruned as per their configured expiration, but there is no special facility to prune additional snapshots if free space gets tight.

 

[pricemc1]

You've got the layering mixed up. In broad terms, you have:

1. OS block devices
2. The pool layer, which is a fancy key-value store
3. The Dataset and Snpashot Layer, which implements POSIX-compliant filesystems, zvols and snapshots

Snapshots are little more than the root pointer of a dataset that was preserved instead of being deleted as the dataset's state evolved.

None of that applies to ZFS. Snapshots don't take up space (besides a few kB for the relevant metadata). The old data that did not get deleted is what takes up space. As such, you don't have "space for data" and "space for snapshots", nor can you restrict the size of snapshots, because that would be the same as restricting how much you can delete/overwrite.

Again, not a thing. Snapshots can be automatically pruned as per their configured expiration, but there is no special facility to prune additional snapshots if free space gets tight.

Thanks for taking the time to reply.

I recognize that the snapshot itself is metadata that takes up very little space, but the blocks that the snapshot references and reserves do take up space because they can't be deleted until there is no snapshot that no longer references them. If my data has a 50% change rate between a given pair of snapshots and those snapshots are happening pretty frequently then a pretty sizable amount of space is starting to get consumed by the blocks being referenced by each snapshot. Where the space for those blocks comes out of becomes important. If the space comes out of the same area that is used by the active file system then I'm going to see my OS thinking that it has X amount of space free when in reality it has Y amount of free space. I would refer to that as "snapshot overflow" but maybe there's some different terminology in the TrueNAS world? How do I avoid that? With storage systems that have a concept of "snapshot reserve" I can insure that my active file system always has a guaranteed amount of space so that no matter what happens with snapshots the only way the active file system is going to run out of space is if my operating system's activities causes it too (i.e. I copy to much data to my active file system for instance). So again I ask, am I missing something here in regards to TrueNAS and any kind of similar capability/function?

In regards to options for when space begins to run tight. I have some QNAP devices that run ZFS. They have the option to stop taking snapshots when space is full or overwrite older ones to free up space. Based on what you are saying TrueNAS doesn't have similar capabilities and I need to basically manage space myself to insure things dont break when space becomes short. Am I understanding you correctly?

Thanks,
pricemc1

 

[Ericloewe]

If the space comes out of the same area that is used by the active file system then I'm going to see my OS thinking that it has X amount of space free when in reality it has Y amount of free space.

In a way, though that doesn't sound like a sustainable condition. It's an odd workload...

With storage systems that have a concept of "snapshot reserve" I can insure that my active file system always has a guaranteed amount of space so that no matter what happens with snapshots the only way the active file system is going to run out of space is if my operating system's activities causes it too (i.e. I copy to much data to my active file system for instance). So again I ask, am I missing something here in regards to TrueNAS and any kind of similar capability/function?

That's the only way you can run out of space, since the snapshots take up zero space when created.

They have the option to stop taking snapshots when space is full

Conceptually possible, not implemented in TrueNAS. You could probably hack it together with the API and a script.

overwrite older ones to free up space

That's not a thing, I guess that would technically be deleting snapshots. Possible, but also not typically done, mostly due to tricky interactions with replication and a general distaste for automatically deleting user data. Again, could be scripted.

Based on what you are saying TrueNAS doesn't have similar capabilities and I need to basically manage space myself to insure things dont break when space becomes short. Am I understanding you correctly?

I guess you could put it that way. Typically you would have a time-based retention policy that keeps things stable over long-enough periods that you can plan for expansion, rather than a usage-based policy.

 

[sfatula]

Well, they do take up around zero space when created but the space used by snapshots over time can be significant. Here's one of mine:

zfs list -o used,usedbysnapshots,usedbydataset tank/Data USED USEDSNAP USEDDS 161G 58.1G 103G

So, the dataset has 103G of data, but snapshots are consuming 58G as they track changes over time.

 

[Ericloewe]

Yeah, but you can't know that in advance, that's my main point.

 

[sfatula]

You really never want to get to low space on ZFS. Really, if you are over 70 maybe 80 at the high end, you will suffer. And if you get to 100%, you're in big trouble usually (more than other filesystams). So, speaking to the OP here, your issue is not at all what you think. You should never be worried about snapshot space as that indicates you don't have enough space in the first place.

 

[winnielinnie]

So, the dataset has 103G of data, but snapshots are consuming 58G as they track changes over time.

As @Ericloewe mentioned to the OP, it still applies:

that doesn't sound like a sustainable condition. It's an odd workload...

That's not the fault of snapshots, per se. Let's pretend you weren't taking any snapshots. The only thing that exists (and is retrievable) is what lives in your live filesystem. "But that's all I really care about." If that's the case, you don't need (or don't need that many) snapshots for such a workload.



As for not wanting to "run out of space", you could set a quota policy on the dataset. What's more important is to change/understand your workload to flow with ZFS and snapshots.

 

[sfatula]

As for not wanting to "run out of space", you could set a quota policy on the dataset. What's more important is to change/understand your workload to flow with ZFS and snapshots.

Exactly my point! He's worried about the wrong thing, at least the way I read it. Snapshot space can be managed by quotas or even refreservation. Not familiar with netapp but maybe that's basically the same thing. (edited like 4 times, how many typos can I make)

 

[winnielinnie]

(edited like 4 times, how many typos can I make)

Take a snapshot of your post now, since you fixed the typos. That way if you edit it again and type more mistakes you ca... wait...

 

[sfatula]

Take a snapshot of your post now, since you fixed the typos. That way if you edit it again and type more mistakes you ca... wait...

Well, I actually have a health excuse but not wanting to share publicly. (No typos!)

 

[pricemc1]

Thanks all, to those who have commented so far.

I wanted to ask about the "reserved space for this dataset settings" setting. If I set this value does that effectively reserve space outside of the active file system for anything other than standard data? If I set it, would snapshots use that space?

 

[Ericloewe]

I wanted to ask about the "reserved space for this dataset settings" setting. If I set this value does that effectively reserve space outside of the active file system for anything other than standard data? If I set it, would snapshots use that space?

In literal terms, it imposes a quota on other datasets such that the reservation is enforced.

That said, I was going through the docs for refreservations and it seems that they might do what you want to do:

Code:

refreservation=size|none|auto
       The minimum amount of space guaranteed to a dataset, not including its
       descendents.  When the amount of space used is below this value, the
       dataset is treated as if it were taking up the amount of space
       specified by refreservation.  The refreservation reservation is
       accounted for in the parent datasets' space used, and counts against
       the parent datasets' quotas and reservations.

       If refreservation is set, a snapshot is only allowed if there is enough
       free pool space outside of this reservation to accommodate the current
       number of "referenced" bytes in the dataset.

       If refreservation is set to auto, a volume is thick provisioned (or
       "not sparse").  refreservation=auto is only supported on volumes.  See
       volsize in the Native Properties section for more information about
       sparse volumes.

       This property can also be referred to by its shortened column name,
       refreserv.

That said, the caveats about easily breaking replication would seem to still apply. Also, I won't pretend to understand the specifics of how this is accounted. It's a bit of a weird one, because it combines two seemingly very orthogonal things:

1. Reserve space without accounting for descendents [the typical use of this property]
2. Reserve space so that snapshots do not affect the reservation

I suspect that the word "descendants" is doing extra duty and includes snapshots. I'm not sure how consistent that is across the ZFS docs, but it's not unprecedented.

 

[sfatula]

Let's go with an example. Say you want 100G for a dataset (space) reserved, and, 50G for snapshots.

zfs set reservation=150G dataset zfs set refquota=100G dataset zfs set refreservation=100G dataset

The refreservation guarantees space (but not for snapshots & clones leaving the rest avialable for that) and the quota limits space. The reservation guarantees total space for everything. If I understand the OP correctly, this is what he wants?

Note that this example is for a dataset with no descendents.

 

[pricemc1]

Let's go with an example. Say you want 100G for a dataset (space) reserved, and, 50G for snapshots.

zfs set reservation=150G dataset zfs set refquota=100G dataset zfs set refreservation=100G dataset

The refreservation guarantees space (but not for snapshots & clones leaving the rest avialable for that) and the quota limits space. The reservation guarantees total space for everything. If I understand the OP correctly, this is what he wants?

Note that this example is for a dataset with no descendents.

I'm still learning the GUI at this point. I haven't had much time to delve into the CLI as of yet. Is it possible to accomplish the equivalent of these 3 commands through the GUI. The quota one I see that you can and it seems like one of the others you can as well but I'm not sure which one and if there is a different place for the third one?