Dangerous .zfs/snapshot directories

[mpfusion]

A knowledgeable forum member mentioned that you should never enter the
.zfs/snapshots/… directories:

“Going inside those directories can be a major nightmare and can be pretty
devastating to your pool. So it is NOT something you should think of doing,
ever.”​

1. What devastating can happen entering those directories? Going there doesn't change a single file and shouldn't do any harm AFAIK. Even reading the files with atime enabled, snapshots are read-only, so it shouldn't matter at all.
2. What can happen restoring files (copying files from within .zfs/snapshots/…) to the main file system?
3. Why are the snapshots exposed in such a convenient way if entering those directories is that dangerous to the data?
4. If it's that dangerous why doesn't the manual explicitly warn against entering those directories or restoring files that way?

I also failed to find any Oracle or FreeBSD documentation about avoiding those
directories. Other FreeNAS forum members also seem to think it's OK to enter
the directories. That's at least what I found while searching the forums:

https://forums.freenas.org/index.ph...on-zfs-searching-snapshots.20186/#post-114941
https://forums.freenas.org/index.ph...fs-snapshots-actually-stored.3919/#post-15500

 

[jgreco]

I'd call it a mysterious FUD claim. ZFS explicitly goes to a lot of effort to expose the snapshots as part of the filesystem namespace. Snapshots are exposed as a read-only namespace, so there's no chance of writing data and damaging a snapshot. However, there is some chance of screwing with the automatic snapshot management, because if you NFS export a snapshot, it cannot be automatically destroyed and I have no idea how the FreeNAS middleware code will handle that. Best guess is that it'll orphan the snapshot and flag an error, which means you probably need to clean up the mess manually later.

So to answer point by point

1) With the caveat that automatic management can be impacted as noted, it should not result in actual harm. atime is ineffective within a snapshot, by the way. The meta is read-only as well.

2) Your main filesystem's files are overwritten and your previous files are restored.

3) ZFS goes to a lot of effort to expose snapshots in this manner because there are two different primary ways administrators want to restore data: a filesystem at a time or individual files at a time. Restoring an entire filesystem is handled by the ZFS snapshot manipulation commands. Restoring individual files is handled through the .zfs/snapshot mechanism. The .zfs directory is deliberately invisible so that conventional system utilities (think: find) do not go totally insane. This is a carefully designed interface that someone thought a lot about and then went to a lot of trouble to implement.

4) Because absent some compelling description of some new and previously unknown catastrophic bug, it's perfectly fine. If it wasn't fine, they wouldn't have gone to all that effort to expose it within the filesystem namespace.

Do note that I'd categorize it as "probably a bad idea" to NFS export a snapshot if your system is configured to manage snapshots automatically. The FreeNAS snapshot and replication systems are a bit twitchy in my experience. It is an appliance, and it is not necessarily expecting you to do clever things.

 

[fracai]

I agree with the FUD characterization. I got CJ to elaborate on the claims of danger and it basically amounted to two anecdotes of users losing data after having accessed those directories. For me, this amounts to stating that you shouldn't ever create any data you care about because there's a chance it'll be destroyed and you'll feel bad for having lost it.

I understand and can appreciate being concerned about preventing data loss, but without documented, repeatable bugs there are any number of issues that could have lead to data loss and been misattributed to accessing the .zfs/snapshot directories.

 

[jgreco]

As far as I'm concerned, then, that's the same as problems with users losing data after accessing any other directories.

I can definitely picture there being certain problems due to the interactions of complicated software systems, such as the automated snapshot system, but ZFS itself is absolutely designed at a basic level to provide access to that data. You can absolutely tease complicated software systems into doing strange, bizarre, and incredible things, but I'm going to draw the line at random anecdotal evidence secondhand originating from unknown parties. There's very possibly other factors at play here that might have appeared to cause data loss or other corruption, but the .zfs/snapshot feature is a basic ZFS "this is Expected to WORK" type of thing.

 

[mpfusion]

That seems reasonable, otherwise I probably would have found some post about this issue before.

My understanding so far:

1. It's safe to enter the .zfs/snapshots/ directories and restore the data within.
2. Sharing the snapshots via CIFS is supported as well (a feature of CIFS).
3. Sharing the snapshot directories via NFS, on the other hand, is not supported (Is this documented somewhere? I can't find it in the manual.), but should not result in catastrophic data loss if done anyway.

Thanks to both of you for the input.

 

[jgreco]

1) I'd say "read the data within." Restoring is a more specific subset involving a read from the snapshot and then a write somewhere else (often with a namespace similarity). The word "restore" is too specific.

2) CIFS does have functionality integrated to allow access to ZFS snapshots. As far as I'm aware, that's expected to work, even if in practice it doesn't always (I think this is still an unresolved bug. It isn't dangerous to your data.)

3) There are lots of things that are unsupported. As an example, exporting an ext3 filesystem via NFS is unsupported, and also not documented as being unsupported anywhere, TTBOMK. Typically Dru documents the things that ARE supported, because documenting all the unsupported things would result in an encyclopedia-sized volume. I expect that best practice for the NFS export of a snapshot would be to export it as read-only.

 

[mpfusion]

As an example, exporting an ext3 filesystem via NFS is unsupported, and also not documented as being unsupported anywhere

I didn't even know FreeNAS can even handle ext3, I thought it's ZFS only. But I'm not using ext3 anyway (at least not that I'm aware of).

I expect that best practice for the NFS export of a snapshot would be to export it as read-only.

I'm not sure what that will change. The snapshots are read-only by nature. So the data won't change, if the NFS mount is rw or ro, one cannot change the snapshots. The problem with NFS exported snapshots, if I understand correctly, is that the snapshot may not be purged automatically because a user has the directory open and FreeNAS would mark it as “busy” and deleting it fails. This is unrelated to it being exported rw or ro. It might be that you get better error reporting because the client immediately “knows” that the mount is read-only.

But I'm not exporting the snapshots, so it doesn't really matter at this point if it would work or not.

 

[mpfusion]

Ignore my advice if you want. Just don't expect me to give a crap when it blows up in your face.

Actually I'd like to understand what I'm doing and what is supported and what isn't. And I failed to find any information on entering the .zfs directories being dangerous, hence this thread.

I'm not ignoring you, I'd like to learn. What I was doing might have been the wrong approach, or totally unsupported or supported in general, but because of bug #xyz at the moment rather dangerous until it is fixed.

I'm really not going to write more on this topic. Take my advice or leave it.

Actually you wrote nothing on this topic. At least I just couldn't find anything. The only thing I found is that you wrote to “not think of doing this ever!” which makes it sound very dangerous and something every ZFS user should be aware of.

There is actually a bug ticket where someone mentioned the problem…

A bug ticket mentioning this issue would be very interesing. I'd be happy to read it. That would mean the problem is known, documented and can possibly be reproduced.

 

[cyberjock]

If you'd like to talk about it, I'd be willing to since it's very complicated. Talk to DrKK to get into his Mumble server and we can talk about it sometime. But it's really lengthy and I'm just not in the mood to write a book today (nothing personal, I've written about this in depth before and I don't really want to hit it again).

 

[fracai]

With cyberjock's permission I'd post the conversation we had about this, but there isn't really any additional information than this thread anyway.

There are other threads that hint at this issue with similar levels of expression. In general I respect CJs advice, however harshly provided, but in this instance there just isn't any substance. In all my research I haven't seen any outside confirmation of an issue and nothing to suggest these directories aren't intended to be used or that an issue here would be anything but a bug that should be verified and fixed. It's fine to exercise caution, but the same symptoms can be attributed to any number of causes and I don't see any reason to attribute it to accessing a read only directory.

https://forums.freenas.org/index.php?threads/how-to-find-a-deleted-file-in-snapshots.23882/

I know it'd be an annoyance to write up, but I'd be interested in reading a complete account of the issue. For all the references to this being written about in depth, I haven't been able to find anything.

 

[cyberjock]

fracai, you are welcome to post what I said. I doubt it's much help though. ;)

 

[fracai]

Agreed, there isn't much there. But, if this thread collects everything known about this issue and actually produces some sort of knowledge it's worth the effort. When I get chance I'll probably take you up on the offer of discussing this via Mumble.

I've seen you reference a not-to-be-talked-about .snapshot directory a couple times now. Most recently this was in regard to locating a deleted file.

I fully understand that it's likely a dangerous procedure, but I'm curious.

Could you provide more information regarding this "feature", or a link to where it is described? Or even a non-ambiguous name or search term?

I've been around long enough to know that I'm my own data's worst enemy and I respect that enough not to do any thing rash.

That said, I'd still be interested in learning more about things that could destroy my data if I blunder in to them.

Thanks for your help.

Haha. You're right that for most people they are their data's worst enemy.

So when a snapshot is taken a file structure is created from the transaction that is dumped in the root directory of the given file system named .zfs. Inside .zfs is a directory called snapshot. Inside that is a list of all snapshots that exist for that file system, by name. Inside each of those directories is a copy of the directory structure as it was when the snapshot was made.

Now here's the messy part. You can, in theory, play with those files. Unfortunately it has serious consequences if any kind of write is made to anything inside there. chmod, chown, touch, etc would all backfire in a very nasty way. Consequences range from nothing to an immediate panic to everything looks fine, and then later when ZFS figures out things don't match up you have an unmountable pool. :p

I have no doubt if you Google ".zfs snapshot directory" you'll find plenty of info on the topic as well as tons of "do not go in there" type stuff. ;)

Whoa, seriously?
The dangerous directory you're talking about is the "<dataset>/.zfs/snapshot" hierarchy? That's not read only!?
I go in there all the time. How can that not be read only?

Yeah, it is but isn't. Hard to explain and I don't fully understand the machinations under the hood. I just know I've personally seen 3 pools go bad in the last month because people would change the permissions of a file that were incorrect to something more limiting and realize that people could still access the file via the snapshots. So they'd then go in and start trying to change the snapshot data. It doesn't end well. ;)

To make matters worse, sometimes you can run the command and it won't give an error or anything, but days or weeks later the pool will fail and you'll have no clue what went wrong because it was days or weeks ago.

OK, this sounds like it's just a bug then. Like, sometimes an edge case leads to the the snapshot not being mounted read-only.
Now we just need to find someone willing to test this to narrow down what triggers it.
I feel, marginally, safer and, marginally, less safe. I think that sums up my entire computing experience.

There is someone out there that has some code to fix this, but it creates new edge cases. Last I hear it's a known problem with ZFS and a fix is planned, but there is no ETA at present. I just take the conservative side and never go in there. Especially on someone else's server. ;)

 

[mpfusion]

I agree with @fracai and conclude that this issue at the moment lacks substance:

1. There is no bug report describing this very issue.
2. No verifiable information can be found on the net.
3. CJ has seen three pools go bad and assumes reading a read-only directory was the cause. This has not been verified in any way.
4. He goes “So they'd then go in and start trying to change the snapshot data.” and “…serious consequences if any kind of write is made…”
   This issue is not at all about deliberately changing the snapshot data, but about *reading* the .zfs/snapshot directories.
5. “There is someone out there that has some code to fix this…” This also seems like a very vague statement.

I do believe there might be a problem or a bug. But the only reference we have so far is that CJ saw someone lose his the pool for an unknown reason. Furthermore, he did not see anyone entering the directories and then lose the data. And that's what this thread is about.

So unless it's confirmed that this is indeed an issue I'll keep using those directories. Sure, it's a risk, but what are the chances?

And if something goes belly-up, that's what you have backups for.

 

[SirMaster]

If using the .zfs/snapshot directory is dangerous then it must be a FreeBSD/FreeNAS bug, because on other platforms like Solaris and Illumos, the official documentation tells you to access snapshot data through these directories.

Although even on FreeNAS, the SMB service is accessing those directories to provide previous file version access so I can't really imagine it's dangerous.

 

[jgreco]

I'm going to go so far as to say that I can imagine maybe there's some issue if you try to do something stupid in the snapshot directories such as change things. It is supposed to be all read-only though.

 

[IAPro]

I'm really sorry about digging up an old thread but this is the only place I can find anything about this. I've been using ZFS for years but the fact that it's even possible to write to anything under .zfs came as a surprise to me. I wrote not to one of the snapshots but to the .zfs/snapshots directory itself by mistake.

I'm looking to find out if anyone knows exactly what this does and do I need to rebuild the pool. I have backups via zfssend but am not sure if I make a current backup using zfssend that it will capture the brokenness.

There do not seem to be any adverse affects right now except a snapshot named the same thing as the directory I mistakenly tried to write there (I don't dare delete it without some offline testing of the replicated pool) but I am extremely concerned about this biting me days, weeks or months later, especially after backups have rolled over.

If the pool will zfssend/recv and export/import does that mean I'm safe?

This is on FreeBSD 10.3-RELEASE-p29. Yes I know I need to update. This server provides VMware datastores via NFS for a couple hundred VMs and scheduling downtime is not easy.

Edit: I duplicated exactly what I did in a test pool on a 10.3-RELEASE-p29 host. It seems to create one directory there and that appears to be where it stops, but I don't know for sure:

Code:

root@storage01:/home/user # rsync -aHSv testdir /testpool/.zfs/snapshot
sending incremental file list
rsync: failed to set times on "/testpool/.zfs/snapshot/testdir": Read-only file system (30)
rsync: recv_generator: mkdir "/testpool/.zfs/snapshot/testdir/testsubdir" failed: Read-only file system (30)
*** Skipping any contents from this failed directory ***
testdir/
testdir/testsubdir/

sent 18,008 bytes  received 298 bytes  36,612.00 bytes/sec
total size is 2,704,859,702  speedup is 147,758.10
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1200) [sender=3.1.2]
root@storage01:/home/user # zfs list -t snapshot | grep testpool
testpool@testdir 0 - 19K -
root@storage01:/home/user # ls -l /testpool/.zfs/snapshot/guacamole/
total 0

I can zfs destroy the "snapshot" in the test pool, export it and import it again, so I'm hoping I'm OK. I have backups and am backing up again, I'm just concerned about some sort of unseen corruption that is going to bite me later on.

Interestingly 11.3-RELEASE-p5 does not allow this idiocy at all, stopping me at making the directory under snapshot with rsync: recv_generator: mkdir "/testpool/.zfs/snapshot/testdir" failed: Operation not supported (45).

 

[fracai]

So, this would be the first indication that there's actually a bug here. You didn't post any example output, but I'm assuming your first rsync test actually transferred files in to the .zfs/snapshot directory? And 11.3-p5 allowed a directory to be created, but nothing more? I'd report all of that to the FreeNAS bug tracker and maybe upstream as well.

 

[IAPro]

It didn't seem to do anything on 10.3 except create a directory under the .zfs/snapshot directory. Which shows up if you do a zfs list -t snapshot. Which you can then remove with zfs destroy. It did not seem to allow any files or directories to be created under that directory nor did it allow any files to be created directly in .zfs/snapshot. This is easy to duplicate, simply install FreeBSD 10.3-RELEASE-p29, create a test pool, and just mkdir a directory under the snapshot dir.

To be clear, this is only on FreeBSD 10.3 that I was able to duplicate this creation of the first directory. FreeBSD 11.3 prevents even the creation of the first directory, as I would have expected would be the case. Again, 11.3 does not allow the creation of anything, file or directory, under .zfs/snapshot. And absolutely none of this involved any attempt to modify, create or delete any other directories or their content under .zfs/snapshot, so I cannot speak to that.

What really concerns me though, was that the rsync, which was fairly large, was still running when I realized what I had done, probably a few minutes after I had started it. My attempts to duplicate this in a test pool to see what actually happens does seem to take an amount of time commensurate with the size of what I'm trying to rsync into the directory, but then everything errors out and there's nothing there but the first level directory. At no time does the size of the used space in the pool as shown by zpool list grow though.

I have not been able to get anything to behave badly otherwise - zfs send/recv and import/export of a small test pool work OK, creation and deletion of other snapshots in concert with this work, etc. It's like it just creates the initial dir and then writes into the ether - until it finishes and then you get a whole bunch of rsync: mkstemp "filename" failed: Read-only file system (30). So I'm not completely sure what is happening without being intimately familiar with the zfs internals and code, hence my question.

EDIT: I did try just using cp to try to copy a 1GB file into place. It does fail immediately and I'm guessing the delay was possibly rsync reading the source files?

I suppose another thing I could try is to write a known pattern to a file under the initial dir, export the test pool and then analyze the file that it's built in to see if the pattern appears... But at this point I think I may have duplicated a bug that has since been fixed in FreeBSD 11.3 that doesn't seem to be, at least at this point, destructive. I'll keep an eye on things and report back.

This has been the first time I've ever thought that some sort of fsck-like utility might have been a good idea. ZFS has always been 100% dependable and rock solid for me without one, over almost 20 years, and many, many petabytes of reads and writes that my career and reputation have depended on, in Solaris and then in FreeBSD. Bonwick and his team, and all of the contributors since, are genius.

 

[fracai]

Just noticed that this was on v10. Well there's your problem!
Good to know it doesn't seem to affect 11.3

I'm going to assume rsync takes so long because it keeps trying to transfer data and is denied writing.