Mounting snapshot directory using NFS from linux broken

Status
Not open for further replies.

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
Hi,
we are trying to mount from a linux box from a FreeNAS 8 server:

This is how the directory looks like on the FreeNAS 8:
[root@freenas] /mnt/pool01/rootfs64_6/.zfs/snapshot# ls -al
total 69
dr-xr-xr-x 48 root wheel 48 Feb 20 15:41 ./
dr-xr-xr-x 4 root wheel 4 Feb 20 15:41 ../
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00001/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00002/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00003/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00004/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00005/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00006/
drwxr-xr-x 21 root wheel 22 May 12 2011 00007/
drwxr-xr-x 21 root wheel 22 May 12 2011 00008/
drwxr-xr-x 21 root wheel 22 May 12 2011 00009/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00010/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00011/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00012/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00013/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00014/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00015/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00016/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00017/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00018/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00019/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00020/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00021/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00022/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00023/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00024/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00025/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00026/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00027/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00028/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00029/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00030/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00031/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00032/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00033/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00034/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00035/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00036/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00037/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00038/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00039/
drwxr-xr-x 22 root wheel 23 Nov 28 13:27 00040/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00041/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00042/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00043/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00044/
drwxr-xr-x 23 root wheel 25 Feb 15 14:18 00045/
drwxr-xr-x 23 root wheel 25 Feb 15 14:18 00046/
[root@freenas] /mnt/pool01/rootfs64_6/.zfs/snapshot#


When we mount using a debian squeeze linux box we see the following:
root@controlnode01.muc:~# mount -o rw,noatime,vers=3,nolock,noacl bgmucpo-nas02:/mnt/pool01/rootfs64_6/.zfs/snapshot /tmp/mnt/
root@controlnode01.muc:~# ls -al /tmp/mnt/
total 38
dr-xr-xr-x 14 root root 14 2012-02-20 15:41 .
drwxrwxrwt 7 root root 4096 2012-02-21 11:24 ..
crw-rw---- 1 root audio 116, 14 2011-04-20 18:42 00001
crw-rw---- 1 root audio 116, 40 2011-04-20 18:42 00002
crw-rw---- 1 root audio 116, 42 2011-04-20 18:42 00003
crw-rw---- 1 root audio 116, 44 2011-04-20 18:42 00004
crw-rw---- 1 root audio 116, 46 2011-04-20 18:42 00005
crw-rw---- 1 root audio 116, 72 2011-04-20 18:42 00006
drwxr-xr-x 21 root root 22 2011-05-12 12:22 00007
crw-rw---- 1 root audio 116, 76 2011-04-20 18:42 00008
crw-rw---- 1 root audio 116, 78 2011-04-20 18:42 00009
crw-rw---- 1 root audio 116, 104 2011-04-20 18:42 00010
crw-rw---- 1 root audio 116, 106 2011-04-20 18:42 00011
crw-rw---- 1 root audio 116, 108 2011-04-20 18:42 00012
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00013
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00014
crw-rw---- 1 root audio 116, 25 2011-04-20 18:42 00015
crw-rw---- 1 root audio 116, 26 2011-04-20 18:42 00016
crw-rw---- 1 root audio 116, 27 2011-04-20 18:42 00017
crw-rw---- 1 root audio 116, 28 2011-04-20 18:42 00018
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00019
crw-rw---- 1 root audio 116, 30 2011-04-20 18:42 00020
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00021
crw-rw---- 1 root audio 116, 56 2011-04-20 18:42 00022
crw-rw---- 1 root audio 116, 57 2011-04-20 18:42 00023
crw-rw---- 1 root audio 116, 58 2011-04-20 18:42 00024
crw-rw---- 1 root audio 116, 59 2011-04-20 18:42 00025
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00026
crw-rw---- 1 root audio 116, 61 2011-04-20 18:42 00027
crw-rw---- 1 root audio 116, 62 2011-04-20 18:42 00028
crw-rw---- 1 root audio 116, 63 2011-04-20 18:42 00029
crw-rw---- 1 root audio 116, 88 2011-04-20 18:42 00030
crw-rw---- 1 root audio 116, 89 2011-04-20 18:42 00031
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00032
crw-rw---- 1 root audio 116, 91 2011-04-20 18:42 00033
crw-rw---- 1 root audio 116, 92 2011-04-20 18:42 00034
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00035
crw-rw---- 1 root audio 116, 94 2011-04-20 18:42 00036
crw-rw---- 1 root audio 116, 95 2011-04-20 18:42 00037
drwxr-xr-x 21 root root 22 1970-01-01 01:00 00038
crw-rw---- 1 root audio 116, 121 2011-04-20 18:42 00039
crw-rw---- 1 root audio 116, 122 2011-04-20 18:42 00040
crw-rw---- 1 root audio 116, 123 2011-04-20 18:42 00041
drwxr-xr-x 23 root root 25 2011-12-07 14:05 00042
crw-rw---- 1 root audio 116, 125 2011-04-20 18:42 00043
crw-rw---- 1 root audio 116, 126 2011-04-20 18:42 00044
drwxr-xr-x 23 root root 25 2012-02-15 14:18 00045
crw-rw---- 1 root audio 116, 1 2011-04-20 18:42 00046
root@controlnode01.muc:~#

And even stranger:
root@controlnode01.muc:~# mount -o rw,noatime,vers=3,nolock,noacl bgmucpo-nas02:/mnt/pool01/rootfs64_6/.zfs/snapshot/00045 /tmp/mnt/
mount.nfs: bgmucpo-nas02:/mnt/pool01/rootfs64_6/.zfs/snapshot/00045 failed, reason given by server: Permission denied
root@controlnode01.muc:~#

What is wrong with the NFS server in combination with the .zfs directory???

Can anybody give some hints?

Thanks
Rainer
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
From FreeNAS to FreeNAS it almost works...

Testing it from a FreeNAS to a FreeNAS box:
Directory listing works as expected:
[root@backup01] ~# mount 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot /tmp/mnt/
[root@backup01] ~# ls -al /tmp/mnt/
total 70
dr-xr-xr-x 48 root wheel 48 Feb 20 15:41 ./
drwxrwxrwt 11 root wheel 1024 Feb 21 14:04 ../
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00001/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00002/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00003/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00004/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00005/
drwxr-xr-x 21 root wheel 22 Apr 28 2011 00006/
drwxr-xr-x 21 root wheel 22 May 12 2011 00007/
drwxr-xr-x 21 root wheel 22 May 12 2011 00008/
drwxr-xr-x 21 root wheel 22 May 12 2011 00009/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00010/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00011/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00012/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00013/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00014/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00015/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00016/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00017/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00018/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00019/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00020/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00021/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00022/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00023/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00024/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00025/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00026/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00027/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00028/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00029/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00030/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00031/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00032/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00033/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00034/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00035/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00036/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00037/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00038/
drwxr-xr-x 21 root wheel 22 Jan 1 1970 00039/
drwxr-xr-x 22 root wheel 23 Nov 28 13:27 00040/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00041/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00042/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00043/
drwxr-xr-x 23 root wheel 25 Dec 7 14:05 00044/
drwxr-xr-x 23 root wheel 25 Feb 15 14:18 00045/
drwxr-xr-x 23 root wheel 25 Feb 15 14:18 00046/
[root@backup01] ~#

But mounting the snapshot directly does not work...
[root@backup01] ~# mount 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot/00046 /tmp/mnt/
[tcp] 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot/00046: Permission denied
^C
[root@backup01] ~#

This is very inconsistent here...

Anybody a clue?

Thanks
Rainer
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
Ok, so by changing the /etc/exports file entry via GUI to
/mnt/pool01/rootfs64_6/.zfs/snapshot/00046 -alldirs -maproot=root:wheel

we get the following error message:
Feb 21 14:12:21 freenas mountd[6198]: -alldirs requested but /mnt/pool01/rootfs64_6/.zfs/snapshot/00046 is not a filesystem mountpoint
Feb 21 14:12:21 freenas mountd[6198]: bad exports list line /mnt/pool01/rootfs64_6/.zfs/snapshot/00046 -alldirs -maproot

So the snapshot directory containing further directories is not a mountpoint?
Strange error message...

Removing the parameters
...
/mnt/pool01/rootfs64_6/.zfs/snapshot/00046
...
gives
Feb 21 14:26:56 freenas mountd[6198]: can't export /mnt/pool01/rootfs64_6/.zfs/snapshot/00046
Feb 21 14:26:56 freenas mountd[6198]: bad exports list line /mnt/pool01/rootfs64_6/.zfs/snapshot/00046

Any idea?
anybody
 

peterh

Patron
Joined
Oct 19, 2011
Messages
315
I would have this in /etc/exports :

/mnt/pool01/rootfs64_6 -alldirs -maproot=root:wheel

then you should be able to mount anything below that firectory

what does "showmount -e < freenas>" show ?
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
Hi,
ok - I just reconfigured:
cat /etc/exports
...
/mnt/pool01/rootfs64_6 -alldirs -maproot=root:wheel
...

showmount from a client:
root@controlnode01.muc:~# showmount -e 10.20.30.17
Export list for 10.20.30.17:
...
/mnt/pool01/rootfs64_6 (everyone)
...
root@controlnode01.muc:~#

Subdir mounting from 10.20.30.17:
root@controlnode01.muc:~# mount 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot/00047 /tmp/mnt
mount.nfs: 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot/00047 failed, reason given by server: Permission denied
root@controlnode01.muc:~# mount 10.20.30.17:/mnt/pool01/rootfs64_6/.zfs/snapshot /tmp/mnt
root@controlnode01.muc:~#

So up to the snapshot-dir the submount works fine, but to mount the snapshot itself is not possible.
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
It is even worse:
Looking at a different share in the freenas and linux client:

FreeNAS:
[root@freenas] /mnt/pool01/tftproot/.zfs/snapshot# ls -al
total 71
dr-xr-xr-x 49 root wheel 49 Feb 20 16:44 ./
dr-xr-xr-x 4 root wheel 4 Feb 20 16:44 ../
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00001/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00002/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00003/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00004/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00005/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00006/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00007/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00008/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00009/
drwxr-xr-x 5 root wheel 5 May 2 2011 00010/
drwxr-xr-x 5 root wheel 5 May 2 2011 00011/
drwxr-xr-x 5 root wheel 5 May 2 2011 00012/
drwxr-xr-x 5 root wheel 5 May 2 2011 00013/
drwxr-xr-x 5 root wheel 5 May 2 2011 00014/
drwxr-xr-x 5 root wheel 5 May 2 2011 00015/
drwxr-xr-x 5 root wheel 5 May 2 2011 00016/
drwxr-xr-x 5 root wheel 5 May 2 2011 00017/
drwxr-xr-x 5 root wheel 5 May 2 2011 00018/
drwxr-xr-x 5 root wheel 5 May 2 2011 00019/
drwxr-xr-x 5 root wheel 5 May 2 2011 00020/
drwxr-xr-x 5 root wheel 5 May 2 2011 00021/
drwxr-xr-x 5 root wheel 5 May 2 2011 00022/
drwxr-xr-x 5 root wheel 5 May 2 2011 00023/
drwxr-xr-x 5 root wheel 5 May 2 2011 00024/
drwxr-xr-x 5 root wheel 5 May 2 2011 00025/
drwxr-xr-x 5 root wheel 5 May 2 2011 00026/
drwxr-xr-x 5 root wheel 5 May 2 2011 00027/
drwxr-xr-x 5 root wheel 5 May 2 2011 00028/
drwxr-xr-x 5 root wheel 5 May 2 2011 00029/
drwxr-xr-x 5 root wheel 5 May 2 2011 00030/
drwxr-xr-x 5 root wheel 5 May 2 2011 00031/
drwxr-xr-x 5 root wheel 5 May 2 2011 00032/
drwxr-xr-x 5 root wheel 5 May 2 2011 00033/
drwxr-xr-x 5 root wheel 5 May 2 2011 00034/
drwxr-xr-x 5 root wheel 5 May 2 2011 00035/
drwxr-xr-x 5 root wheel 5 May 2 2011 00036/
drwxr-xr-x 6 root wheel 6 Oct 21 14:36 00037/
drwxr-xr-x 6 root wheel 6 Oct 21 14:36 00038/
drwxr-xr-x 6 root wheel 6 Oct 21 14:36 00039/
drwxr-xr-x 6 root wheel 6 Oct 21 14:36 00040/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00041/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00042/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00043/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00044/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00045/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00046/
drwxr-xr-x 6 root wheel 6 Dec 7 08:21 00047/
[root@freenas] /mnt/pool01/tftproot/.zfs/snapshot#

root@mx03.muc:/tmp/mnt# df .
Filesystem 1K-blocks Used Available Use% Mounted on
10.20.30.17:/mnt/pool01/tftproot/.zfs/snapshot
2862878496 33760 2862844736 1% /tmp/mnt
root@mx03.muc:/tmp/mnt# ls -al
total 17
dr-xr-xr-x 49 root root 49 Feb 20 16:44 .
drwxrwxrwt 6 root root 120 Feb 23 13:17 ..
drwxr-xr-x 6 root root 6 Dec 7 08:21 00001
drwxr-xr-x 6 root root 6 Dec 7 08:21 00006
drwxr-xr-x 5 root root 5 May 2 2011 00012
drwxr-xr-x 5 root root 5 May 2 2011 00015
drwxr-xr-x 5 root root 5 May 2 2011 00018
drwxr-xr-x 5 root root 5 May 2 2011 00020
drwxr-xr-x 5 root root 5 May 2 2011 00027
drwxr-xr-x 5 root root 5 May 2 2011 00034
drwxr-xr-x 6 root root 6 Oct 21 14:36 00039
drwxr-xr-x 6 root root 6 Dec 7 08:21 00043
drwxr-xr-x 6 root root 6 Dec 7 08:21 00044
root@mx03.muc:/tmp/mnt

Looking at snapshots using NFS from a linux client seems completely broken!!!
 

peterh

Patron
Joined
Oct 19, 2011
Messages
315
Yes, it seems that the snapshot directories are "not-really-directories".
It should be nice with a comment from a zfs developer here ....
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
We installed now OpenIndiana with napp-it using ZFS version 28 - there it works as expected.

Still missing TFTP and Time machine though...
 

peterh

Patron
Joined
Oct 19, 2011
Messages
315
On freebsd the snapshot directory is a new filesystem, thats why nfs mount files in your setup
( adjusting /etc/exports to include the snapshotdirs would most likley work, but then one has
to enumerate all snapshot dirs ( as these are separate filesystems)

following program will print out filesystem id ( among other things)
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>


main(int argc,char **argv)
{
struct stat statbuf;
char *fi;
mode_t mode;
fi = argv[1];
printf ("file:%s ",fi);

lstat(fi,&statbuf);
mode = statbuf.st_mode;

printf("Mode: ");
/* printf (" %x ",statbuf.st_mode);*/
printf(" %0o ",statbuf.st_mode);

if ( S_ISREG(mode)) printf(" regular file ");
if ( S_ISDIR(mode)) printf(" directory ");
if ( S_ISCHR(mode)) printf(" character device ");
if ( S_ISBLK(mode)) printf(" block device ");
if ( S_ISFIFO(mode)) printf(" fifo ");
if ( S_ISLNK(mode)) printf(" symbolic link ");
if ( S_ISSOCK(mode)) printf(" socket ");
printf("Device: %8x \n",statbuf.st_dev);

printf ("\n");
}

where st_dev is defined as : st_dev The numeric ID of the device containing the file.

Can you compile & run on your solaris system and confirm that solaris either ;
- uses the same filesystem-id for the snapshot dirs or
- allows filesystems to inhereit exports (which breaks UNIX-std i think)
( nope, open group (http://pubs.opengroup.org/onlinepubs/) says nothing about this as far as i can see )
 

peterh

Patron
Joined
Oct 19, 2011
Messages
315
I managed to compile and run on a nexenta ( 134 ) and yes indeed, device number changes
for all of the snapshots.
exposing filesystems mounted on subdir of an exported filesystem is a nono according to NFS rules
( unless the subdir's filesystem has a valid "export" ( or sharenfs as sun calls it) rule.) and is explicitly mounted.

Strange enough i can mount a freenas 8.0.3 filesystem
exported "/mnt/fnas/arkiv -alldirs -maproot=root: -network" and i can traverse down through the snapshots"

[dhcp-0:.zfs/snapshot/auto-20120229.0900-6m] peter% pwd
/Volumes/arkiv/.zfs/snapshot/auto-20120229.0900-6m
[dhcp-0:.zfs/snapshot/auto-20120229.0900-6m] peter% ls
EyeTV backup freebsd-pxeboot peter update
Freebsd donethistoo osx-app skivarkiv

Mystery tightens :smile: Device id has changed but it's not traversing a mountpoint !
 

rstumbaum

Dabbler
Joined
Dec 20, 2011
Messages
17
As far as I understand it is not allowed to re-export a NFS-mounted Filesystem.
So mounting on server2 a Filesystem from server1 and then server2 exporting that filesystem to server3 is not allowed.

But local sub-mounts being NFS exported are fine and very well allowed, e.g. /dev/sda1 on /export, /dev/sda2 on /export/test and then NFS-exporting /export.
 
Status
Not open for further replies.
Top