Schrödinger's Datasets (Mounted & Not, at the Same Time)

[yottabit]

I've imported old pools that were running on CORE 12.0-U2 into a fresh SCALE 21.08-BETA.1 install.

The older pools (many years) imported fine, and I have verified everything is mounted and accessible, among many datasets and zvols.

The newer pool (1-2 years) imported fine, but only 1 of the 3 datasets are mounted, even though zfs says all 3 are mounted. The GUI shows all 3 datasets too.

Code:

[jacob.mcdonald@nas1 ~]$ sudo zpool status ssd_scratch
  pool: ssd_scratch
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:04:24 with 0 errors on Sat Sep 11 17:06:20 2021
config:

        NAME                                    STATE     READ WRITE CKSUM
        ssd_scratch                             ONLINE       0     0     0
          7410e22c-3e87-11eb-8620-0cc47ab48630  ONLINE       0     0     0
          74ac731c-3e87-11eb-8620-0cc47ab48630  ONLINE       0     0     0

errors: No known data errors

[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/incoming_media
NAME                        PROPERTY    VALUE                            SOURCE
ssd_scratch/incoming_media  mountpoint  /mnt/ssd_scratch/incoming_media  default

[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/peg_takeout_archives
NAME                              PROPERTY    VALUE                                  SOURCE
ssd_scratch/peg_takeout_archives  mountpoint  /mnt/ssd_scratch/peg_takeout_archives  default

# This is the only dataset that has a mountpoint on the filesystem.
[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/plex_transcode
NAME                        PROPERTY    VALUE                            SOURCE
ssd_scratch/plex_transcode  mountpoint  /mnt/ssd_scratch/plex_transcode  default

[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/incoming_media
NAME                        PROPERTY    VALUE                            SOURCE
ssd_scratch/incoming_media  mountpoint  /mnt/ssd_scratch/incoming_media  default

[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/peg_takeout_archives
NAME                              PROPERTY    VALUE                                  SOURCE
ssd_scratch/peg_takeout_archives  mountpoint  /mnt/ssd_scratch/peg_takeout_archives  default

[jacob.mcdonald@nas1 ~]$ sudo zfs get mountpoint ssd_scratch/plex_transcode
NAME                        PROPERTY    VALUE                            SOURCE
ssd_scratch/plex_transcode  mountpoint  /mnt/ssd_scratch/plex_transcode  default

[jacob.mcdonald@nas1 ~]$ mount | grep ssd_scratch
ssd_scratch on /mnt/ssd_scratch type zfs (rw,xattr,nfs4acl)
ssd_scratch/peg_takeout_archives on /mnt/ssd_scratch/peg_takeout_archives type zfs (rw,xattr,nfs4acl)
ssd_scratch/plex_transcode on /mnt/ssd_scratch/plex_transcode type zfs (rw,xattr,nfs4acl)
ssd_scratch/incoming_media on /mnt/ssd_scratch/incoming_media type zfs (rw,xattr,nfs4acl)

[jacob.mcdonald@nas1 ~]$ sudo ls -al /mnt/ssd_scratch
total 16
drwxr-xr-x  3 root           root            3 Sep 11 17:31 .
drwxr-xr-x 22 root           root           28 Sep 11 18:52 ..
drwxr-xr-x  3 jacob.mcdonald jacob.mcdonald  3 Sep 11 17:00 plex_transcode

Any ideas what to do next here?

 

[sretalla]

try zfs mount -a

 

[yottabit]

Code:

[jacob.mcdonald@nas1 ~]$ sudo zfs mount -a
[jacob.mcdonald@nas1 ~]$ ls -a /mnt/ssd_scratch/
.  ..  plex_transcode

Nope, no change. And as I wrote in the first post, ZFS already thinks it's mounted, and so does fstab. :-/

 

[yottabit]

Still no luck here. I can zfs send | zfs recv the dataset, but I cannot destroy the original dataset because the mountpoint cannot be unmounted (because it isn't mounted, even though fstab and zfs think it is); using -f doesn't help.

I think I'm going to try booting a live image distro and seeing if I can remove the problematic datasets that way.

I realize this is primarily a user forum, but often ixSystems is active here. Should I wait for them to see this? Or should I create a ticket? I am happy to preserve state on this problematic pool for a while for investigation since I don't urgently need the data on this particular pool.

P.S. I cannot export this pool either, even though I have no shares or config depending on it, and even all swap is off. zpool complains that it cannot unmount one of the datasets, just like destroy above. -f has no effect.

P.P.S. No open files, either, except stat is hanging up on the dirty mountpoints:

Code:

nas1# lsof | grep ssd_scratch
lsof: WARNING: can't stat() zfs file system /mnt/ssd_scratch/peg_takeout_archives
      Output information may be incomplete.
lsof: WARNING: can't stat() zfs file system /mnt/ssd_scratch/incoming_media
      Output information may be incomplete.

 

[yottabit]

I need to reclaim these datasets eventually... I don't have enough room to keep a duplicate on-pool via the zfs send | zfs recv method, so I will need to boot a live image from another OS and see if I can delete the problematic datasets...

But before I do this, does anyone from ix want to investigate? Paging Kris? I supposed I should file a ticket: 00011820.

 

[yottabit]

Okay, now with a Jira ticket! https://jira.ixsystems.com/browse/NAS-112348

 

[yottabit]

Full updates are in the bug linked in #6, but it looks like I finally found the root cause:

I booted into singleuser mode, imported the pool, destroyed the pool, wiped the first few MBs of each disk, then rebooted into normal multiuser mode. The pool showed an error in the UI and I was finally able to successfully export/disconnect it.

Afterward I looked at the filesystem and noticed that the mountpoint had files I had created before, assuming they were being created on the pool, but in fact were just in the mountpoint on what I thought was the rootfs. These files were persistent across reboots, which surprised me.

Looking at the output of mount, I didn't see any /mnt mountpoint, so these files must have been written to the rootfs, right?

Code:

$ sudo findmnt -n -o SOURCE /
boot-pool/ROOT/21.08-BETA.1

$ sudo findmnt -n -o SOURCE /mnt
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m

$ sudo zfs list vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m
NAME USED AVAIL REFER MOUNTPOINT
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m 23.4G 8.57T 3.66G /mnt

Oh no. Check that out. I use zfs send from an Ubuntu workstation to backup its filesystems to the NAS. It looks like one of those datasets is clobbering the /mnt mountpoint! This is not the behavior I noticed on TrueNAS CORE with these same pools.

This does explain a lot of the slowness I was experiencing when using the "new" datasets I created on what was supposed to be a striped SSD pool. I was writing to my HDD pool instead.

So the mystery remains. Why is this behavior different on Linux/SCALE than on BSD/CORE? At least now I know how to fix it: remote the mountpoint from the datasets sent from Ubuntu. But how can I avoid this in the future? Is there any way to not auto-mount specific datasets (including children)? For now I will set the mountpoint for those datasets to legacy, but I'm not sure how that will impact a restore if I ever need to do so.

 

[sretalla]

Is there any way to not auto-mount specific datasets (including children)?

Set the property canmount to noauto:
zfs set canmount=noauto pool/dataset

Maybe there's a recursive option for child datasets... maybe that's all inheriting anyway... I haven't checked/tested that. If not, you'll need to explicitly do it for all the children too.

 

[yottabit]

Thanks, was pursuing setting mountpoint=none, but canmount=noauto is much cleaner. I suspect this does prevent me from doing an easy restore if I needed to though, without setting that canmount=auto after the restore, but at least I won't have to maintain any state on the mountpoints.

This is a mess:

Code:

$ sudo zfs list -r vol1/remote_backups/nuc5_ubuntu
NAME                                                                         USED  AVAIL     REFER  MOUNTPOINT
vol1/remote_backups/nuc5_ubuntu                                             31.3G  8.57T      153K  /mnt/vol1/remote_backups/nuc5_ubuntu
vol1/remote_backups/nuc5_ubuntu/ROOT                                        23.4G  8.57T      153K  /mnt/vol1/remote_backups/nuc5_ubuntu/ROOT
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m                          23.4G  8.57T     3.66G  /mnt
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/srv                       332K  8.57T      153K  /mnt/srv
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/usr                       920K  8.57T      153K  /mnt/usr
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/usr/local                 767K  8.57T      204K  /mnt/usr/local
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var                      18.4G  8.57T      153K  /mnt/var
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/games                 153K  8.57T      153K  /mnt/var/games
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib                  18.3G  8.57T     1.25G  /mnt/var/lib
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/AccountsService   677K  8.57T      179K  /mnt/var/lib/AccountsService
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/NetworkManager   1.67M  8.57T      243K  /mnt/var/lib/NetworkManager
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/apt              92.1M  8.57T     83.0M  /mnt/var/lib/apt
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/dpkg             93.8M  8.57T     50.8M  /mnt/var/lib/dpkg
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/libvirt          16.8G  8.57T      249K  /mnt/var/lib/libvirt
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/lib/libvirt/images   16.8G  8.57T     11.0G  /mnt/var/lib/libvirt/images
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/log                  89.9M  8.57T     48.1M  /mnt/var/log
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/mail                  153K  8.57T      153K  /mnt/var/mail
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/snap                 1.11M  8.57T      856K  /mnt/var/snap
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/spool                 447K  8.57T      179K  /mnt/var/spool
vol1/remote_backups/nuc5_ubuntu/ROOT/ubuntu_hrkc9m/var/www                   153K  8.57T      153K  /mnt/var/www
vol1/remote_backups/nuc5_ubuntu/USERDATA                                    7.82G  8.57T      153K  /mnt/vol1/remote_backups/nuc5_ubuntu/USERDATA
vol1/remote_backups/nuc5_ubuntu/USERDATA/root_kopi51                        1.17M  8.57T     1.17M  /mnt/vol1/remote_backups/nuc5_ubuntu/USERDATA/root_kopi51
vol1/remote_backups/nuc5_ubuntu/USERDATA/yottabit_kopi51                    7.81G  8.57T     7.81G  /mnt/vol1/remote_backups/nuc5_ubuntu/USERDATA/yottabit_kopi51

I don't know why this didn't interfere with my other zpool mounted on /mnt/vol1, and I don't know why this behavior is different in Linux/SCALE v. BSD/CORE.

Now that I have a workaround, I'm unsure how I should proceed backing up my zfs datasets from the workstation without clobbering the TrueNAS SCALE filesystem mounts.