can't mount "unlocked by ancestor" dataset

[winnielinnie]

Hold on. Did your original pool also use a "special" vdev and/or metadata vdev?

 

[valhalla]

no, it was a single vdev, no ZLOG or Caching or whatsoever. just Nested Datasets and (on the main) two ZVOLs for iSCSI. But those were never replicated to the backup system (one a Steam game drive the other a VM one, both iSCSI).

lets see, this is now "stuck" for a few min

 

[winnielinnie]

no, it was a single vdev, no ZLOG or Caching or whatsoever. just Nested Datasets and (on the main) two ZVOLs for iSCSI.

If it's not due to using any special vdevs on the original pool, and you can't even create a new user key, I'm running out of ideas. This is not looking good.

I would run a full scrub on your Backup pool.

This might require the brain of someone who is more savvied to the underlying ZFS mechanics.

 

[winnielinnie]

It seems that I won the lottery ticket this weekend, as my primary NAS system failed due to power outage and two disks in a raid-z2 died.

If it's RAIDZ2, then losing 2 drives means you can still resilver the missing two drives with new ones. Losing 2 drives in a RAIDZ2 vdev won't fail the pool.

 

[valhalla]

that was a typo, raidz1. I recreated a raidz2 now because of this on the main system.

 

[winnielinnie]

Can you view recent logs messages in another terminal while the "change-key" operation is still hanging?

 

[valhalla]

Its still hanging yes, I am looking at the load, which is at 3.08. So something is going on.

also when I ran a sudo zpool status it was hanging.

 

[valhalla]

I have also indeed the feeling that this is going to be a pretty much game over scenario, as I don't really believe in miracles.

 

[winnielinnie]

I have also indeed the feeling that this is going to be a pretty much game over scenario, as I don't really believe in miracles.

I have to run now, but I would try to ping someone who is more versed in low-level ZFS.

Perhaps even filing a bug report with a debug?

@HoneyBadger any ideas?

I was going to ping you earlier, when I suspected this was a case of special vdevs. (But it's not. No such special vdevs exist.)

Regardless of properly loading the master key (i.e, "unlocking" the dataset), he can neither mount it nor even change the user key.

If he tries to change the user key in TrueNAS SCALE via the GUI, it crashes the system.

If he tries to do it via the command-line in Ubuntu, it hangs indefinitely.

 

[valhalla]

as FYI, I rebooted the system, imported the pool again with the old key (works) and let it scrub (mouting of course still doesn't work).

 

[valhalla]

WinnieLinnie, whatever the outcome is, I truly appreciate your support on this.

 

[valhalla]

scrub was completed without errors:

 

[winnielinnie]

scrub was completed without errors.

I'm seriously out of ideas.

Either I'm missing an obviously simple thing, or this requires low-level ZFS diagnosis.

Summary for anyone else stumbling upon this so far:

• No issues with hardware or drives
• No errors from a full scrub
• Key can load/unload properly
• Cannot mount an unlocked ("key loaded") dataset
• Cannot change a key for the unlocked dataset
• Cannot even get around this on an Ubuntu system

 

[valhalla]

Little update. I have successfully moved the mountpoint of one of the Datasets outside the mnt/Backup folder and mounted it successfully (/mnt/mnt/test). I changed it with the zfs set mountpoint command and then manually mounted the barbora dataset and I have now again access to that. Same trick with synthing (which should be syncthing but that aside). Obviously I tried the same trick with the valhalla one but I have no permissions:

Code:

root@fremen[/mnt]# zfs set mountpoint=/test Backup/valhalla
cannot mount 'Backup/valhalla': Permission denied
property may be set but unable to remount filesystem

Code:

NAME                                                         USED  AVAIL  REFER  MOUNTPOINT
Backup                                                      3.66T  3.48T   307K  /mnt/Backup
Backup/barbora                                              5.69G  3.48T  5.68G  /mnt/mnt/test
Backup/synthing                                             40.7G  3.48T  40.7G  /mnt/syncthing
Backup/valhalla                                             2.96T  3.48T  2.80T  /mnt/test
Backup/vmbkp                                                 663G  3.48T   264G  /mnt/Backup/vmbkp

EDIT: This tells me the relationship between the valhalla dataset being inherited by Backup seems to be now 99.999% the culprit.

Double EDIT: So to summarize:

1) vmbkp is the only dataset that mounts properly, after applying the MAIN system key into /mnt/Backup/vmbkp
2) barbora, synthing refused to mount INSIDE the Backup pool folder (mnt/Backup/barbora and synthing did not get created and got permissions error)
3) moved the mointpoints of both barbora and synthing OUTSIDE the Backup pool folder and mount properly and can access the data.
4) any new directory you want to create (mkdir) within the /mnt/Backup folder gives a permission denied as ROOT:

Code:

root@fremen[/mnt/Backup]# mkdir test
mkdir: cannot create directory ‘test’: Operation not permitted

This leads me to believe the above statement that Backup pool in particular and valhalla dataset got somehow messed up and may indeed require some deeper ZFS diagnostics to get to the bottom of this.

 

[winnielinnie]

Only mounting "Backup" (without mounting any of the children), what folders exist under /mnt/Backup?

If so, are these folders empty?

What are the permissions for /mnt/Backup after it's mounted?

 

[valhalla]

the permissions is root:

Code:

root@fremen[/mnt]# ls -l
total 28
drwxr-xr-x 4 root    root 4 Nov 27 12:43 Backup

root@fremen[/mnt/Backup]# ls -l
total 27
drwxr-xr-x 2 root root 2 Dec  4 10:12 valhalla
drwxr-xr-x 2 root root 2 Nov 27 12:43 vmbkp

those 2 folders only exisit under /mnt/backup But if only the Backup one is mounted (meaning I have not unlocked the vmbkp one for example because that would then mount it as well) the folders are both empty:

Code:

root@fremen[/mnt/Backup/valhalla]# ls -l

total 0


root@fremen[/mnt/Backup/vmbkp]# ls -ls

total 0

root@fremen[/mnt/Backup/vmbkp]#

It does not make tons of sense if the /mnt/Backup has root:root permissions and you can't create either a directory or touch a file into it.

Code:

root@fremen[/mnt/Backup]# touch  test
touch: cannot touch 'test': Operation not permitted
root@fremen[/mnt/Backup]# mkdir test
mkdir: cannot create directory ‘test’: Operation not permitted
root@fremen[/mnt/Backup]#

 

[valhalla]

btw, I cannot unmount Backup because it will complain that it is busy

root@fremen[/mnt/Backup]# zfs unmount Backup
cannot unmount '/mnt/Backup': pool or dataset is busy

 

[winnielinnie]

those 2 folders only exisit under /mnt/backup

There should be no folders within /mnt/Backup

Can you safely remove them with "rmdir"? (Only with Backup mounted; without any children mounted.)

Do not use "rm". Only use "rmdir", since it will only work on empty directories.

btw, I cannot unmount Backup because it will complain that it is busy

Because you're trying to do it while still in the mount's path.

 

[valhalla]

haha DOH, you are right. So when unmount no directories. When I mount it has both valhalla and vmbkp but they are not removable:

Code:

root@fremen[/mnt/Backup]# rmdir valhalla
rmdir: failed to remove 'valhalla': Operation not permitted
root@fremen[/mnt/Backup]# rmdir vmbkp
rmdir: failed to remove 'vmbkp': Operation not permitted

 

[valhalla]

That is also the reason the others were not mounting because you got the same permission denied error. That is why I could narrow it down to the pool itself.