SOLVED Can not unlock pool after upgrade to 22.02.0.1 "directory is not empty" / Some Datasets mount empty with workaround

[ChaosBlades]

'/mnt/tank' directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)

Looks like even with the pool locked I have folders named as my dataset structure on 3 of my pools experiencing this issue. On 3 other pools I do not have any directories when the pool is locked and I do not have this issue.

I can not manually delete the directories with rm -R * as root 'permission denied'.

How do I use the "force" command? I don't see any zpool unlock command and searching around for what the command is not providing any help.

Looks like this was an issue with RC concerning docker creating some directories but my docker pool unlocks just fine.

Edit:
This is passphrase not key encryption btw.
OK, So....

zfs load-key -a
zfs mount -a

works without any errors.
Thoughts?

 

[morganL]

Thoughts?

Glad you seem t have fixed this. How did you "solve"?
The community appreciates the answers so that someone else can handle a similar situation.

 

[ChaosBlades]

Glad you seem t have fixed this. How did you "solve"?
The community appreciates the answers so that someone else can handle a similar situation.

By running those two commands. So I guess it would be more of a workaround. I'll remove the Solved tag to avoid confusion.

 

[morganL]

By running those two commands. So I guess it would be more of a workaround. I'll remove the Solved tag to avoid confusion.

My misunderstanding... I thought you were asking whether those commands should work?

I would agree it is "solved"..... as long as it only happens with pre-release software updates. I'll change back.

If it happens with post-release updates, then it should be relisted as a bug.

 

[ChaosBlades]

Things have gotten stranger...

1 Dataset on 2 of the 3 Pools I am having this issue with mounted empty. Rolling back a Snapshot succeeded but still resulted in a blank directory. Cloning the Dataset from the same Snapshot provided access to the files.

The problem Dataset can not be unmounted by any means (WebUI or Shell), nor can it be deleted because it needs to be unmounted in order to do that. I ended up needing to reboot the system so everything was locked. Delete the Dataset while the the Pool was still locked. Recreate the Dataset then restore the files.

This solved that problem but I still can not unlock these 3 Pools via the WebUI. I have to use the two commands in OP.

I am thinking the WebUI middleware is doing extra steps other than the two commands I posed in OP to unlock the Dataset that are bugging out.

Edit: This is TrueNAS-SCALE-22.02.0.1 the latest Stable version that released a few days ago. This is not Pre-Release.
Edit2: Once I get all this data moved around and something at least working again I'll post to Jira with a debug file.

 

[morganL]

It would be worth documenting the histories of the impacted pools... how/when were they created and what was the update path.
I don't see evidence of others having the issue.. so is there anything relatively unique?

 

[ChaosBlades]

I have solved the issue... for real this time.

I replaced both my boot SSDs (just in case I needed to roll back) and reinstalled TrueNAS SCALE fresh then uploaded my saved config file. This install was originally from the inital FreeNAS 11 release.

Nuke the entire site fom orbit

It's the only way to be sure.Clip from Aliens (1986)

youtu.be