I think we already have one bug report of a similar nature :
https://ixsystems.atlassian.net/browse/NAS-124776
If you can send me a debug by pm I can get this reviewed.
I'm not sure if this is related, but I had a long-standing bug with TrueNAS Angelfish and Bluefin when it came to the ix-applications dataset.
- If I turned on replication from the system volume to a backup volume,
- And the replication included the "ix-applications" dataset,
- And the backup volume mounted before, the system volume...
The machine would persistently refuse to mount all of my pools. In older versions of TrueNAS, it used to have a 15 minute timeout for mounting ZFS pools. The system would countdown the 15 minutes, and then fail the import, and move along.
At that time I had a prod pool, and a DR pool.
What I noticed was that the ZFS mount scripts for boot mounted the volumes in Alphabetical order.
So, if my prod pool was,
prodpool-1
and the DR pool was,
disaster-recovery-prodpool-1
The system would mount the ZFS pool "disaster-recovery-prodpool-1" and then hang for the 15 minutes refusing to mount the regular prodpool-1, if it contained a replicated copy of the ix-applications dataset.
I tried reinstalling with different versions of Angelfish and Bluefin, loading configs from backups, etc. It was really annoying. I figured it out, when I deleted the DR pool. Once the DR pool was gone, the system booted fine. If I re-created the pool, and ix-applications was included in the replications, bang, hung dead system. I had to do some SQL-fu on the freenas.db as when the system came up, I could not reliably remove the DR pool with the replicated copy of the ix-applications dataset. Something in the boot process wanted to grab onto any of the "ix-applications" mounts, whether it be on the system volume or a replicated backup volume.
No encryption or anything like that.
From that time, I've never trusted the ix-applications dataset or configuration on TrueNAS. I explicitly have my replication ignore ix-applications dataset, and things work ok. I'm wondering if some of the upgrade bugs and missing datasets are similar. The messages in the forum here are very similar to what I would experience. ZFS pools missing, not mounting, etc. I know some of the causes are linked to encryption and unencrypted ix-applications, but I also know that a system with two copies of ix-applications, where the non-system copy is located in a pool that mounts before production pool that has ix-applications in it, will hang up the boot process and cause no end of pain.
Hope this helps,