Hard Crash When Changing Encryption Option on Replicated Dataset

[michaelding313]

Greeting nice people of the forums,


Was moving datasets from a old server (server A) to a new replacement (server B)
Wanted to send a few select datasets from server A to server B. I used replication tasks to do so.
Everything is setup in the GUI and no command line used.
I have the same issue with multiple replicated datasets, below is a minimal example with one of them.

Datasets names replicated:
Server A: hddTank/workDir/shared -> Server B: bigSSD/workDir/shared

Both servers use a common LDAP server for user info. root and admin accounts are local.

Symptom:

After replicating the dataset to server B, and disabling read-only on the dataset in question. I get an error
[EFAULT] Fialed retrieveing GROUP quotas for bigSSD/workDir/shared

Dataset replicated on Server A, hddTank/workDir/shared

Replication Job in question

I am letting the replication task create it's own datasets, as there are multiple source datasets. Note I didn't include the encryption root.

User quotas show up fine on Server A, however once replicated, they failed to load.

I dug far enough to figure out the lack of user quotas on Server B is likely from zfs userspace bigSSD/workDir/shared failing:
on Server A (source):

the same, replicated dataset on Server B:

going one level up on Server B, quotas show no problem

Dataset versions are also the same across the two system, at 5.

The only reason I can think of is this may have something to do with the fact that the dataset hddTank/workDir/shared is not the encryption root? Does replicating a sub dataset and not its encryption root causes issues like this?
The two systems also have different TrueNAS OS versions, 22.02.4 -> 23.10.2

I have resorted to creating all new datasets on Server B, and sending files over with RSYNC jobs to avoid weirdness. I don't mind losing the snapshots as the old server will be kept around for a couple of weeks.



Server Info:

Server A (source):
OS Version:TrueNAS-SCALE-22.02.4
Product:X10DRH
Model:Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Memory:63 GiB

Disks in pool: 4x segate Exos X18 14TB, in RAIDZ1

Server B (destination):
OS Version:TrueNAS-SCALE-23.10.2
Product:AS -2115HS-TNR
Model:AMD EPYC 9454P 48-Core Processor
Memory:188 GiB

Disks in pool: 7x Kioxia KCD6XLUL15T3, in RAIDZ2



Some insight or advise would be greatly appreciated.

 

[michaelding313]

Investigated some more and the issue might be linked to the order between unlocking the dataset (these datasets are encrypted with passphrase) and setting readonly=false.

Code:

 Unlock -> set readonly=false 

does not work

Code:

 set readonly=false -> Unlock 

works

Update:
This is not reliable. Changing encryption options / unlock order fixes issues for some datasets, but causes hard crash on other datasets.
See below

 

[michaelding313]

This gets stranger. Replicated my datasets in question again with replication task, and some datasets replicate fine (with quotas working) but other datasets not.
Now there is a new found system crash


I did:
- Replicate the datasets
- New datasets are their own encryption roots on new system (I have readonly=off for all, replication task set to ignore target readonly)
- Modify the replicated dataset, and set encryption to inherit
- Do that for a few replicated datasets
- Try to change the small datasets back to their own encryption root
- System crashes immediately and decisively, no logs seem to make it out.
- I tried journelctl --follow in a root shell over SSH, no log entries before reset
- Also tried pointing syslog to another system, with "Debug" verbosity. Also nothing make it across

 

[michaelding313]

The system crashes seem to only happen if I touch encryption settings of specific broken datasets,or a parent that contains a "broken" dataset.

A broken dataset:

Code:

root@nas01[/home/admin]# zfs get all bigSSD/workDir/user
NAME                 PROPERTY                 VALUE                     SOURCE
bigSSD/workDir/user  type                     filesystem                -
bigSSD/workDir/user  creation                 Mon Mar 25 11:02 2024     -
bigSSD/workDir/user  used                     6.91T                     -
bigSSD/workDir/user  available                49.9T                     -
bigSSD/workDir/user  referenced               5.73T                     -
bigSSD/workDir/user  compressratio            1.93x                     -
bigSSD/workDir/user  mounted                  no                        -
bigSSD/workDir/user  quota                    none                      received
bigSSD/workDir/user  reservation              none                      received
bigSSD/workDir/user  recordsize               128K                      default
bigSSD/workDir/user  mountpoint               /mnt/bigSSD/workDir/user  default
bigSSD/workDir/user  sharenfs                 off                       default
bigSSD/workDir/user  checksum                 on                        default
bigSSD/workDir/user  compression              lz4                       inherited from bigSSD
bigSSD/workDir/user  atime                    off                       inherited from bigSSD
bigSSD/workDir/user  devices                  on                        default
bigSSD/workDir/user  exec                     on                        default
bigSSD/workDir/user  setuid                   on                        default
bigSSD/workDir/user  readonly                 off                       inherited from bigSSD
bigSSD/workDir/user  zoned                    off                       default
bigSSD/workDir/user  snapdir                  hidden                    local
bigSSD/workDir/user  aclmode                  discard                   local
bigSSD/workDir/user  aclinherit               discard                   local
bigSSD/workDir/user  createtxg                1077143                   -
bigSSD/workDir/user  canmount                 on                        default
bigSSD/workDir/user  xattr                    sa                        received
bigSSD/workDir/user  copies                   1                         local
bigSSD/workDir/user  version                  5                         -
bigSSD/workDir/user  utf8only                 off                       -
bigSSD/workDir/user  normalization            none                      -
bigSSD/workDir/user  casesensitivity          sensitive                 -
bigSSD/workDir/user  vscan                    off                       default
bigSSD/workDir/user  nbmand                   off                       default
bigSSD/workDir/user  sharesmb                 off                       default
bigSSD/workDir/user  refquota                 none                      received
bigSSD/workDir/user  refreservation           none                      received
bigSSD/workDir/user  guid                     12020725785550621332      -
bigSSD/workDir/user  primarycache             all                       default
bigSSD/workDir/user  secondarycache           all                       default
bigSSD/workDir/user  usedbysnapshots          1.18T                     -
bigSSD/workDir/user  usedbydataset            5.73T                     -
bigSSD/workDir/user  usedbychildren           0B                        -
bigSSD/workDir/user  usedbyrefreservation     0B                        -
bigSSD/workDir/user  logbias                  latency                   default
bigSSD/workDir/user  objsetid                 4615                      -
bigSSD/workDir/user  dedup                    off                       default
bigSSD/workDir/user  mlslabel                 none                      default
bigSSD/workDir/user  sync                     standard                  default
bigSSD/workDir/user  dnodesize                legacy                    default
bigSSD/workDir/user  refcompressratio         1.92x                     -
bigSSD/workDir/user  written                  0                         -
bigSSD/workDir/user  logicalused              8.33T                     -
bigSSD/workDir/user  logicalreferenced        6.96T                     -
bigSSD/workDir/user  volmode                  default                   default
bigSSD/workDir/user  filesystem_limit         none                      default
bigSSD/workDir/user  snapshot_limit           none                      default
bigSSD/workDir/user  filesystem_count         none                      default
bigSSD/workDir/user  snapshot_count           none                      default
bigSSD/workDir/user  snapdev                  hidden                    default
bigSSD/workDir/user  acltype                  posix                     local
bigSSD/workDir/user  context                  none                      default
bigSSD/workDir/user  fscontext                none                      default
bigSSD/workDir/user  defcontext               none                      default
bigSSD/workDir/user  rootcontext              none                      default
bigSSD/workDir/user  relatime                 on                        default
bigSSD/workDir/user  redundant_metadata       all                       default
bigSSD/workDir/user  overlay                  on                        default
bigSSD/workDir/user  encryption               aes-256-gcm               -
bigSSD/workDir/user  keylocation              none                      default
bigSSD/workDir/user  keyformat                passphrase                -
bigSSD/workDir/user  pbkdf2iters              350000                    -
bigSSD/workDir/user  encryptionroot           bigSSD/workDir            -
bigSSD/workDir/user  keystatus                available                 -
bigSSD/workDir/user  special_small_blocks     0                         default
bigSSD/workDir/user  snapshots_changed        Tue Mar 26 14:12:26 2024  -
bigSSD/workDir/user  org.truenas:managedby    172.16.43.3               received
bigSSD/workDir/user  org.freenas:description                            local

A not broken dataset

Code:

root@nas01[/home/admin]# zfs get all bigSSD/workDir/services/www_data
NAME                              PROPERTY                 VALUE                                  SOURCE
bigSSD/workDir/services/www_data  type                     filesystem                             -
bigSSD/workDir/services/www_data  creation                 Mon Mar 25  6:58 2024                  -
bigSSD/workDir/services/www_data  used                     28.0G                                  -
bigSSD/workDir/services/www_data  available                49.9T                                  -
bigSSD/workDir/services/www_data  referenced               27.9G                                  -
bigSSD/workDir/services/www_data  compressratio            1.08x                                  -
bigSSD/workDir/services/www_data  mounted                  yes                                    -
bigSSD/workDir/services/www_data  quota                    none                                   received
bigSSD/workDir/services/www_data  reservation              none                                   received
bigSSD/workDir/services/www_data  recordsize               128K                                   default
bigSSD/workDir/services/www_data  mountpoint               /mnt/bigSSD/workDir/services/www_data  default
bigSSD/workDir/services/www_data  sharenfs                 off                                    default
bigSSD/workDir/services/www_data  checksum                 on                                     default
bigSSD/workDir/services/www_data  compression              zstd                                   inherited from bigSSD/workDir/services
bigSSD/workDir/services/www_data  atime                    off                                    inherited from bigSSD
bigSSD/workDir/services/www_data  devices                  on                                     default
bigSSD/workDir/services/www_data  exec                     on                                     default
bigSSD/workDir/services/www_data  setuid                   on                                     default
bigSSD/workDir/services/www_data  readonly                 off                                    inherited from bigSSD
bigSSD/workDir/services/www_data  zoned                    off                                    default
bigSSD/workDir/services/www_data  snapdir                  hidden                                 local
bigSSD/workDir/services/www_data  aclmode                  discard                                inherited from bigSSD
bigSSD/workDir/services/www_data  aclinherit               discard                                inherited from bigSSD/workDir
bigSSD/workDir/services/www_data  createtxg                1073433                                -
bigSSD/workDir/services/www_data  canmount                 on                                     default
bigSSD/workDir/services/www_data  xattr                    sa                                     received
bigSSD/workDir/services/www_data  copies                   1                                      local
bigSSD/workDir/services/www_data  version                  5                                      -
bigSSD/workDir/services/www_data  utf8only                 off                                    -
bigSSD/workDir/services/www_data  normalization            none                                   -
bigSSD/workDir/services/www_data  casesensitivity          sensitive                              -
bigSSD/workDir/services/www_data  vscan                    off                                    default
bigSSD/workDir/services/www_data  nbmand                   off                                    default
bigSSD/workDir/services/www_data  sharesmb                 off                                    default
bigSSD/workDir/services/www_data  refquota                 none                                   received
bigSSD/workDir/services/www_data  refreservation           none                                   received
bigSSD/workDir/services/www_data  guid                     565632788266028773                     -
bigSSD/workDir/services/www_data  primarycache             all                                    default
bigSSD/workDir/services/www_data  secondarycache           all                                    default
bigSSD/workDir/services/www_data  usedbysnapshots          38.7M                                  -
bigSSD/workDir/services/www_data  usedbydataset            27.9G                                  -
bigSSD/workDir/services/www_data  usedbychildren           0B                                     -
bigSSD/workDir/services/www_data  usedbyrefreservation     0B                                     -
bigSSD/workDir/services/www_data  logbias                  latency                                default
bigSSD/workDir/services/www_data  objsetid                 54290                                  -
bigSSD/workDir/services/www_data  dedup                    off                                    default
bigSSD/workDir/services/www_data  mlslabel                 none                                   default
bigSSD/workDir/services/www_data  sync                     standard                               default
bigSSD/workDir/services/www_data  dnodesize                legacy                                 default
bigSSD/workDir/services/www_data  refcompressratio         1.08x                                  -
bigSSD/workDir/services/www_data  written                  21.3M                                  -
bigSSD/workDir/services/www_data  logicalused              30.0G                                  -
bigSSD/workDir/services/www_data  logicalreferenced        30.0G                                  -
bigSSD/workDir/services/www_data  volmode                  default                                default
bigSSD/workDir/services/www_data  filesystem_limit         none                                   default
bigSSD/workDir/services/www_data  snapshot_limit           none                                   default
bigSSD/workDir/services/www_data  filesystem_count         none                                   default
bigSSD/workDir/services/www_data  snapshot_count           none                                   default
bigSSD/workDir/services/www_data  snapdev                  hidden                                 default
bigSSD/workDir/services/www_data  acltype                  posix                                  inherited from bigSSD/workDir
bigSSD/workDir/services/www_data  context                  none                                   default
bigSSD/workDir/services/www_data  fscontext                none                                   default
bigSSD/workDir/services/www_data  defcontext               none                                   default
bigSSD/workDir/services/www_data  rootcontext              none                                   default
bigSSD/workDir/services/www_data  relatime                 on                                     default
bigSSD/workDir/services/www_data  redundant_metadata       all                                    default
bigSSD/workDir/services/www_data  overlay                  on                                     default
bigSSD/workDir/services/www_data  encryption               aes-256-gcm                            -
bigSSD/workDir/services/www_data  keylocation              none                                   default
bigSSD/workDir/services/www_data  keyformat                passphrase                             -
bigSSD/workDir/services/www_data  pbkdf2iters              350000                                 -
bigSSD/workDir/services/www_data  encryptionroot           bigSSD/workDir                         -
bigSSD/workDir/services/www_data  keystatus                available                              -
bigSSD/workDir/services/www_data  special_small_blocks     0                                      default
bigSSD/workDir/services/www_data  snapshots_changed        Mon Mar 25  7:01:25 2024               -
bigSSD/workDir/services/www_data  org.freenas:description                                         local
bigSSD/workDir/services/www_data  org.truenas:managedby    172.16.43.4                            received