Hard Crash When Changing Encryption Option on Replicated Dataset

Mar 22, 2024
Greeting nice people of the forums,

Was moving datasets from a old server (server A) to a new replacement (server B)
Wanted to send a few select datasets from server A to server B. I used replication tasks to do so.
Everything is setup in the GUI and no command line used.
I have the same issue with multiple replicated datasets, below is a minimal example with one of them.

Datasets names replicated:
Server A: hddTank/workDir/shared -> Server B: bigSSD/workDir/shared

Both servers use a common LDAP server for user info. root and admin accounts are local.


After replicating the dataset to server B, and disabling read-only on the dataset in question. I get an error
[EFAULT] Fialed retrieveing GROUP quotas for bigSSD/workDir/shared


Dataset replicated on Server A, hddTank/workDir/shared


Replication Job in question

I am letting the replication task create it's own datasets, as there are multiple source datasets. Note I didn't include the encryption root.

User quotas show up fine on Server A, however once replicated, they failed to load.

I dug far enough to figure out the lack of user quotas on Server B is likely from zfs userspace bigSSD/workDir/shared failing:
on Server A (source):

the same, replicated dataset on Server B:

going one level up on Server B, quotas show no problem

Dataset versions are also the same across the two system, at 5.

The only reason I can think of is this may have something to do with the fact that the dataset hddTank/workDir/shared is not the encryption root? Does replicating a sub dataset and not its encryption root causes issues like this?
The two systems also have different TrueNAS OS versions, 22.02.4 -> 23.10.2

I have resorted to creating all new datasets on Server B, and sending files over with RSYNC jobs to avoid weirdness. I don't mind losing the snapshots as the old server will be kept around for a couple of weeks.

Server Info:

Server A (source):

OS Version:TrueNAS-SCALE-22.02.4
Model:Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Memory:63 GiB

Disks in pool: 4x segate Exos X18 14TB, in RAIDZ1

Server B (destination):
OS Version:TrueNAS-SCALE-23.10.2
Product:AS -2115HS-TNR
Model:AMD EPYC 9454P 48-Core Processor
Memory:188 GiB

Disks in pool: 7x Kioxia KCD6XLUL15T3, in RAIDZ2

Some insight or advise would be greatly appreciated.
Mar 22, 2024
Investigated some more and the issue might be linked to the order between unlocking the dataset (these datasets are encrypted with passphrase) and setting readonly=false.

 Unlock -> set readonly=false 
does not work

 set readonly=false -> Unlock 

This is not reliable. Changing encryption options / unlock order fixes issues for some datasets, but causes hard crash on other datasets.
See below
Last edited:
Mar 22, 2024
This gets stranger. Replicated my datasets in question again with replication task, and some datasets replicate fine (with quotas working) but other datasets not.
Now there is a new found system crash

I did:
- Replicate the datasets
- New datasets are their own encryption roots on new system (I have readonly=off for all, replication task set to ignore target readonly)
- Modify the replicated dataset, and set encryption to inherit
- Do that for a few replicated datasets
- Try to change the small datasets back to their own encryption root
- System crashes immediately and decisively, no logs seem to make it out.
- I tried journelctl --follow in a root shell over SSH, no log entries before reset
- Also tried pointing syslog to another system, with "Debug" verbosity. Also nothing make it across
Mar 22, 2024
The system crashes seem to only happen if I touch encryption settings of specific broken datasets,or a parent that contains a "broken" dataset.

A broken dataset:

root@nas01[/home/admin]# zfs get all bigSSD/workDir/user
NAME                 PROPERTY                 VALUE                     SOURCE
bigSSD/workDir/user  type                     filesystem                -
bigSSD/workDir/user  creation                 Mon Mar 25 11:02 2024     -
bigSSD/workDir/user  used                     6.91T                     -
bigSSD/workDir/user  available                49.9T                     -
bigSSD/workDir/user  referenced               5.73T                     -
bigSSD/workDir/user  compressratio            1.93x                     -
bigSSD/workDir/user  mounted                  no                        -
bigSSD/workDir/user  quota                    none                      received
bigSSD/workDir/user  reservation              none                      received
bigSSD/workDir/user  recordsize               128K                      default
bigSSD/workDir/user  mountpoint               /mnt/bigSSD/workDir/user  default
bigSSD/workDir/user  sharenfs                 off                       default
bigSSD/workDir/user  checksum                 on                        default
bigSSD/workDir/user  compression              lz4                       inherited from bigSSD
bigSSD/workDir/user  atime                    off                       inherited from bigSSD
bigSSD/workDir/user  devices                  on                        default
bigSSD/workDir/user  exec                     on                        default
bigSSD/workDir/user  setuid                   on                        default
bigSSD/workDir/user  readonly                 off                       inherited from bigSSD
bigSSD/workDir/user  zoned                    off                       default
bigSSD/workDir/user  snapdir                  hidden                    local
bigSSD/workDir/user  aclmode                  discard                   local
bigSSD/workDir/user  aclinherit               discard                   local
bigSSD/workDir/user  createtxg                1077143                   -
bigSSD/workDir/user  canmount                 on                        default
bigSSD/workDir/user  xattr                    sa                        received
bigSSD/workDir/user  copies                   1                         local
bigSSD/workDir/user  version                  5                         -
bigSSD/workDir/user  utf8only                 off                       -
bigSSD/workDir/user  normalization            none                      -
bigSSD/workDir/user  casesensitivity          sensitive                 -
bigSSD/workDir/user  vscan                    off                       default
bigSSD/workDir/user  nbmand                   off                       default
bigSSD/workDir/user  sharesmb                 off                       default
bigSSD/workDir/user  refquota                 none                      received
bigSSD/workDir/user  refreservation           none                      received
bigSSD/workDir/user  guid                     12020725785550621332      -
bigSSD/workDir/user  primarycache             all                       default
bigSSD/workDir/user  secondarycache           all                       default
bigSSD/workDir/user  usedbysnapshots          1.18T                     -
bigSSD/workDir/user  usedbydataset            5.73T                     -
bigSSD/workDir/user  usedbychildren           0B                        -
bigSSD/workDir/user  usedbyrefreservation     0B                        -
bigSSD/workDir/user  logbias                  latency                   default
bigSSD/workDir/user  objsetid                 4615                      -
bigSSD/workDir/user  dedup                    off                       default
bigSSD/workDir/user  mlslabel                 none                      default
bigSSD/workDir/user  sync                     standard                  default
bigSSD/workDir/user  dnodesize                legacy                    default
bigSSD/workDir/user  refcompressratio         1.92x                     -
bigSSD/workDir/user  written                  0                         -
bigSSD/workDir/user  logicalused              8.33T                     -
bigSSD/workDir/user  logicalreferenced        6.96T                     -
bigSSD/workDir/user  volmode                  default                   default
bigSSD/workDir/user  filesystem_limit         none                      default
bigSSD/workDir/user  snapshot_limit           none                      default
bigSSD/workDir/user  filesystem_count         none                      default
bigSSD/workDir/user  snapshot_count           none                      default
bigSSD/workDir/user  snapdev                  hidden                    default
bigSSD/workDir/user  acltype                  posix                     local
bigSSD/workDir/user  context                  none                      default
bigSSD/workDir/user  fscontext                none                      default
bigSSD/workDir/user  defcontext               none                      default
bigSSD/workDir/user  rootcontext              none                      default
bigSSD/workDir/user  relatime                 on                        default
bigSSD/workDir/user  redundant_metadata       all                       default
bigSSD/workDir/user  overlay                  on                        default
bigSSD/workDir/user  encryption               aes-256-gcm               -
bigSSD/workDir/user  keylocation              none                      default
bigSSD/workDir/user  keyformat                passphrase                -
bigSSD/workDir/user  pbkdf2iters              350000                    -
bigSSD/workDir/user  encryptionroot           bigSSD/workDir            -
bigSSD/workDir/user  keystatus                available                 -
bigSSD/workDir/user  special_small_blocks     0                         default
bigSSD/workDir/user  snapshots_changed        Tue Mar 26 14:12:26 2024  -
bigSSD/workDir/user  org.truenas:managedby               received
bigSSD/workDir/user  org.freenas:description                            local

A not broken dataset
root@nas01[/home/admin]# zfs get all bigSSD/workDir/services/www_data
NAME                              PROPERTY                 VALUE                                  SOURCE
bigSSD/workDir/services/www_data  type                     filesystem                             -
bigSSD/workDir/services/www_data  creation                 Mon Mar 25  6:58 2024                  -
bigSSD/workDir/services/www_data  used                     28.0G                                  -
bigSSD/workDir/services/www_data  available                49.9T                                  -
bigSSD/workDir/services/www_data  referenced               27.9G                                  -
bigSSD/workDir/services/www_data  compressratio            1.08x                                  -
bigSSD/workDir/services/www_data  mounted                  yes                                    -
bigSSD/workDir/services/www_data  quota                    none                                   received
bigSSD/workDir/services/www_data  reservation              none                                   received
bigSSD/workDir/services/www_data  recordsize               128K                                   default
bigSSD/workDir/services/www_data  mountpoint               /mnt/bigSSD/workDir/services/www_data  default
bigSSD/workDir/services/www_data  sharenfs                 off                                    default
bigSSD/workDir/services/www_data  checksum                 on                                     default
bigSSD/workDir/services/www_data  compression              zstd                                   inherited from bigSSD/workDir/services
bigSSD/workDir/services/www_data  atime                    off                                    inherited from bigSSD
bigSSD/workDir/services/www_data  devices                  on                                     default
bigSSD/workDir/services/www_data  exec                     on                                     default
bigSSD/workDir/services/www_data  setuid                   on                                     default
bigSSD/workDir/services/www_data  readonly                 off                                    inherited from bigSSD
bigSSD/workDir/services/www_data  zoned                    off                                    default
bigSSD/workDir/services/www_data  snapdir                  hidden                                 local
bigSSD/workDir/services/www_data  aclmode                  discard                                inherited from bigSSD
bigSSD/workDir/services/www_data  aclinherit               discard                                inherited from bigSSD/workDir
bigSSD/workDir/services/www_data  createtxg                1073433                                -
bigSSD/workDir/services/www_data  canmount                 on                                     default
bigSSD/workDir/services/www_data  xattr                    sa                                     received
bigSSD/workDir/services/www_data  copies                   1                                      local
bigSSD/workDir/services/www_data  version                  5                                      -
bigSSD/workDir/services/www_data  utf8only                 off                                    -
bigSSD/workDir/services/www_data  normalization            none                                   -
bigSSD/workDir/services/www_data  casesensitivity          sensitive                              -
bigSSD/workDir/services/www_data  vscan                    off                                    default
bigSSD/workDir/services/www_data  nbmand                   off                                    default
bigSSD/workDir/services/www_data  sharesmb                 off                                    default
bigSSD/workDir/services/www_data  refquota                 none                                   received
bigSSD/workDir/services/www_data  refreservation           none                                   received
bigSSD/workDir/services/www_data  guid                     565632788266028773                     -
bigSSD/workDir/services/www_data  primarycache             all                                    default
bigSSD/workDir/services/www_data  secondarycache           all                                    default
bigSSD/workDir/services/www_data  usedbysnapshots          38.7M                                  -
bigSSD/workDir/services/www_data  usedbydataset            27.9G                                  -
bigSSD/workDir/services/www_data  usedbychildren           0B                                     -
bigSSD/workDir/services/www_data  usedbyrefreservation     0B                                     -
bigSSD/workDir/services/www_data  logbias                  latency                                default
bigSSD/workDir/services/www_data  objsetid                 54290                                  -
bigSSD/workDir/services/www_data  dedup                    off                                    default
bigSSD/workDir/services/www_data  mlslabel                 none                                   default
bigSSD/workDir/services/www_data  sync                     standard                               default
bigSSD/workDir/services/www_data  dnodesize                legacy                                 default
bigSSD/workDir/services/www_data  refcompressratio         1.08x                                  -
bigSSD/workDir/services/www_data  written                  21.3M                                  -
bigSSD/workDir/services/www_data  logicalused              30.0G                                  -
bigSSD/workDir/services/www_data  logicalreferenced        30.0G                                  -
bigSSD/workDir/services/www_data  volmode                  default                                default
bigSSD/workDir/services/www_data  filesystem_limit         none                                   default
bigSSD/workDir/services/www_data  snapshot_limit           none                                   default
bigSSD/workDir/services/www_data  filesystem_count         none                                   default
bigSSD/workDir/services/www_data  snapshot_count           none                                   default
bigSSD/workDir/services/www_data  snapdev                  hidden                                 default
bigSSD/workDir/services/www_data  acltype                  posix                                  inherited from bigSSD/workDir
bigSSD/workDir/services/www_data  context                  none                                   default
bigSSD/workDir/services/www_data  fscontext                none                                   default
bigSSD/workDir/services/www_data  defcontext               none                                   default
bigSSD/workDir/services/www_data  rootcontext              none                                   default
bigSSD/workDir/services/www_data  relatime                 on                                     default
bigSSD/workDir/services/www_data  redundant_metadata       all                                    default
bigSSD/workDir/services/www_data  overlay                  on                                     default
bigSSD/workDir/services/www_data  encryption               aes-256-gcm                            -
bigSSD/workDir/services/www_data  keylocation              none                                   default
bigSSD/workDir/services/www_data  keyformat                passphrase                             -
bigSSD/workDir/services/www_data  pbkdf2iters              350000                                 -
bigSSD/workDir/services/www_data  encryptionroot           bigSSD/workDir                         -
bigSSD/workDir/services/www_data  keystatus                available                              -
bigSSD/workDir/services/www_data  special_small_blocks     0                                      default
bigSSD/workDir/services/www_data  snapshots_changed        Mon Mar 25  7:01:25 2024               -
bigSSD/workDir/services/www_data  org.freenas:description                                         local
bigSSD/workDir/services/www_data  org.truenas:managedby                            received