2 New identical zfs pools, only 1st pool has .system dataset subfolders. Why?

[rigel]

Configuration

TrueNAS-SCALE-22.12.2
CPU: 16 Cores AMD EPYC
RAM: 64Gb DDR4 ECC Registered
Boot drive: 50GB SATA SSD MLC
1st zfs pool "archive": 4 x 4TB NVMe M.2 SSD in raidz-1
2nd zfs pool "projects": 4 x 1.88TB NVMe M.2 SSD in raidz-1

When I run command

Code:

zfs list -o space

it outputs this:

Code:

root@truenas[~]# zfs list -o space
NAME                                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
boot-pool                                                  44.8G  2.64G        0B     96K             0B      2.64G
boot-pool/ROOT                                             44.8G  2.63G        0B     96K             0B      2.63G
boot-pool/ROOT/22.12.2                                     44.8G  2.63G     4.01M   2.63G             0B         0B
boot-pool/ROOT/Initial-Install                             44.8G     8K        0B      8K             0B         0B
boot-pool/grub                                             44.8G  8.17M        0B   8.17M             0B         0B
archive                                                    10.4T  7.42G        0B    140K             0B      7.42G
archive/.system                                            10.4T  21.3M        0B    174K             0B      21.1M
archive/.system/configs-40cc91dacae0491e84781ab81ded8ba4   10.4T   256K        0B    256K             0B         0B
archive/.system/cores                                      1024M   140K        0B    140K             0B         0B
archive/.system/ctdb_shared_vol                            10.4T   140K        0B    140K             0B         0B
archive/.system/glusterd                                   10.4T   151K        0B    151K             0B         0B
archive/.system/rrd-40cc91dacae0491e84781ab81ded8ba4       10.4T  18.4M        0B   18.4M             0B         0B
archive/.system/samba4                                     10.4T   326K        0B    326K             0B         0B
archive/.system/services                                   10.4T   140K        0B    140K             0B         0B
archive/.system/syslog-40cc91dacae0491e84781ab81ded8ba4    10.4T  1.46M        0B   1.46M             0B         0B
archive/.system/webui                                      10.4T   140K        0B    140K             0B         0B
archive/archive                                            10.4T  7.40G        0B   7.40G             0B         0B
projects                                                   4.81T  27.5G        0B    140K             0B      27.5G
projects/projects                                          4.81T  27.5G        0B   27.5G             0B         0B

Does anyone know what is this .system subfolders are and why the 2nd pool is missing them? What is the logic here?

I thought zfs pools are created independently. But if my second zfs pool "projects" relies on some system files on the first zfs pool "archive" and something goes wrong with any of 1st pool's drives then both pools are affected?

 

[rigel]

Ok, I figured one can move this .system dataset to a boot volume drive by navigating to
System Settings > Advanced > Storage > Configure > Select System Dataset Pool

So now it looks like this:

Code:

NAME                                                        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
boot-pool                                                   44.8G  2.66G        0B     96K             0B      2.66G
boot-pool/.system                                           44.8G  16.1M        0B    112K             0B      16.0M
boot-pool/.system/configs-40cc91dacae0491e84781ab81ded8ba4  44.8G   188K        0B    188K             0B         0B
boot-pool/.system/cores                                     1024M    96K        0B     96K             0B         0B
boot-pool/.system/ctdb_shared_vol                           44.8G    96K        0B     96K             0B         0B
boot-pool/.system/glusterd                                  44.8G   104K        0B    104K             0B         0B
boot-pool/.system/rrd-40cc91dacae0491e84781ab81ded8ba4      44.8G  13.9M        0B   13.9M             0B         0B
boot-pool/.system/samba4                                    44.8G   232K        0B    232K             0B         0B
boot-pool/.system/services                                  44.8G    96K        0B     96K             0B         0B
boot-pool/.system/syslog-40cc91dacae0491e84781ab81ded8ba4   44.8G  1.23M        0B   1.23M             0B         0B
boot-pool/.system/webui                                     44.8G    96K        0B     96K             0B         0B
boot-pool/ROOT                                              44.8G  2.63G        0B     96K             0B      2.63G
boot-pool/ROOT/22.12.2                                      44.8G  2.63G     4.03M   2.63G             0B         0B
boot-pool/ROOT/Initial-Install                              44.8G     8K        0B      8K             0B         0B
boot-pool/grub                                              44.8G  8.17M        0B   8.17M             0B         0B
archive                                                     10.4T  5.70G        0B    140K             0B      5.70G
archive/archive                                             10.4T  5.70G        0B   5.70G             0B         0B
projects                                                    4.81T  27.0G        0B    140K             0B      27.0G
projects/projects                                           4.81T  27.0G        0B   27.0G             0B         0B

However, why this .system dataset was created on the zfs pool in the first place? What is the logic and are there any disadvantages of keeping this system dataset on a boot volume? Considering both boot volume and zfs pool are using ssd drives.

 

[Patrick M. Hausen]

The system dataset is frequently written to. Sometimes people use media that does not have a good write endurance for their boot pool. E.g. USB thumb drives, although they are explicitly not recommended. Or cheap consumer SSDs. So the default is to put the system dataset on the first storage pool created - which for most home/hobbyist users used to be built from HDDs.

Added bonus: in case the boot medium does eventually fail there will be a copy of the configuration database on the storage pool, even if you never saved a configuration backup manually.

 

[rigel]

The system dataset is frequently written to. Sometimes people use media that does not have a good write endurance for their boot pool. E.g. USB thumb drives, although they are explicitly not recommended. Or cheap consumer SSDs. So the default is to put the system dataset on the first storage pool created - which for most home/hobbyist users used to be built from HDDs.

Added bonus: in case the boot medium does eventually fail there will be a copy of the configuration database on the storage pool, even if you never saved a configuration backup manually.

Thank you so much for your explanation. Now that makes sense. However for my use case it is the boot drives that have 10 DWPD endurance and SSD drives for zfs pools are 1DWPD or lower.

Is it correct to think that if let's imagine my above system with boot drive fails and I will keep my .system dataset on a boot drive. Can I take all my zfs pools drives:
1st zfs pool "archive": 4 x 4TB NVMe M.2 SSD in raidz-1
2nd zfs pool "projects": 4 x 1.88TB NVMe M.2 SSD in raidz-1

Connect them to another new TrueNAS system, and safely import zfs pools and have all the files back? The only thing would be missing are those system configurations like shares, network and other settings? Or this .system dataset contains some metadata that is required to import zfs pools and by loosing .system dataset you are basically loosing all zfs pools data and files?

 

[Patrick M. Hausen]

All your data is on your storage pools. The system dataset contains logs and configuration. Nothing your data depends on. It might if you use encryption. I honestly don't know. But you are advised to save your keys offline, anyway.