swap disappeared from boot pool - half broken

[Void64]

TrueNAS-13.0-U6.1

New install. Using two 480G enterprise grade SSD's in a zfs mirrored boot pool. Upon install, chose to create the swap on the boot disks. (which is fine, we realize they're not used hardly at all, if ever)

Crated the main zpool for data. By default this caused TrueNAS to move the swap and create 2G mirrors on every device. (I get it). But I already have a mirrored zpool on the boot devices with a TON of space that's not being used.

Destroyed the pool and recreated it with the swap size set to "0". This worked as expected on the primary storage pool. No swap partitions.

However, when I did this and rebooted the server, TrueNAS will only pick up ONE of the SSD's and not the gmirror partition. In fact, it looks like it destroyed the gmirror and now I only have two "adaXp3" partitions as type "freebsd-swap"

I tried manually recreating the gmirror, and I can stop swap, create the mirror and then turn the swap back on for the mirror, but if I reboot, it reverts again by destroying the gmirror and goes back to using one one of the two drives.

Mirror before with swap:

Code:

Name    Status  Components
mirror/swap0  COMPLETE  ada1p3 (ACTIVE)
                        ada0p3 (ACTIVE)
                    

=>       40  937703008  ada0  GPT  (447G)
         40     532480     1  efi  (260M)
     532520   33554432     3  freebsd-swap  (16G)
   34086952  903610368     2  freebsd-zfs  (431G)
  937697320       5728        - free -  (2.8M)


=>       40  937703008  ada1  GPT  (447G)
         40     532480     1  efi  (260M)
     532520   33554432     3  freebsd-swap  (16G)
   34086952  903610368     2  freebsd-zfs  (431G)
  937697320       5728        - free -  (2.8M)

After reboot,, partition table is the same, however mirror is gone and I see:

Code:

swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/ada1p3.eli  16777216        0 16777216     0%

I would expect this to be "/dev/mirror/swap0.eli" which is what it was before I rebuilt the primary storage pool.

Is there a way to have gmirror persist and have TrueNAS use that device? I'm not really sure why it keeps destroying the gmirror. That has always been persistent within FreeBSD, so something must keep breaking and recreating it.

I know /etc/fstab is not persistent either, so it's not like I can list the new device there. I'm trying to figure out how/where TrueNAS devices how/what devices to use for swap when it starts... not having much luck.

 

[Void64]

I just tried again, deleting my main pool and re-creating it. When I do so, with swap on pool set to "0" it does not create swap partitions/mirrors on the data drives as expected. When the pool completes it appears the gmirror swap on the boot device is correct.

Code:

nas1# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli  16777216        0 16777216     0%

nas1# gmirror status
        Name    Status  Components
mirror/swap0  COMPLETE  ada1p3 (ACTIVE)
                        ada0p3 (ACTIVE)
                        

But when I reboot the server...

Code:

nas1# gmirror status

nas1# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/ada1p3.eli  16777216        0 16777216     0%

 

[Void64]

How does TrueNAS setup/turn-on swap on boot? It doesn't appear to be in /etc/fstab where FreeBSD does it by default. So there must be someplace in the TrueNAS startup that is turning on swap and setting this up. I cannot for the life of me find it. I would like to look at this and perhaps understand if we're looking at some sort of bug so I can report it as such.

 

[Void64]

This is some kind of bug, will be submitting.

To recreate: Install TrueNAS on it's own mirrors boot drive. Choose to install 16GB swap at install time. (where it should be). Swap on the system now is mirrorr/swap0

From there, in advanced options choose NOT to install swap into new pools (ie: swap 0 GiB). Create a new pool. New pool is created with no swap partitions/mirrors as expected.

Reboot

mirror/swap0 is gone and system will only try to use one of the members for swap, in our case, the second drive.

After reboot:

Code:

root@nas1[~]# ls -l /dev/ada*
crw-r-----  1 root  operator   0xc5 Mar 27 16:47 /dev/ada0
crw-r-----  1 root  operator   0xd7 Mar 27 16:47 /dev/ada0p1
crw-r-----  1 root  operator   0xd9 Mar 27 16:47 /dev/ada0p2
crw-r-----  1 root  operator   0xdb Mar 27 16:47 /dev/ada0p3
crw-r-----  1 root  operator   0xc7 Mar 27 16:47 /dev/ada1
crw-r-----  1 root  operator   0xdd Mar 27 16:47 /dev/ada1p1
crw-r-----  1 root  operator   0xdf Mar 27 16:47 /dev/ada1p2
crw-r-----  1 root  operator   0xe1 Mar 27 16:47 /dev/ada1p3
crw-r-----  1 root  operator  0x121 Mar 27 16:48 /dev/ada1p3.eli

root@nas1[~]# gpart show ada0
=>       40  937703008  ada0  GPT  (447G)
         40     532480     1  efi  (260M)
     532520   33554432     3  freebsd-swap  (16G)
   34086952  903610368     2  freebsd-zfs  (431G)
  937697320       5728        - free -  (2.8M)

root@nas1[~]# gpart show ada1
=>       40  937703008  ada1  GPT  (447G)
         40     532480     1  efi  (260M)
     532520   33554432     3  freebsd-swap  (16G)
   34086952  903610368     2  freebsd-zfs  (431G)
  937697320       5728        - free -  (2.8M)

root@nas1[~]# dmesg | grep ELI
GEOM_ELI: Device ada1p3.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: accelerated software