Why does TrueNAS SCALE didn't use the 16GB swap memory?

[lowrents]

Hi!

joining the TrueNAS SCALE family almost 4 days ago, I am currently trying to understand how the OS handles swap memory.

The OS is currently running on an external SSD (Crucial MX500 via USB 3.2) on a single partition using this guide on reddit which basically edits the install script before the initial installation. During the installation I accepted the 16GB swap creation and I am also able to see the partition using parted for example.

Show : parted output

Code:

root@truenas[/mnt/ssd-storage/ix-applications]# parted /dev/sde print
Disk /dev/sde: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number  Start   End     Size    File system  Name  Flags
 1      2097kB  3146kB  1049kB                     bios_grub, legacy_boot
 2      3146kB  540MB   537MB   fat32              boot, esp
 4      540MB   17.7GB  17.2GB                     swap
 3      17.7GB  86.4GB  68.7GB  zfs
 5      86.4GB  1000GB  914GB   zfs

But I can also see, that each of the 4 drives are also holding a 2GB swap partition, which seems normal and could be solved via CLI.

Show : output lsblk

Code:

root@truenas[~]# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda            8:0    0   3.6T  0 disk
├─sda1         8:1    0     2G  0 part
│ └─md127      9:127  0     2G  0 raid1
│   └─md127  253:0    0     2G  0 crypt [SWAP]
└─sda2         8:2    0   3.6T  0 part
sdb            8:16   0   3.6T  0 disk
├─sdb1         8:17   0     2G  0 part
│ └─md127      9:127  0     2G  0 raid1
│   └─md127  253:0    0     2G  0 crypt [SWAP]
└─sdb2         8:18   0   3.6T  0 part
sdc            8:32   0   3.6T  0 disk
├─sdc1         8:33   0     2G  0 part
│ └─md126      9:126  0     2G  0 raid1
│   └─md126  253:1    0     2G  0 crypt [SWAP]
└─sdc2         8:34   0   3.6T  0 part
sdd            8:48   0   3.6T  0 disk
├─sdd1         8:49   0     2G  0 part
│ └─md126      9:126  0     2G  0 raid1
│   └─md126  253:1    0     2G  0 crypt [SWAP]
└─sdd2         8:50   0   3.6T  0 part
sde            8:64   0 931.5G  0 disk
├─sde1         8:65   0     1M  0 part
├─sde2         8:66   0   512M  0 part
├─sde3         8:67   0    64G  0 part
├─sde4         8:68   0    16G  0 part
└─sde5         8:69   0   851G  0 part

Looking for the way linux normally seems to handles the swap I've the seen that root@truenas# cat /etc/sysctl.conf shows that this document doesn't seem to be used. On the other hand root@truenas# cat /etc/fstab produces an output and seems to mount two devices.

Show : output fstab

Code:

root@truenas[~]# cat /etc/fstab
boot-pool/grub    /boot/grub    zfs    relatime,defaults    0    0
tmpfs    /run/lock    tmpfs    rw,nosuid,nodev,noexec,relatime,size=100m    0    0

Also root@truenas[~]# swapon --show does.

Show : swapon output

Code:

NAME      TYPE      SIZE   USED PRIO
/dev/dm-0 partition   2G     2G   -2
/dev/dm-1 partition   2G 425.5M   -3

I've also tried do dd a swapfile (with chmod 600, mkswap, swapon) using this guide which doesn't work.

So my question is, how does SCALE handles swap and how can I utilise the 16GB swap partition created during the setup.

I think I am currently at a point where I cannot proceed due to my lack in linux knowledge. But perhaps some else does.
Thanks for any idea in advance!

 

[Samuel Tai]

SCALE ignores manual edits to /etc/fstab, because it regenerates that file from its configuration database. For swap safety, SCALE tries really hard to create 3-way mirrors of identically-sized swap partitions (default size of 2G) over multiple disks. This is to prevent system crashes if a swap partition disappears on a disk failure. With a mirror, swap can continue operating if a disk hosting a swap partition dies.

Number of pool disks	Swap behavior
1	1 active swap partition
2	2-way mirrored swap partition
3	3-way mirrored swap partition
4	3-way mirrored swap partition, 1 spare swap partition
5	3-way mirrored swap partition, 2 spare swap partitions
6	2-way stripe of 3-way mirrored swap partitions
7	2-way stripe of 3-way mirrored swap partitions, 1 spare swap partition

And so on. This means one-off odd-sized swap partitions are completely ignored, because they can't fit within this scheme.

 

[lowrents]

Thank you very much for taking your time to reply. I appreciate your explanation. It is not a good idea to edit the configuration database in order to change the /etc/fsatb I suppose?

One question then still remains: Why does the system let me create the 16GB swap partition on my boot device during installation? Is it any kind of leftover from an older release on the installer, and not used by the system anymore? Because after accepting the prompt I expected to have 16GB of swap, it's kind of misleading.

 

[Samuel Tai]

It is not a good idea to edit the configuration database in order to change the /etc/fstab I suppose?

It's a very bad idea. The only safe ways to modify the configuration database are via the web UI and the SCALE API. So far as I know, neither exposes any way to modify swap mounts in /etc/fstab.

One question then still remains: Why does the system let me create the 16GB swap partition on my boot device during installation? Is it any kind of leftover from an older release on the installer, and not used by the system anymore? Because after accepting the prompt I expected to have 16GB of swap, it's kind of misleading.

I suspect this is a carryover from the Debian installer. The 16G swap partition is likely only used during the install.

 

[lowrents]

As I found this post it seems to be possible to disable the default swap on the HDDs. I also found something in the API reference (search for swapondrive) but I am actually unsure, if this can be used for define the swap partition on the SSD, but it looks like. But even if this API call will succeed, would it be possible that the system actually uses this swap since it won't have a mirror.

Alright, so I also think it is safe to remove this partition.

 

[Samuel Tai]

The API call sets the default swap partition size on new pool disks. It's possible to set this to 0, to disable swap on new pool disks, but this is not recommended. It doesn't allow for adding swap partitions on an ad-hoc basis.

 

[lowrents]

Okay got it, thank you for the explanation. I was hoping to be able to avoid using the HDD swap, on the one hand to take advantage of the much faster SSD and on the other hand to potentially put the disks into sleep in case they are not used overnight for example. But with the swap partition I think it won't be possibles.