Swap gets uses although there is unused RAM

[chuck32]

Hello again,

I recently installed truenas scale as my new hypervisor. Specs:

Mainboard asrock b560m-itx/ac
CPU i3 10100
32 GB RAM
M2 (internal)
- 500 GB as boot drive
SATA (internal)
- 4x WD Red Plus 4 TB directly connected via, raidz2
Delock 5 port SATA PCI Express x4 Card
- 512GB and 256 GB SSD (2,5") for VMs and Apps, mirror

VMs:
Home Assistant 3 GB RAM allocated
Ubuntu 4 GB allocated (switched to 2 GB around 22:10 in the screenshot)

During the day there was a cloud backup task running, should have finished around 21:00 I think. I don't know why the swap was freed during the day, I was at work.

I found a user had a similiar problem here.

I wasn't able to replicate top -w o swap like mentioned in the post but I found out how to add swap as a column and just filtered for SWAP>0

However the user was running 2x4x4Tb and got the recommendation to upgrade, I only run half the storage.

Is the swap usage reason for concern? Edit: also I'm pretty sure truenas asked to create a 16 Gb swap partition during installation, slightly confused why the swap is 4 Gb

My problem is, my mainboard only has two RAM slots, and since I bought slower RAM (2666 Mhz / CL 19) I can't really pair it with the 3200 MHz / CL 16) from my main rig nor would I have use for so much RAM in that machine.
Upgrading would basically mean I would need to invest in 2x32 GB modules and scrap the 2x16 GB modules I have right now. Maybe I could sell them on ebay, but I doubt I'd get much of my money back although it's just a month old. The whole build got more expensive down the road anyway since I didn't pick the hardware for truenas but with VMs in mind and now I'm learning the hard way that truenas would have required some different choices.

Thanks again for your help!

 

[mltam]

I don't know about bsd, and I don't know if you're using the linux dist. But linux will use swap to free up memory. The idea is to put those parts of memory that are accessed very rarely to disk, so that more memory can be used for other things, e.g. as buffer to speed up disk access.

There are kernel parameters that can change the swappiness. But if you just want to know if something is wrong, then I don't see anything wrong. (You can also use the command "free -h" to see the state of memory)

 

[chuck32]

I'm using the current scale installation so linux based. So basically if I don't see the physical RAM running out completely to 0 gb I'm good?

Still wondering, now that the topic came up, why it's only 4gb when during install I was asked whether I want to create a 16 gb partition.

Thank you for your help! Great community here!

 

[mltam]

I don't know why that happened. Must be either a bug or a problem with the interface.
You could add a swapfile to the swap, to make things somewhat more secure. Normally these work pretty well. I'm not sure what the gurus of FreeNAS think about it in this case. You can simply make a swap file of size 12GB on the partition that's too big, if you don't want to take other, more drastic measures. In general, running out of physical memory once is not so bad. Running out many times is bad, since in each case your machine runs slowly, swapping memory in and out. Running out of physical+swap is horrible, since it starts to kill jobs.

 

[chuck32]

Thanks!
I investigated further and saw that apparantly the 16 gb swap does not get used at all. nvme0n1p4 seems to be the partition created during install. I'm not sure if I misclicked somewhere during pool creation, however after I set up the 4x4 tb pool, the system dataset was on that pool. I changed it via the GUI to my boot pool.

How can I move the swap files from all drives to the nvme ssd? I guess I don't need SWAP to be mirrored because if I move it to the unmirrored boot drive and that fails I can't run the system anyway and have to reinstall and restore the config on a new drive.
Or if unmirrored is discouraged, how can move the SWAP to my 2 SSDs (or the two + boot drive) attached via the HBA? I'd prefer the swap on SSDs rather than my HDD backup pool.
edit: Or if it wouldn't be recommended to remove the 2gb SWAPs, how can I activate the 16 gb swap and maybe use that one first. Should never fill up anyway.

Code:

admin@truenas[~]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0   3.6T  0 disk
├─sda1        8:1    0     2G  0 part
│ └─md126     9:126  0     2G  0 raid1
│   └─md126 253:0    0     2G  0 crypt [SWAP]
└─sda2        8:2    0   3.6T  0 part
sdb           8:16   0   3.6T  0 disk
├─sdb1        8:17   0     2G  0 part
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:1    0     2G  0 crypt [SWAP]
└─sdb2        8:18   0   3.6T  0 part
sdc           8:32   0   3.6T  0 disk
├─sdc1        8:33   0     2G  0 part
│ └─md126     9:126  0     2G  0 raid1
│   └─md126 253:0    0     2G  0 crypt [SWAP]
└─sdc2        8:34   0   3.6T  0 part
sdd           8:48   0   3.6T  0 disk
├─sdd1        8:49   0     2G  0 part
│ └─md126     9:126  0     2G  0 raid1
│   └─md126 253:0    0     2G  0 crypt [SWAP]
└─sdd2        8:50   0   3.6T  0 part
sde           8:64   0 238.5G  0 disk
├─sde1        8:65   0     2G  0 part
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:1    0     2G  0 crypt [SWAP]
└─sde2        8:66   0 236.5G  0 part
sdf           8:80   0 465.8G  0 disk
├─sdf1        8:81   0     2G  0 part
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:1    0     2G  0 crypt [SWAP]
└─sdf2        8:82   0 463.8G  0 part
zd16        230:16   0    50G  0 disk
zd32        230:32   0    20G  0 disk
nvme0n1     259:0    0 465.8G  0 disk
├─nvme0n1p1 259:1    0     1M  0 part
├─nvme0n1p2 259:2    0   512M  0 part
├─nvme0n1p3 259:3    0 449.3G  0 part
└─nvme0n1p4 259:4    0    16G  0 part  

 

[mltam]

Ok, it seems there is much more magic to TrueNAS swap than I thought. Here are two threads on the subject:

Thread on how swap is handled
Info on how to relocate

This also answers the question of what happened to your swap.

but, I wouldn't do anything till someone more knowledgeable than me comments on this...

 

[chuck32]

Update: I got a good deal on a 64 Gb Kit, since the upgrade swap wasn't used anymore.
The screenshot is with 11 Gb dedicated for VMs (I increased the RAM for the Ubuntu VM since now I have enough to spare).

Thanks again!