Understanding memory usage

[MrToddsFriends]

I am also just doing basic file sharing for now. And a backup every now and then. Surely the swap file exists to make things faster which means even if it is there it would help rather than hinder the performance?

Using portions of the swap area while there is plenty of RAM in use for ARC can potentially hinder performance if parts of the system software are swapped out, for example. Eliminating the ability to swap is not a good idea in general, though.

 

[Chris Moore]

I am also just doing basic file sharing for now. And a backup every now and then. Surely the swap file exists to make things faster which means even if it is there it would help rather than hinder the performance?

Cache helps performance. Swap exists to keep the system from running out of memory. If the system did run out of memory, it would cause the system to lockup or crash. Any time the contents of RAM is swapped to disk, the system is slowed down by the process and when what is in the swap space on disk is needed back in RAM, moving that data slows the system down again. In an ideal situation, you would either not use swap at all or have your swap space on a SSD, so it is as fast as possible. The default configuration for FreeNAS is to have a small amount (2GB) of swap space on each drive other than the boot drive. Part of the reason they don't use the boot drive for swap is the original design idea of having the system boot from a USB stick. The folks at iXsystems are still suggesting that as the default (best) option and even going so far as explicitly saying that using SSD or HDD is not recommended, which makes it sound like a bad idea.
The thing that is happening here is that the ARC (Adaptive Replacement Cache) is filling up RAM and when the system has something happen that makes it need some more memory, it needs to swap something out.
Earlier in this thread, I commented about not seeing any swap usage on my system and I figured out the reason for that. It isn't down to the amount of RAM you have or what processes you are running, it is down to how the default system is tuned to limit the size of the ARC. There are tunables (http://doc.freenas.org/11/system.html?highlight=tunabl#tunables) that can be set to manage how much RAM can be used for ARC and I set some of those a long time ago and have been carrying that configuration forward for so many years I forgot that I had done it.
I set a tunable called vfs.zfs.arc_max which you can find out more about here:
https://forums.freenas.org/index.php?threads/vfs-zfs-arc_max-has-immediate-effect.55768/
In the thread pointed out by @MrToddsFriends , above, they discuss what looks like a better way to do it.

 

[Johnny Fartpants]

My 128 GB home system would disagree with you:
View attachment 23446

I second that. My 256GB machine looks the same.