swap space usage

[bestboy]

Hi,

some time ago I read on the forum somewhere* that a FreeNAS box should not be using its swap space.

Could anybody clarify why that is? I don't really see why the system should not make use of the swap space which is part of its virtual memory after all.
Also is this statement to be considered absolute ("It will never use it. If it does, something is wrong.") or is it more like a rule-of-thumb ("Ideally it would never use it, but it could happen.").

So what I'm tying to find out is basically

• is it ok to use swap?
  • if no
    • why is that?
  • if yes
    • when is it ok?
    • how much is ok?
    • is there a way to see what was swapped out?

Background of me asking is that I recently did some experimentation with kernel memory tunables and noticed an increase in swap usage as a result. I did not really notice any issues but still reverted my changes as I remembered that vague "Thou shalt not use thy swap space" in the back of my head.
However, once I did revert back I could still see small amounts of swap being used from time to time. Now I'm not sure how to treat swap usage and whether it is normal or not.

Code:

Mem: 126M Active, 1588M Inact, 28G Wired, 1486M Free
ARC: 22G Total, 11G MFU, 9612M MRU, 818K Anon, 744M Header, 1349M Other
Swap: 12G Total, 12K Used, 12G Free

The purpose of this thread is to get some clarification about how swap space is supposed to be used as well as to ask around if/how you guys see swap usage on your boxes. So feel free to share your thoughts on the matter and/or participate in the poll.

About the poll:
You can lookup your swap usage in the GUI via "Display system processes" or with top from the command line. If your system is making notable use of swap space there will also be given a percentage.

Please only take the poll if your system

• meets the minimum requirements
• has its ARC filled and
• is kinda busy at the time you look for its swap usage

*TODO: find reference

 

[Luna]

So what I'm tying to find out is basically

• is it ok to use swap
  • if no
    • why is that?

It's a trait of ZFS more than of FreeNAS. As far as I'm aware, ZFS doesn't allow using swap space in any cases. And as it would happen, the lion's share of your RAM usage is going to be ZFS. The rest is swappable, which is probably what you're seeing.

 

[jgreco]

It's not okay to use swap (*). If you're using swap, there's insufficient memory in the system, or some other problem. A FreeNAS box is *massive* when compared to a FreeBSD system, there's nothing in the underlying host that should cause 8GB or more of RAM to be consumed.

(*) It does happen that sometimes a small amount of swap will be used, such as the 8GB filer here that's swapped 7MB out over the last 64 days. That's probably okay.

 

[bestboy]

Thanks for the contribution so far, but I feel the matter is still not entirely clear to me.

If I'd try to summarize the given reply's and doing a little reading between the lines I'd come up with the following interpretation:
Seeing small amounts of swap space being used is ok. Typically we expect most of the RAM to be allocated by the ARC which is not swapable (that would be insane). So the amount of potential swap space used is more or less determined by whatever few bits of RAM are left to the kernel and the base system. On a typical file sharing FreeNAS installation where most of the RAM is used up by the ARC (I believe there is a hardcoded 7/8 limit) there is almost no "room" or need for swapping.

OK, but what if the FreeNAS box is not so "massive" or dedicated?
There could be numerous rather large applications sitting in jails like Plex, Owncloud or full-blown databases. There is also the possibility of VMs hosted on the FreeNAS machine via VirtualBox. Those would probably also allocate large blocks of memory for their guest operating systems that could be swapable (cf. virtual memory ballooning and swapping). So do we expect to see more swap space being used in those cases? Would that be a reason to be more lenient?

 

[mav@]

While ZFS does not very like to free memory, it still can do it. Memory consumed by ARC is not fixed. If there is significant pressure, ZFS will gradually step back. Swap is there mostly for some edge cases of some large and fast allocations or memory leaks. If ZFS can not shrink ARC fast enough, or it is already at lower limit, system swapper will push something else out to swap. It it just a safety belt, because if that rare situation is reached and there is no swap configured, then some services may get killed that is not good.

 

[jgreco]

Thanks for the contribution so far, but I feel the matter is still not entirely clear to me.

If I'd try to summarize the given reply's and doing a little reading between the lines I'd come up with the following interpretation:
Seeing small amounts of swap space being used is ok. Typically we expect most of the RAM to be allocated by the ARC which is not swapable (that would be insane). So the amount of potential swap space used is more or less determined by whatever few bits of RAM are left to the kernel and the base system. On a typical file sharing FreeNAS installation where most of the RAM is used up by the ARC (I believe there is a hardcoded 7/8 limit) there is almost no "room" or need for swapping.

OK, but what if the FreeNAS box is not so "massive" or dedicated?
There could be numerous rather large applications sitting in jails like Plex, Owncloud or full-blown databases. There is also the possibility of VMs hosted on the FreeNAS machine via VirtualBox. Those would probably also allocate large blocks of memory for their guest operating systems that could be swapable (cf. virtual memory ballooning and swapping). So do we expect to see more swap space being used in those cases? Would that be a reason to be more lenient?

I like that you're obviously trying to figure this out for yourself and think your way through it. Bonus points to you.

I believe in FreeBSD 10 the new memory management will be a little better at coping with things like that (reducing ZFS usage upon sudden demand). Under 9, I believe there's a much greater possibility of a system with insufficient resources (in particular, perhaps not exclusively) being caught off-guard by sudden demand, because ZFS sort-of self-manages its own memory usage and will try to free memory as it observes the host system being under memory pressure. This is something I've not had the time to investigate so I'm kind of handwaving a little bit while I say this.

I would suggest that the better resolution to that issue would be to make sure that you're not under-resourcing your FreeNAS system. You should probably figure memory requirements for those separately, and allow the system to start up those things and allocate memory for them early on, where possible. It would also be possible to reduce space for the ARC somewhat to provide more space for sudden userland growth.

The big danger with swap is that there's a possibility that one of the disks backing it could fail, taking a critical but infrequently used bit of the system along with it for the ride. That's bad, and why you don't want swap to be used.

 

[AngeleToR]

Sorry to bump this back, but regarding swap management, even with 0 zero KB of swap used, I've had a failure of a single drive in a 5 disk RAIDZ pool and that took my whole system down with panics everywhere...

So by your answer, can I assume that the swap problem is still around??? I like my system to be tolerant to disk failures, and it is except for this swap annoyance.

 

[MrToddsFriends]

Stux wrote a script that helps to prevent the negative consequences of swap usage.

https://forums.freenas.org/index.ph...ny-used-swap-to-prevent-kernel-crashes.46206/

If really zero swap was used your system most probably suffered from another problem.

 

[AngeleToR]

I have no words.... Big thanks MrToddsFriends for pointing me to that script, and big thanks to the script creator!!! I'll give it a go, but looks promising... I will also deactivate completely swap on the most prone to fail devices, the spinners disks, but allow it on my SSD mini RAID...

Shiny day is today!!!!

 

[jclende]

Is this script still the recommended solution for mitigating swap issues? In FreeNAS 9.10, I am still seeing crashes when a drive is pulled unexpectedly with swap on.

I currently have swap disabled, but I'd rather have it on with some redundancy.

 

[MrToddsFriends]

Is this script still the recommended solution for mitigating swap issues? In FreeNAS 9.10, I am still seeing crashes when a drive is pulled unexpectedly with swap on.

I'm still using this script after upgrading to FreeNAS 11.0 as this version has the same (or at least a similar) unwanted swap usage behavior as seen in 9.10.1. The origin of the script dates back to the time when 9.10.1 was the current stable version.

I currently have swap disabled, but I'd rather have it on with some redundancy.

Unfortunately, I do not know any current schedule, how/when mirrored swap will be implemented in FreeNAS 11.

 

[rogerh]

Sorry to go back to this, just for an interesting observation. My system regularly, right up to 11.0_U2, used to use about 5 to 40 MB of swap after a few hours of uptime. I have no idea what process leads to this, but I presume it is something transient as usually ARC is using about 12GB of my 16GB. I used the suggestion at:

https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-the-boot-drive.69/

in the 'Resources' section of the forum to transfer my swap to a file on the mirrored boot disks. Since then, about a week, no swap has been used. I should point out that my boot disks, despite being SSDs, are somewhat slow as they are run on PIO4 which is a rather obsolete IDE protocol. I wonder if the system finds access to swap so slow that it manages without by the time it gets access to the file? It doesn't seem to have broken anything.

Edit. Apparently mirrored swap is planned for 11.1

https://bugs.freenas.org/issues/23523

and it will be interesting to go back to the default use of swap on the data disks and see if the system starts swapping again.

 

[Stux]

As the author of the script, and also the two resources...l

Swap happens. You should have swap enabled.

If your boot is an ssd, I'd suggest using the following instructions:
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-the-boot-drive.69/

To relocate swap to your boot ssd.

Alternatively if you have another partition of device which is more reliable than your pool, then use this resource:

https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

If neither of these options apply to you, then I suggest using the script on a regular basis via cron (Ie every 10 minutes) to try to reduce the chance of swap being in use on a drive that fails suddenly

If you are going to remove a drive, you should offline it first. This will vacate any used swap.

And hopefully mirrored swap makes it into 11.1. While spelunking through the swap setup code it looked like it was mostly there already in 11.0, but perhaps only enabled on TrueNAS.... Yoh FreeNAS heathens you ;)

Ps: my lightly used 128GB filer saw 640M of swap last week.

 

[rogerh]

As the author of the script, and also the two resources...l

Swap happens. You should have swap enabled.

100% agree with that. What I am describing is that I *do* have a swap file on the boot SSDs enabled, but my system, with nothing else changed in the server or usage, seems to have stopped using swap. But I am not relying on that continuing, it's just interesting.