How to stop FreeNAS from using swap

[Chris Moore]

It is just that I didn't see swap usage before 11.1 and now I am seeing the usage spike. From what I know zfs relies on RAM to do lots of things and that is why ECC ram is recommended but if swap that is spread across my 6 drives and some of the drives start to malfunction, there is a slight chance of loosing my data or having it in unrecoverable state as swap might get corrupted too.

The swap in 11.1 sets up the available partitions on the disks as a series of ZFS mirrors. In my system, I have 12 drives in the main pool so I get six mirrors and all the parity and checksum rules of ZFS apply. The data in swap is no more likely to get corrupted than the data in your main storage pool. If you only have 6 disks, you will have three mirrors of 2GiB each, or 6GiB total swap space, which aligns with the illustration you posted. It should be perfectly safe.
Prior versions of FreeNAS did not mirror the swap, so it was not as safe before the new version.

 

[bodriye]

I just saw a post on FreeNAS reddit with similar issue but this time the system seems to run out of swap space and crash.
The person seems to also be running docker. https://www.reddit.com/r/freenas/comments/7mymfg/freenas_seems_to_be_crashing/

 

[danb35]

The swap in 11.1 sets up the available partitions on the disks as a series of ZFS mirrors.

No, it doesn't; ZFS isn't involved at all with the swap. FreeNAS uses GEOM mirrors for swap, not ZFS mirrors.

 

[Stux]

Until 11.1, the real issue with swap was that it was striped instead of mirrored, and if a drive failed (or was pulled) and swap was in use there was a high likelihood of a kernel crash.

Hence the script.

It’s probably not necessary with 11.1.

 

[Chris Moore]

No, it doesn't; ZFS isn't involved at all with the swap. FreeNAS uses GEOM mirrors for swap, not ZFS mirrors.

Sorry, I don't know what got in my head earlier. I posted this back on December 16th:

Code:

> GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
> GEOM_MIRROR: Device mirror/swap0 launched (2/2).
> GEOM_MIRROR: Device mirror/swap1 launched (2/2).
> GEOM_MIRROR: Device mirror/swap2 launched (2/2).
> GEOM_MIRROR: Device mirror/swap3 launched (2/2).
> GEOM_MIRROR: Device mirror/swap4 launched (2/2).
> GEOM_ELI: Device mirror/swap0.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap1.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap2.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap3.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware

https://forums.freenas.org/index.php?threads/hard-drive-failure-bad-things-happen.59816/#post-424148

 

[wblock]

Until 11.1, the real issue with swap was that it was striped instead of mirrored

Not striped by ZFS or FreeBSD. In other words, a failure of one does not mean an automatic failure of all.

 

[Paul van der Zwan]

I just saw a post on FreeNAS reddit with similar issue but this time the system seems to run out of swap space and crash.
The person seems to also be running docker. https://www.reddit.com/r/freenas/comments/7mymfg/freenas_seems_to_be_crashing/

I ran into some similar issue memory on a FreeBSD 11.1 VM.
You could check : https://redmine.ixsystems.com/issues/27270
I am postponing upgrading from 11.0 to 11.1 until 11.1-U1 is available.

 

[Stux]

Not striped by ZFS or FreeBSD. In other words, a failure of one does not mean an automatic failure of all.

High likelihood that critical swapped data would be present on the disk that died, if swap was in use at all.

 

[bodriye]

Swap usage creeping up again

 

[nightshade00013]

Is it possibly due to not having quite enough RAM in the system for what you are trying to do with it? If you are really worried about it maybe grab a small SSD and move the swap to there https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

I know the system is maxed out with ram but I know when I ran a single VM (4GB of ram) with 48GB of ram along with my jails I was starting to touch a little swap. I suspect that there are things happening to be able to keep the VM going that causes the system to use a little more than the allocated ram. Along with the system's needs it can quickly add up.

 

[bodriye]

If you are really worried about it maybe grab a small SSD and move the swap to there https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Unfortunately I have used up all the sata power connectors that came with my psu. I can buy some from 3rd party but I am not 100% on how reliable these power connectors are.
As for not having enough ram, that is possible but if you see the bug report the person had 144G of memory and it still swapped when the person allocated 4G to the memory.
This seems to me like a bug in 11.1

 

[nightshade00013]

Can always use an adapter to change a molex into a SATA power connector. Would be a lot better than splitting a SATA power line as those have a larger fire hazard.

It almost seems like maybe something in BHYVE is forcing swap when a VM is enabled. Maybe someone who has a straight FreeBSD system with some BHYVE VM's could check this out. With 144GB of RAM there should be no real reason for the system to use swap at all.

 

[pschatz100]

The swap in 11.1 sets up the available partitions on the disks as a series of ZFS mirrors. In my system, I have 12 drives in the main pool so I get six mirrors and all the parity and checksum rules of ZFS apply. The data in swap is no more likely to get corrupted than the data in your main storage pool. If you only have 6 disks, you will have three mirrors of 2GiB each, or 6GiB total swap space, which aligns with the illustration you posted. It should be perfectly safe.
Prior versions of FreeNAS did not mirror the swap, so it was not as safe before the new version.

On a small machine at home, I have one volume on three disks in a raidz1 and another volume on an SSD. That's four disks with a 2GB swap partition on each one. In FreeNAS 11.1, I see 4GB of swap space reported. Is that correct?

I plan to replace the three disk raidz1 volume with a four disk raidz2 volume. After I do that, there will be a total of five drives in my system. What should I expect to see for reported swap space?

 

[danb35]

That's four disks with a 2GB swap partition on each one. In FreeNAS 11.1, I see 4GB of swap space reported. Is that correct?

Yes. It's using two-device mirrors (so 2 x 2 GB in a mirror = 2 GB of capacity), and with four devices, there would be two of those mirrors. I'd guess that, with a fifth disk, you wouldn't see a change, but that's only a guess.

 

[Chris Moore]

On a small machine at home, I have one volume on three disks in a raidz1 and another volume on an SSD. That's four disks with a 2GB swap partition on each one. In FreeNAS 11.1, I see 4GB of swap space reported. Is that correct?

I plan to replace the three disk raidz1 volume with a four disk raidz2 volume. After I do that, there will be a total of five drives in my system. What should I expect to see for reported swap space?

Your first question, yes, that is correct. For your second question, I did a little experiment and I ask that you hang with me while I explain.

When I started building my current FreeNAS system, the default was to make a 2GB swap partition on each data drive and add all those swap parts together for the total swap space. Since I had 12 data drives, and 32GB of RAM, and I knew from my previous build that I didn't use swap very often and when I did use it I didn't use very much of it. I knew that I did not need 24GB of swap. What I did back then was configure my drives to have 1GB swap partitions and that gave me 12 GB of swap space that I very rarely used and when I did use some it was usually less that 1GB.
Fast forward to now and the advent of mirrored swap, and instead of having 12GB, I have 6GB of swap because the system is making mirror sets. and then striping those mirrors together. I still don't use much swap, usually 10MB or less, so I still have more than I need...

I did a quick build on my test system with three drives and although each drive had a 2GB swap space, I only ended up with one mirror created and only 2GB of usable system swap. I expect that the reason is, the odd number drive didn't have a mirror and was therefore not usable in the swap configuration.
I have to guess that the same would be true for your proposed reconfiguration. That 5th (odd number) drive would be hanging out lonely, with no partner, and your system would still be sitting at 4GB of swap.
The solution (like in square dancing) is to make sure that everyone (every drive) has a partner.

Does that help?

 

[Chris Moore]

Can always use an adapter to change a molex into a SATA power connector. Would be a lot better than splitting a SATA power line as those have a larger fire hazard.

@bodriye
I have used these SATA Power Adapter Cables from Silverstone in four different builds and found them to be reliable and of very high quality.
https://www.ebay.com/itm/Silverston...ith-Power-Stabilizing-Capacitors/252426576772
They also make a two port model:
https://www.ebay.com/itm/SilverSton...-w-Power-Stabalizing-Capacitors-/331488878828

 

[0x6a6576]

Are you still having this issue OP? My nas still uses a lot of SWAP.

 

[bodriye]

Are you still having this issue OP? My nas still uses a lot of SWAP.

Not really, I just gave up on using VMs/docker and switched to iocage jails. I get about 400MB swap usage with only heavy jail being plex.

 

[Stux]

Have you looked at this thread?
https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/

 

[bodriye]

Have you looked at this thread?
https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/

What would be recommended

Code:

vm.v_free_target
vfs.zfs.arc_free_target
vfs.zfs.zio.use_uma

values be for 32 gb of ram with no VMs to avoid swap usage?