Swap with 9.10

[hugovsky]

I'm seeing swap space being used after a few days of uptime since upgrade to 9.10. Is this normal? It's happening with 2 systems. Details below.

System 1:

Xeon 1231, X10SL7 with 32 GB.
pool 1: RAIDZ2 with 6x 3TB WD red
pool 2: striped mirrors 4x128GB SSD
FreeNAS-9.10-STABLE-201604111739 (896cc83)
uptime: 1 day, 20:43, 0 users
Cifs shares and zfs replication to system 2
Just 2-3 users at the moment


System 2:

A1SAi-2750F with 32 GB
pool 1: RAIDZ2 6x 1TB WD Green
FreeNAS-9.10-STABLE-201604111739 (896cc83)
Only for replication from system 1
uptime: 2 days before reboot

 

[m0nkey_]

It's only using a few KB on the swap, so it doesn't look like anything to be concerned about. Do you have any jails or bhyve VMs?

Also, post details from the ZFS tab for ARC.

 

[hugovsky]

No jails, no VMs. The second system is only for replication. I'll post ZFS details ASAP.

 

[hugovsky]

System 1 @ now:





System 2 @ 16:00 before reboot

 

[Stux]

I have this on my 8GB backup system. It quite consistently ends up with an average of 200MB of swap used. Generally its some unused processes which get ejected.

Its not a problem, except when a disk fails, the box crashes with a pagerd fault. Which is a problem.

Is there a tuneable or something so that I can limit the ARC or something, so that it *doesn't* use the swap, although there is swap available?

And I guess... perhaps if it needs the swap, it seems like it never stops using it... perahps a periodic script to swap-off a given disk would do the trick?

 

[Stux]

Code:

# swapoff /dev/ada0p1.eli ; swapon /dev/ada0p1.eli
# swapoff /dev/ada1p1.eli ; swapon /dev/ada1p1.eli
# swapoff /dev/ada2p1.eli ; swapon /dev/ada2p1.eli
# swapoff /dev/ada3p1.eli ; swapon /dev/ada3p1.eli
# swapoff /dev/ada4p1.eli ; swapon /dev/ada4p1.eli

Does result in swap usage going back to zero, in I think a safe manner (ie at all times there is still 8GB of swap available if it is *actually* needed), so I guess there's no reason I couldn't do that on a cron, but is it normally for this to generally end up with a few hundred megs of swap used after a day or so?

Code:

# swapctl -l
Device:       1024-blocks     Used:
/dev/ada0p1.eli   2097152         0
/dev/ada1p1.eli   2097152         0
/dev/ada2p1.eli   2097152         0
/dev/ada3p1.eli   2097152         0
/dev/ada4p1.eli   2097152         0

 

[Spearfoot]

I have this on my 8GB backup system. It quite consistently ends up with an average of 200MB of swap used. Generally its some unused processes which get ejected.

Its not a problem, except when a disk fails, the box crashes with a pagerd fault. Which is a problem.

Is there a tuneable or something so that I can limit the ARC or something, so that it *doesn't* use the swap, although there is swap available?

And I guess... perhaps if it needs the swap, it seems like it never stops using it... perahps a periodic script to swap-off a given disk would do the trick?

View attachment 12613
View attachment 12614
View attachment 12615

Are you using 9.3 or 9.10? My systems never used swap with 9.3, but both of them use a small amount now that I've upgraded to 9.10.

I have always set up a 4GB swap file as described in section 13 of this article and configured FreeNAS with a 'STRONGLY DISCOURAGED' System->Advanced setting of 0. I wouldn't do this if FreeNAS was running on flash, but I installed it on SSDs.

 

[Stux]

Are you using 9.3 or 9.10? My systems never used swap with 9.3, but both of them use a small amount now that I've upgraded to 9.10.

I have always set up a 4GB swap file as described in section 13 of this article and configured FreeNAS with a 'STRONGLY DISCOURAGED' System->Advanced setting of 0. I wouldn't do this if FreeNAS was running on flash, but I installed it on SSDs.

I'm using 9.10. I have the "Swap size on each drive in GiB" set to "2", which is I believe the default.

Three reasons:

1) Its the default
2) It provides a safety buffer. If in the future I have to replace a disk with a disk which is a little bit smaller, I can reduce the swap partition on the new disk to make it work
3) Since I'm booted off flash, I can't have swap on the boot device, and *if* BSD/ZFS *needed* more than 8GB of memory to mount a filesystem, this would prevent a catastrophic hang, since with an extra 2GB per disk, it will eventually get there.

BUT the 10GB of swap should only be used in a pathological situation... at least that was my understanding. If it is getting used regularly on 9.10, and wasn't on 9.3, then it seems like some balancing variable, etc, is out of whack in 9.10.

In the meantime,

# swapoff -a ; swapon -a

will 'fix' the situation temporarily without a reboot.

 

[Ericloewe]

"Swap size on each drive in GiB" set to "2", which is I believe the default.

Yup.

BUT the 10GB of swap should only be used in a pathological situation... at least that was my understanding.

Yup.

 

[cyberjock]

If you're having to use 200MB of RAM for regular system use, that means you're overloading the system for its resources. Either add more RAM, turn swap off (and this may cause crashes and other problems), or simply reduce your workload.

As for 9.3 versus 9.10, I'd expect 9.10 to use more resources. Just like everything in this world, with each release more resources are needed. :/

 

[Stux]

Unfortunately the system's memory is maxed. Yorkfield class CPUs have an 8GB limit.

Hard to imagine a lighter loaded system. Two Macs write to an AFP share via TimeMachine every other hour.

Since I swapped off/on the swap earlier this week it's been on zero swap usage. Will see what happens after a reboot.

 

[cyberjock]

Well, we weren't joking when we said 8GB was the minimum. This is the dilemma you end up with when you use hardware that maxes out with the minimum requirements. :/

 

[Stux]

Well, we weren't joking when we said 8GB was the minimum. This is the dilemma you end up with when you use hardware that maxes out with the minimum requirements. :/

Yes, I'm perfectly aware of that, so back to the question

"Is there a tuneable or something so that I can limit the ARC or something, so that it *doesn't* use the swap, although there is swap available?"

This machine has 6gb of ARC almost all the time. How come it uses 200MB of swap sometimes.

I have a workaround, but perhaps there is something amiss with 9.10 too, that's what the thread is about.

 

[Stux]

So, swap usage has gone back to 5.3MB. sometime over the last 24 hours.

You can see over the same period there was plenty of RAM available... after all the purple line is pretty much the ARC... which *can* be dumped.... right? And indeed should be dumped in order to prevent swapping... right?

Free gets down to 59MB. Should that not be more like 1GB?

You can see the ARC pretty much mirrors the Wired line... less 1GB.

Is it possible that the issue occurred during this usage spike:

(when another server was restarted and remounted an iscsi zvol)

And Arc hit ratio for completeness.

Now...

Is the swap usage *really* caused by a lack of memory. Or is it caused by something being not quite right?

It appears to me, that the ARC is growing too big. What is its constraint supposed to be?

 

[Stux]

This script seems to do the trick, if I run it every few hours etc, then it should minimise the chance of a system crash when/if a disk fails.

Code:

$ cat dump_swap.sh
#!/bin/bash

# pages in all swap by iteratively cycling swap off/on on all swap devices
# providing there is more than one swap device this should be safe as if
# swap is truly necessary it will be shuttled between devices since there
# will never be no swap devices available.
# 'swapoff -a ; swapon -a' is not safe if there is not enough ram to contain
# all of the swap.

for swap_device in `awk '/swap/{print $1}' /etc/fstab`;
do
  echo "Paging in swap on $swap_device"
  swapoff $swap_device ; swapon $swap_device
done

Sorry, i'm not very good with bash ;)

 

[MrToddsFriends]

Since upgrading to 9.10 I'm seeing swap usage on my system as well, usually a few hundreds MBytes after a few days of uptime, so this seems to be normal in 9.10. In 9.3.1 swap was hardly ever used.

A1SAi-2750F with 32 GB
Pool1: Single mirror vdev 2x3TB
Pool2: Single mirror vdev 2x2TB

The upgrade from 9.3.1 to 9.10 was performed "at the W of Week 12" (time axis of attached screenshots). The increased arc size seems to be a direct consequence of a more aggressive default setting in 9.10. According to arc_summary.py vfs.zfs.arc_max has a value of 32296763392 on my system in in 9.10 and was significantly lower in 9.3.1. No autotune in action, no tunables set.

I don't know if swap usage in 9.10 is a consequence of the increased maximum arc size or some other changes that came with the upgrade.

 

[toadman]

I'm seeing it too in 9.10. But over 1GB of swap being used. I went from 8GB RAM to 12GB. Still saw it. Started playing with vm.kmem_size (set to 10Gib) and vfs.zfs.arc_max (set to 8GiB). Once a scrub started i'm seeing 2GB of swap used.

What's puzzling is I am showing an arc of 5.2G, wired of 6.2G, and inactive of 4.7G. Why would the system swap if it's got 4.7G of inactive memory sitting there??

(No jails running, etc. NFS, CIFS, SSH, SMART, and SNMP are the only services turned on.)

 

[MrToddsFriends]

After updating to FreeNAS-9.10.1 (d989edd) I did not see any swap usage so far, uptime now 8 days, 5:01.

Can anyone tell if this is an intended new behavior caused by changes in configuration/design in 9.10.1 or just a random observation by me?

I couldn't find anything related in the 9.10.1 release notes and on bugs.freenas.org.

 

[toadman]

I can confirm that after updating to 9.10.1 I am not seeing any swapping (yet, after 5 days). And what's more, the slow erosion of the arc size I was seeing is now gone. The arc is maintaining it's size and there is not a growing amount of inactive memory.

I'm not sure what changed, but the memory behavior is now back to what it was traditionally.

 

[MrToddsFriends]

Yesterday I saw swap usage on my system for the first time since running 9.10.1 (32GB RAM, plenty of space for ARC) during a scrub. So while the behavior in 9.10.1 seems to be closer to that of 9.3, it still doesn't seems to be there.

Judging from the sluggishness of the FreeNAS GUI while I observed this I guess that FreeNAS GUI and/or middleware processes were swapped out and had to be read from disk while the scrub was running.