System running out of swap space

[perana]

Recently upgraded to TrueNAS 12 RELEASE after running the beta's and system was running fine on RC1.

Since upgrading the system is becoming unresponsive after about 24hrs, checking the logs I can see that it is running out of swap space and killing the network connections it seems, system has 32GB of RAM, and looking at logs RAM utilisation is completely fine, so I'm at a bit of a loss

Error messages in console look similar to this for many different processes:
Nov 1 08:51:30 Fileserver kernel: pid 361 (python3.8), jid 0, uid 0, was killed: out of swap space

 

[sretalla]

Check your tunables for vfs.zfs.arc_max and reduce that number by a GB or 2 if it's there.

 

[perana]

Thanks, was set to around 30GB so I've reduced it a bit, will see soon how it is

 

[sretalla]

I'm not sure if it's a slightly bigger OS or something else that caused it, but the autotune parameters from 11 seem to have been not far from the total RAM amount, which TrueNAS 12 seems to be unable to work around.

I reduced it a lot on my server with 128BG of ram (to 92GB) and haven't seen swapping since, but I have no idea what the sweet-spot is (I guess some function of total RAM - VM/Jail needs - System needs).

 

[SweetAndLow]

I'm not sure if it's a slightly bigger OS or something else that caused it, but the autotune parameters from 11 seem to have been not far from the total RAM amount, which TrueNAS 12 seems to be unable to work around.

I reduced it a lot on my server with 128BG of ram (to 92GB) and haven't seen swapping since, but I have no idea what the sweet-spot is (I guess some function of total RAM - VM/Jail needs - System needs).

Don't use auto tune it's not what you think.

 

[sretalla]

It has been off since 9.10 on my system, but the tunables are still there... should I delete those too?

 

[SweetAndLow]

It has been off since 9.10 on my system, but the tunables are still there... should I delete those too?

Yes

 

[perana]

Adjusting the tunable didnt help, I've now removed them all old autotune ones and will see how that goes.

 

[perana]

That didn't help either, I've now reverted to RC1 and will see how that goes again

 

[SweetAndLow]

That didn't help either, I've now reverted to RC1 and will see how that goes again

For perspective in 6 years of running freenas I have used 0 bytes of swap. If swap is being used there is usually something wrong. You can also look at turning off jails to see if one is misbehaving.

 

[perana]

On RC1 it has been fine for 24 hours now, made no other changes so to me it seems like something with the release.