11.2 Beta1 Netdata ram swapped out - system.swapio

[Rickinfl]

Not sure if this is a bug or not and if anyone else is having an issue, but started after upgrade. Netdata keeps sending me errors.

netdata notification
freenas.home.lan is critical
system.swapioCHART
30min ram swapped out = 25.4% of RAM
the amount of memory swapped in the last 30 minutes, as a percentage of the system RAMALARM
swapFAMILY
WARNINGSEVERITY
Wed Jul 11 02:03:00 EDT 2018TIME


Then I get this:


netdata notification
freenas.home.lan recovered
system.swapioCHART
30min ram swapped out (was warning for 30 minutes and 3 seconds)
the amount of memory swapped in the last 30 minutes, as a percentage of the system RAMALARM
swapFAMILY
Recovered from WARNINGSEVERITY
Fri Jul 13 14:51:04 EDT 2018
(was warning for 30 minutes and 3 seconds)TIME

 

[dlavigne]

This has been reported at https://redmine.ixsystems.com/issues/38105. Note that you won't be able to view that ticket until the dev has diagnosed the cause and removed the user's debug info.

 

[Rickinfl]

Thank You Dlavigne

 

[diskdiddler]

I rebooted my darn system ths morning for other reasons, if you've got a debug you can attach now, feel free to? - I have unmarked from private

 

[Rickinfl]

Sending it up now

 

[Rickinfl]

Mark it back Private

 

[diskdiddler]

I think you can actually mark it private when you upload.

 

[diskdiddler]

I've replicated the problem, didn't reboot my machine this time before saving a debug and uploaded mine also.

 

[kaipee]

I'm not entirely sure this is Netdata related.

I purchased my FreeNAS Mini with 16GB RAM roughly 3 years ago and have been running it smoothly since. The recent upgrade to 11.2-BETA1 has seen my RAM usage max out and I am now swapping to disk for the first time ever. Note that these are metrics taken from htop not Netdata (although Netdata had sent me over 100 email alerts about both RAM and Disk Usage in the first day since upgrading to 11.2-BETA1)

 

[Wolfeman0101]

I'm not entirely sure this is Netdata related.

I purchased my FreeNAS Mini with 16GB RAM roughly 3 years ago and have been running it smoothly since. The recent upgrade to 11.2-BETA1 has seen my RAM usage max out and I am now swapping to disk for the first time ever. Note that these are metrics taken from htop not Netdata (although Netdata had sent me over 100 email alerts about both RAM and Disk Usage in the first day since upgrading to 11.2-BETA1)

I feel like something is up with mine too. I have 32GB of RAM and still swapping with only 16TB of storage. I think there is a RAM bug in the 11.2 release.

 

[diskdiddler]

How many jails, VMs?

How big is the array and the amount of disks?

I know there was an issue with BETA1 - but i thought B2 might have fixed it.

 

[kaipee]

For me, just the standard 4 disk in RAID 10. It's a basic FreeNAS Mini.

I have 6 Jails (mostly plugins, a few manually created) that have been running for roughly 3 years without ever swapping (that I've noticed).
16GB RAM.

Edit : top output for memory

Mem: 18M Active, 407M Inact, 349M Laundry, 15G Wired, 263M Free
ARC: 11G Total, 1878M MFU, 8540M MRU, 7396K Anon, 121M Header, 457M Other
9560M Compressed, 10G Uncompressed, 1.08:1 Ratio
Swap: 4096M Total, 1960M Used, 2136M Free, 47% Inuse


 

[Wolfeman0101]

6 4TB disk RAID-Z1. 6 jails and pretty lightweight except for Plex. No VMs.

Xeon E3-1276 v3 @ 3.60GHz
32GB RAM

 

[PopNFresh]

Just wanted to quickly add that I am seeing the same exact issue that you guys are describing. I've got a new build where I started with 11.1 Beta 1 and had no problems. After updating to Beta 2, I started getting flooded with the same Netdata email alerts and swapping starts to occur. I've found that a replication task running for just a few minutes will cause this to happen a lot faster. After a fresh reboot I started my replication tasks and within 12 minutes the RAM usage had maxed out and started swapping.

3 x 1TB disks RAID-Z1 w/ 16GB of RAM. 6 jails and pretty lightweight for me as well. No VMs.

 

[kaipee]

So I had a quick check back on my reports and indeed it shows a major increase in SWAP usage just after I installed 11.2 BETA-1

 

[kaipee]

Now I'm receiving these daily security run alerts

Code:

kernel log messages:

swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed
swap_pager_getswapspace(32): failed

-- End of security output --

 

[kaipee]

Code:

root@FREENAS:~ # sysctl vfs.zfs.arc_max
vfs.zfs.arc_max: 13977464832
root@FREENAS:~ # sysctl hw.physmem
hw.physmem: 17109540864

 

[anodos]

Code:

root@FREENAS:~ # sysctl vfs.zfs.arc_max
vfs.zfs.arc_max: 13977464832
root@FREENAS:~ # sysctl hw.physmem
hw.physmem: 17109540864

What beta version are you currently running?

 

[kaipee]

What beta version are you currently running?

OS Version:
FreeNAS-11.2-BETA2

(Build Date: Aug 1, 2018 17:7)


Processor:
Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (8 cores)



Memory:
16 GB

 

[diskdiddler]

I mean you may as well install Beta 3 to see if it helps, right?